Thanks for posting.
It might be beneficial if we verify a few things:
-It looks like the system specs are right there for FileMaker Server. Is the machine used for any other purposes (file server, DNS, etc.).
-What is the 'running out of memory' error just happening when you open a database or are submitting queries?
-How many concurrent users would you estimate are accessing your CWP databases at a given time?
I apologize I don't have an immediate resolution, but these questions will help me understand how your server machine is being used.