which version of FM server are you using?
FileMaker Server 11, Running on Windows Server 2008 RC2 64-bit
The 800MB is the maximum limit for FM 11 .
You need to use version 12.
That because FM12 is 64bit technology while FM11 is 32 bit.
Do you know the memory limit for Filemaker 12?
I do not know exactly, but in my system that has 6GB of memory the maximum is 3GB (about 50%)
probably is your system you can set 4GB.