Did you disable share point? I forget all the details, but when we researched servers for installing FMS 10 at my current place of employment, we found that SBS systems were not listed by FileMaker Inc as compatible systems for FileMaker server. When we called customer service at FileMaker they indicated some kind of conflict with sharepoint on that version of windows server. Dont' know if that's the issue you are encountering here or not...
Thanks for posting!
PhilModJunk is correct on the recommendation to disable Sharepoint (thanks!). This is because Sharepoint and the web publishing engine use some of the same resources.
How much RAM is it using exactly? Are you using any plugins?
Does the memory usage climb only when users are connecting to it or when idle as well?
Is this a dedicated machine? If not, what else is running?
Thanks for the replys. Disabling sharepoint didn't seem to make a difference. Not using web publishing, no plugins. Its taking just over 3GB of ram. There are no users connected. I just have to start the filemaker server service for this to occour. Stopping the service frees up all the memory.
Its a standard SBS setup, exhange, file services, dhcp, dns. Theres is nothing else running.
The soulution is to enable 32bit apps in the IIS default app pool.