Just in case it helps somebody down the road, digging on the internet leads me to believe the following:
FileMaker Server 5.5 should be OK with Windows Server 2003 and Windows XP but not with Window Server 2008. Windows 7 is unknown. My guess is that it is technically related to Windows Server 2008 and there ARE problems on Windows Server 2008 so my guess is that Server 5.5 and Window 7 are incompatible.
As an aside, FM Pro 6 seems to run fine on Vista and sometimes OK on Windows 7. I have found people reporting problems on Windows 7.
There are known issues with Server 2003 too. The FM traffic is recognized by some Windows component as malformed and Windows will close port 5003 if it received a certain number of those requests. Whether this will bite you or not depends on how heavily used your solution is.
Once Windows closes the port, you need to reboot the server.