It's something of a long shot, but there's a file called server.pem that causes trouble sometimes. You might try replacing it as described in this thread: Upgraded from FileMaker Pro 10.0v1 to FileMaker Pro 11.0v2 - Can't see Databases on Filemaker Server...
That was it! Just copied the missing server.pem to my own C:\Program Files\FileMaker\FileMaker Pro 11 directory, and all was well!
It appears to contain Filemaker's RSA key and certificate. If it does not contain any information about me or my company, I can post it here. It is a mere 5K in size.