I have the same problem with Filemaker 11 (not the advanced version) ... on Windows 7 home premium PC.
I also have an Advanced license (on another PC .. Windows Vista professional) ... but there I have no problems .
Did anyone provided you with a solution?
Here's the most like explanation: FMP installer does not install server.pem file
This is one of many acknowledged bugs that can be found in the Known Bug List here in the Report an Issue section of the forum.It can also be downloaded as a database file from: http://www.4shared.com/file/8orL8apk/FMP_Bugs.html
For me the solution was :
Went to ... File : Manage : Security : Privilege sets ...
There were no active accounts.
I added the Admin account to the privilege set "full access" ... problem solved.