I know this is over a year that this question was posted, but I just had this error yesterday and I couldn't find the answer, and I have tried to go to the forum, but it kept on saying not available as it was busy with updates.
Now, what I have done to fix this, and this is for windows, and I am sure this will be for mac too, is to uninstall Filemaker Server, rename the Filemaker server directory to something like Filemaker Server Old, and removed all the Filemaker server program data from the program data folder. Reinstalled it, and did all the updates, and it worked for me.
I am not sure about the cause, but for me there was an antivirus. ESET, which expired, and as soon as I have uninstalled it it Filemaker server worked fine. but as to why, I don't know