Thank you for your post.
It's difficult to determine what is the cause, but since 8 works on the same machine, you may have a bad server.pem file. Quit out of FileMaker Pro 9 and go to another machine that also has FileMaker Pro 9 installed. Copy the server.pem file and then copy it onto the machine that is having the problem.
The server.pem file is found by right/control-clicking the application, Show Package Contents, Contents -> MacOS.
Once copied, then relaunch FileMaker Pro 9 and see if you are able to access the file.