FileMaker 7 - 11 all use the same format. Server 8 can definitely host server 9 clients, though a few 9 specific features may not work if they rely on server side processing.
Here's one possible issue you can check: Can't login to FMS database on new Mac, other Macs can
This is one of many acknowledged bugs that can be found in the Known Bug List here in the Report an Issue section of the forum.
It can also be downloaded as a database file from: http://www.4shared.com/file/8orL8apk/FMP_Bugs.html
Sorry this didn't work by trashing the server.pem or preference. ANy other tricks to solve this. Thxs