There may be a problem with the server.PEM files, but if you have FileMaker 11 installed on that machine, there's no reason not to use it to access the files originally created with FileMaker 9. The file format is the same and while not all FileMaker 11 features will work in your file if it is hosted from Server 9, the file, which won't have any such features anyway, should work just fine from an FMP 11 client.
For More Information on Server.PEM files,
see (Windows): FMP installer does not install server.pem file
This is one of many acknowledged bugs that can be found in the Known Bug List thread here in the Report an Issue section of the forum.
It can also be downloaded as a database file from: https://www.dropbox.com/s/jt09b82i0xijbu3/FMP%20Bugs.zip
I've looked on the machine which was OS reinstalled and it has 10.7 rather than 10.5.8, am assuming that is having an effect as the two machines which work are both 10.5.8. May have to install 10.5 to see if that changes anything
To repeat: Why bother?
If you have fileMaker 11 installed and working on that machine, you don't need FileMaker 9 in order to access your hosted database. You can simply use FileMaker 11.