Thanks for posting!
Could you tell me the size of your server.pem file on these installations? You can find this file in C:/Program Files/FileMaker/FileMaker Pro 9/. It should be around 5 KB. If it's not, try replacing this file with a copy from a working installation.
My server.pem file is 4KB. It worked fine under Windows 7 after disabling IP V6; however, I had to uninstall it from the computer. When I reinstalled it with IP V6 disabled it still gave the error message. Same thing with the Windows XP SP3 installation.