My FM 11 development server, a Mac Mini Server had it's drives fail. After some hair pulling I was able to get everything running. In the hopes that this may help others I'm putting this out here.
The short version install the original OS from the 10.6 OS X Server installation disks that came with the computer then install FMS 11.0v5 before doing any of the Apple OS updates. When I did this, FMS installed correctly, and I was able to complete the installation and run the FMS admin console and upload all of my databases.
Initially I installed the OS and ran the updates first. This brought the OS up to OS X server 10.6.8. When I installed FMS 11.0v5 the base file installation completed, but when it tried to continue and open the Web page to finish the installation it failed. I ended up with an error message saying to Contact FileMaker and a link to the FMS 11.0V5 kb article. After several attempts to fix this and much wasted time. I thought about this and remembered that when the server was new FMS installed correctly and ran. So I started over by booting from the installation CD. The steps at this point were:
1) Run Disk Utility from the Utilities menu on the OS installation Continue screen. Note: You have to wait a bit for the menu to appear. If you hit continue button you will not be able to do this.
2) Erase the OS disk and close Disk Utility
3) Complete the OS install and reboot
4) Log in and install FMS 11
5) Test and verify that everything is working
6) Install OS updates and bring system up to OS X 10.6.8