Thank you for your post.
What occurs when you try to install FileMaker Server 11? At what point does it fail? Or, does it install properly but you are unable to deploy?
Have you tried reinstalling IIS7? There is a online tutorial I found through a web search engine:
Once installed, then reboot the server, and then install FileMaker Server 11 again.
Please keep me updated with any progress.