Thank you for your post.
Do you have any other applications currently running?
If you create a new user account, login to the new user account and launch FileMaker Pro, are you then able to turn on Instant Web Publishing?
Do you have a copy of FileMaker Server also running on the same machine?
Did you install any updates to Windows after FileMaker Pro was installed. If so, uninstall FileMaker Pro, remove the FileMaker Pro 11 folder to remove any possible conflicts, and then reinstall FileMaker Pro.
Do you have access to another machine? If so, temporarily install FileMaker Pro 11 to that machine and see if you are able to turn on Instant Web Publishing. If this works, then check out the differences between the two machines.
Please keep me updated with any progress.