I had no problem running the trial version of 11 Server Advanced on my MacBookPro with 10.6.3. And after that trial, we replaced a 9 Advanced with 11 Advanced on a Mini running 10.5.8. I think all my IT did was to run the uninstaller to remove the 9 version before installing 11. We have been quite happy with 11 so far.
I am going to also try on another machine tonight that has never had FMP installed on it.
I was looking around on the forums and found a link where it mentioned a possible conflict with ichat server. We are running an openfire server on the same mac mini. I am going to try turning that off and see what happens. Hopefully that isn't the problem. It would be bummer to not be able to use both on the same machine.
When switching from a working IWP setup in FMP 10 Advanced to FMP 11 Advanced, I also had a failure. I could not turn it on through the GUI either. This happened in Mac OS X 10.5.8 as well as 10.6.3. I eventually determined that the problem was that I was running in a (Mac OS X) Standard user account. Once I logged in to an (Mac OS X) admin account, launched the same databases and then went in to Sharing > Instant Web Publishing, I found that I could turn IWP on. I did that and then logged out. This must have changed a file that required admin privileges, as when I next logged in to the Standard User account, IWP was on and working. I don't think I should have to log in to an admin account to do that (I should only have to authenticate as an admin user), but doing that fixed it for me. I am not certain yours will be so easy, but if you are running in a Standard user account, this is definitely worth a try. I believe IWP starts up Apache, so it needs an admin authorization to do that.