Thank you for the post.
A few questions to assist us with narrowing this down:
1. What is the operating system for the FileMaker Server computer?
2. What version of Java is installed?
Additionally, please review the following knowledge base article:
I removed Filemaker Server and reset IIS and although the admin console stated the web publishing engine wasn't able to connect to the server, I tested it through an external web page and it worked!
So the web publishing engine does actually work, however the settings for it can be amended in the admin console as it sees it as not being enabled. Luckily I don't need to change any settings so it works for me :)