While a dedicated machine is recommended, FileMaker Server Advanced and SQL Server (or other applications) can be installed on the same machine so long as these applications are not eating up all of the machine's resources.
Even though you can have SQL Server and FileMaker Server on the same machine, it might be a good idea to test FileMaker Server in an environment where there are no other applications running. Try Run > MSCONFIG > Services tab > check Hide All Microsoft Services > Disable All > Apply > Close > Reboot.
Doing this stops any applications you have in the background. Keep in mind that you will need to re-enable these services (follow instructions above but choose Enable instead of Disable All) after testing so that you are in the same environment as before.
Prior to that, however, I would try to see if FileMaker Server will run via the command line:
C:\Program Files\FileMaker\FileMaker Server\Database Server\ fmsadmin start server
Btw, what version of FileMaker Server are you using? Also, are you able to list what FileMaker services are listed in TaskManager?
Thanks for posting!
My apologies, ts_turtle. I should have mentioned that it is version 9 of FM Server Advanced. And thanks for the suggestions.
Our system admin is a much sought after person, so it took a while before he could try them. Starting from the command line produced the same error message. After disabling the Microsoft services, the error message appeared no more, but the FM database server still appeared to be stuck.
After removal of SQL Server and the application that needed it, and also of FM Server, a re-boot was made and FM Server re-installed. It then ran as usual.
After subsequent re-installation of SQL Server and the associated applications, and re-boot, the FM Admin Console reported again "error was encountered communicating with the server" (which was the very first error message that started of the previous de-/re-installation cycle).
The Console further shows Web Server on, Web Publishing Engine on (for IWP, XML, and XSLT), but Database Server off (for ODBC and JDBC). The latter was on prior to the re-installation of SQL Server.
It looks as though there is some incompatibility with either SQL Server or the special purpose application that uses it. We are considering now to install these 2 on a different server.
Meanwhile, many thanks for your kind support.
FYI: further investigation showed that FM Server 9 and SQL Server can indeed co-exist peacefully on the same machine. The problem only appears when the 3rd party application is installed which needs to make use of both SQL Server and IIS.