In the past I had not paid much attention to this. I pretty much assumed if you were installing FMS, you wanted to do so on a Mac OS X Server machine. When you do this, there are several things that the OS does to behave more like a server like turning off spotlight indexing and some other such things. But when you install the Server app, it takes over the web services among other things. If you look at the FileMaker instructions for FMS, you are now not supposed to install the Server.app to make it a Mac OS X Server. You're supposed to leave it as a regular Mac OS X and then run the FMS install that takes over Apache web services. This also makes sure you don't have any other services on the Mac like file sharing, DNS, mail, etc. Officially, FileMaker will not answer tech support questions if you do install the Server.app. So, running FMS on a Mac really means a single service machine and remember not to install the Server.app that upgrades the OS X to OS X Server.
Out of curiosity, I have installed the Server.app on some FMS machines. Does anyone know a way to uninstall the Server.app? The only way I can think of is to do a system reinstall.