Skip the OS X Server install. FileMaker Server 13 wants complete control of the Apache webserver on both ports 80 (http) and 443 (https). There's a handful of services in OS X Server that want to use the Apple controlled Apache. You can't get both products to share kindly with each other.
FMS on a stock Mavericks install works like a champ. It'll save you lots of headache in the long run.
I know that it's incompatible with a mini and OS X Server 10.9. I agree with Perren - leave FMS13 to its own.
Interesting. So the traditional advice to always run on "server" has changed. This should be posted in the Knowledge Base
It appears to conflict with Windows Server as well
> Skip the OS X Server install