I am currently running FileMaker Server 12 on a computer which also runs OSX Server for centralized "external" authentication for database access. This is in accord with the instructions on FileMaker's External Authentication web page (http://help.filemaker.com/app/answers/detail/a_id/5508/~/setting-up-filemaker-pro-clients-and-filemaker-server-to-use-external) which says "On Mac OS, it is recommended that Mac OS X Server be used to run FileMaker Server."
I am interested in updating to FileMaker 13 Server. However FileMaker 13 Server's specs page indicates is is not compatible with Mavericks Server, only with the standard/client OS. Does that mean external authentication is not possible when running the current version of both the OS and FileMaker? Or has someone found a way to accomplish this?
Depending on how busy your deployment is, it is usually not best practice to install FMS on an OD or AD server.
You can install OSX Server on another machine, set it up in the OD role and make sure that the FMS machine is bound to that domain (passes on the authentication request to the OD server)