Thank you for your post.
At this time, FileMaker Server 11 is not compatible with Mavericks. For more information, see Knowledge Base Article #12643 at:
On a side note, the current version of FileMaker Server (version 12) is also unable to install and deploy. However, there will be an update for FileMaker Server 12 in the "next few months". See Knowledge Base Article #11930 for complete information: