Thank you for your post.
FileMaker Server 12.0v4 (nor 12.0v5) will work under Maverick's (Mac OS X 10.9) operating system. Our Development and Testing departments are aware of this issue, and they are working on an update. No information is available about a release date. Please see Knowledge Base Article #11930 for more information:
I failed to mention that the server is running on a virtual machine using Windows operating system. I am trying to launch the console from my desktop MAC. Don't know if that changes your response but would like to know.
Thanks for the additional information.
Do you know if the server installation was successful? That is, can you launch FileMaker Pro on your Mac and access the FMServer_Sample.fmp12 file remotely? If not, then you should uninstall and reinstall FileMaker Server.
Assuming it is installed correctly, are you able to launch the Admin Console on the server machine?
Assuming this all works, what version of Java do you have installed on your Mac OS X 10.9.1 machine?
Does the server have any firewall set to not allow incoming connections?
Do you have any ports blocked?
A customer with a similar issue posted to the forum. I recommend first reading the following forum thread (with answers and images) to see if this helps:
Keep me informed with any progress.