Can you elaborate a bit on the type of FileMaker Server installation you are attempting? It it all on one machine? Multi-machine? And the hardware configuration of your Mac would be helpful to know as well.
You should also verify you have the latest version of Java installed on your system. Two Java updates were released last week for Mac OS so you will want to make sure that you grabbed both of them.
And finally, you will want to verify all the necessary ports are open and available on your system. You will find all the information you need about ports for FileMaker Server 12 in this knowledge base article:
The Admin Console requires at least 16000 and 16001 to be open and available to your system.