Thank you for the post.
A few questions to assist us with narrowing this down:
1. What version of Mac OS X is the operating system?
2. Which version of the FileMaker Server 12 installer was used?
3. What version of Java is installed?
4. Does the .jnlp file download?
5. What browser(s) have been tried?
If only Safari was tried, then what happens when using FireFox or Chrome?
For OS X 10.6.8 Snow Leopard use the following articles:
For OS X 10.7.5 Lion - 10.8.5 Mountain Lion use the following articles:
For OS X 10.9 Mavericks use the following article:
Let me know if any of the above needs further clarification.