Possible bugs should be reported in Report an Issue.
What OS are your using? There have been a number of issues with dialog boxes reported for FileMaker and the latest version of the Mac OS so you may have found a new manifestation of that issue.
I am using OS 10.9.5. I haven't update yet to Yosemite.
At the moment I got it working back by uninstalling and installing / updating up till version 3. When i upgrade this version to v4, same problems occurs. Though I am pretty sure that when i updated to v4, it worked at first. That's why I thought maybe a config file got corrupted or whatever, so i decided to uninstall en reinstall. But what I cant figure out is why it works in all version previous to v4 when reinstalling.
Are there maybe some hidden files v4 installs? In the documentation stands aswell to remove the plist. As far as I know I did not find any plist listed like in the documentation inside the specified folder ( http://help.filemaker.com/app/answers/detail/a_id/13597/~/uninstalling-filemaker-pro-and-filemaker-pro-advanced )
Sounds like a bug to report over in Report an Issue. I'm not a Mac user at the moment and don't see this behavior on the windows side.