Hmmm, do you see any additional issues with windows/dialog when you have FileMaker running? I've seen a number of cases where you can open a few windows, in FileMaker alone or FileMaker plus other apps and then you can't open any additional windows or dialogs until you close some already open. Sometimes the window/dialog just doesn't open, sometimes the window is missing key objects and sometimes it just crashes FileMaker. I've seen this get worse and worse as the weeks go by until you can't open any windows at all when FileMaker is up and running.
In my case, re-installing FileMaker fixed the issue.