Have you tried opening other applications under these circumstances? Can you open them?
Have you observed any other odd issues within FileMaker such as the inability to open another dialog if you already have several open?
Other applications open with no problem. However, Outlook 2003 acts oddly when Filemaker is open. When creating a new email, there is no Send button. With filemaker closed, there is a Send button. I found that by lowering the file cache setting (Edit>Preferences>Memory) from 256 to 64MB, I can make the Outlook problem go away, but not the PDF problem. The machine is a Dell Precision 490 with 2GB RAM. I'm considering upgrading the memory.
I don't know if the problem I encountered with FileMaker 10 on my Windows XP system was the same as yours or not, but I had a problem where I could open fewer and fewer windows (including dialogs and browser tabs), the more I used FileMaker. Dialogs with missing elements were a symptom of this on my system. I corrected the problem by unistalling and re-installing FileMaker. Don't know if that will fix it in your case, but I shouldn't cost you anything to try it.