Hmm, are you seeing any other odd behavior when you use filemaker? Any other trouble with windows, drop downs or dialogs? (I'm still trying to document an issue with FileMaker 10 on Windows where windows and dialog start screwing up and you may have a case for this issue.)
Any script triggers involved?
That said, it sounds like you have a validation rule that's throwing an error when you commit the record (Commit happens when you change to next record.) The empty dialog appears to be FileMaker trying to display the error message for that validation rule. If you specify a custom validation message, but don't enter any text, you might get such a blank dialog, so I'd check over the fields of your table for any validation rules and their validation error messages to see if one of them is the culprit.