Sounds similar to this known issue:
For More Information see: FileMaker enters a modal-like state on editing
This is one of many acknowledged bugs that can be found in the Known Bug List thread here in the Report an Issue section of the forum.
It can also be downloaded as a database file from: https://www.dropbox.com/s/jt09b82i0xijbu3/FMP%20Bugs.zip
Thank you for your post.
Are you in List view at the time this occurs? Do you have either OnRecordLoad or OnObjectExit script triggers enabled? Any other information you can provide may be helpful.