I recently had a simialr issue.
I suspected I knew the root cause because, when I was adding a new ExecuteSQL calculation field to a table; FMP didn't exactly crash on me, but it was taking too long to update the database. I took the decision to force quit FMP (probably not the best idea). I couldn't edit the layouts, or database manager after opening it again, because it said I was already modifying that layout. After disconnecting my user from the clients list in FMS using the admin console (which didn't fix the issue), I had to close the database from FMS — after reopening everything was back to normal (I then deleted my calculation field that caused the hang).
It was because, I assume, FM locks the database while it updates any changes you've made, and it unlocks it when it finishes; however, in this case, it never had a chance to finish because I force quit the application before it had a chance to unlock the DB.