I was just looking at a FileMaker 12 Database that is a conversion from FileMaker 11. It was using the pause Indefintly trick to turn a window into a modal window. The window has two exits, a button and a custom menu assigned to the close window button (X button upper left on Mac, Upper right on Windows). Both exit options ran the same script and worked perfectly in FileMaker 11.
If the standard FileMaker button was clicked, the script ran, did what it needed to and closed the window. If the window X button was clicked, the custom menu failed to initiat and the script doesn't run. Using the debugger to kill the paused script and then click the Window X button and the custom menu runs the script. So I believe that this part is a bug.
So the solution to this is use the new feature in the Open window command, Advanced Style/Dialog Window Modal. For this, remove the pause step from the script and let FileMaker handle the modal window. When done this way in this instance, both the Custom menu method and the button method work.
In most of the solutions where I use this method of making window modal, I use a button or script trigger to continue the paused script and not the X/close window button. So in the other solutions I have this is not a big issue.