On the Mac you can't get to the red X to kill the script while the dialog is displayed so not a problem.
Thank you for your post.
I am able to replicate the issue with both FileMaker Pro 10 Advanced and FileMaker Pro 11 Advanced on Windows XP SP3 and Vista. I confirm with "John Duncan" that this will not occur under Mac OS.
If I'm already in Preview Mode prior to running the script, I am unable to select the Script Debugger. This only occurs if I switch to Preview Mode just prior to Show Custom Dialog.
I have forwarded your post along with my findings to our Development and Software Quality Assurance (Testing) departments for review and confirmation. When more information becomes available, I will let you know.
This has been fixed in FileMaker 12. You can no longer select the debugger until you dismiss the dialog so you can't strand the dialog box.