Note that this is a Windows-only issue.
The Esc key does not dismiss a custom dialog on Mac OS X.
In FileMaker 13, one workaround is to restrict yourself to using 2 buttons instead of 3 and don't have a default button. Leave the default button postion blank. Get(LastMessageChoice) will be 1 when the Esc key is hit. It will be 2 or 3 for the other button choices.
This has the same issues as making the default button a cancel button every time.
I did forget to mention that this is a Windows-only issue, which I had known. However, that's the majority of computer users in general, and my clients specifically.
This sounds like a possible candidate for using popup buttons to show a dialog instead. It's a little bit more work, but a lot more flexible once in place. In a popup, the escape key will close the popup, by default, so you can trap for that if you want until the user clicks on one of the intended buttons.
Would that work?
Re: "The Esc key does not dismiss a custom dialog on Mac OS X."
That is not quite correct. It may behave differently on Windows, but on my Mac (OSX 10.9.5, FM13) if you hold the Esc key while pressing ANY button in a custom dialog window the dialog window is dismissed and the script is halted.
I believe what you are observing is that the Esc key is serving to halt the script when the script resumes after the dialog is dismissed by a button press. The Esc key is ignored while the modal dialog is active. Once the script resumes the Esc key is registered to halt the script before the subsequent script steps can run.
I expect (on Mac OS X) that you would observe the same behavior if you depressed Command and period instead of Esc.
"Esc key is serving to halt the script when the script resumes"—spot on. By itself the Esc key does nothing.
"you would observe the same behavior if you depressed Command and period"—confirmed.