I know of at least two approaches.
1 is to use an OnRecordLoad script step attached to the layout.
there you need to look with the Get ( RecordOpenState ) for 1:
1 returns for a new record that hasn't been commited.
then your dialog box and Revert Record on cancel.
2 is to use custom menus. There you can attach your own script steps.
If you want to change the behavior of the buttons on the toolbar you need to create a custom menu set and have a script replace the functionality of the Add New Record item. Your script can do anything - such as prompt a user with a custom dialog box. This will require FileMaker Pro Advanced in order to create the custom menu.
I failed to mention the "add new record" button is a custome button generated via filemaker in layout mode, and not the "new record" button in the filemaker menu bar.
Would this method show that same message when just opening that layout?
No, because you're trapping for a specific case: a new record being created.
The trigger will fire every time you go to another record and when entering the layout, but the overhead is not that big.
Just consider it in your own solution and possibly for looping through records use another layout.