I find the default dialog box that pops up when FileMaker is doing data validation to be unintuitive and would like to create my own to replace it.
Can this be done? How?
You can replace the error message that is shown using the Validation options, but you can't change the values displayed in the buttons. If you need something more than that, then you can use a script trigger which does the validation checks and presents your own Show Custom Dialog options.
Oh, that's too bad!
I have found the popover button to be a good solution for this in most cases. I put it where I want it on the layout, and under "Hide object when" I put 1 or true. In other words, the button itself is always hidden. But the popover that opens in that case is still visible once you get it to open. You get it to open by giving the opened popover itself (not the button) an object name, or you give some object on the popover like a field an object name. Then in your script you put a "go to object" step and a pause rather than a "show custom dialog" step. Any buttons on the popover can then be set to resume the script, exit the script or halt the script, depending on what you want them to do.
One hidden jewel with this is that if you call a script and choose to resume the script you have paused, you can end the called script by setting a value for the "exit script" step and pick it up in the next step of the script you have paused with the get(scriptresult) function. This adds a level of functionality that goes way beyond custom dialog's reach.
I have dramatically reduced my use of the show custom dialog script step since I started doing this.
This is a technique for replacing the custom dialog in Filemaker, not specifically related to field validation. To use it for that, you might have to experiment with scripting your validation rules in script triggers rather than putting them in field definitions. This gives you more interactive control like you're wanting, but has the down side of having to be implemented in every context where the field value may be changed. Still, quite a flexible and powerful alternative to the limited popups that you're struggling with.
Retrieving data ...