This can not be done natively with a custom dialog. However, you can use a new window and a new layout to simulate it.
Another option may be Troi Dialog plugin.
It can be a little clunky on a windows system, but you can also do something like this with the New Window command to pop up a small floating window with all the layout features of any typical filemaker layout.
Issues to look out for:
- On windows machines, new window resizes the current window, you have to add extra window commands to minimize (can't eliminate it entirely) the "twitch".
- You also have to pause the script that opens the new window to make it "modal", (keep user from interacting with the background window until they've dismissed the pop up).