This is something that is much easier to set up in Access than it is in FileMaker and an area where Filemaker could improve a lot.
You have two options, both with significant limitations:
1) Show Custom Dialog
If all you need to do is enter plain text into 1 to 3 fields, this is easiest to setup, but you can't use all the layout design features such as conditional formatting and value lists so it only serves up to a apoint, but IS a modal dialog.
2) New Window, with scripted support to simulate modal behavior.
New window can open new window that with specified size and location info to serve as your dialog box. If you include these script steps after opening the window, you can force a situation where the window must be closed before the user can interact with the parent window shown behind it:
Allow user Abort [off]
Pause/Resume Script [Indefinitely]
To close this window, you use a second script that has these two steps:
close window ["Put name of window to be closed here in quotes"]
The Halt script step halts the original infinitely looping script found here. The halt script button option can also be used to halt the loop.
The draw backs to this approach are that it triggers a window resize of the parent window in Windows Systems if the DB window is maximized and you are left with controls in the border of the window that it would be preferrable to remove, but you do not have that option in FileMaker. The close box in this window will be disabled, unless you use FileMaker Advanced to install a custom menu for this layout that uses the above close window script in place of the standard close window option. You can minimize the window resize by selecting and resizing the original window back out the the largest unmaximized size possible using get functions to access the current computer's screen size and then include a step to maximize the windows when the dialog window is closed to return the window to it's original state.
Sorry for the delay in acknowledging your reply. Much appreciated.