I have used layouts and a plug in to create customized dialogues.
A Free Plugin that is available is one called Dialogue Plug In, not sure how I stumbled across it, but its options are almost endless. The link for the plug in is: http://kargas.net/dialog.html
To use a layout as a dialogue, simply create new layout and size the window the appropriate size. Add buttons as required ensuring the script performs in the other window.
I haven't done anything using web view or plugins, but making your own custom dialog is pretty easy now. Just make a layout for it, have your script open a new window with that layout, specify the size and position you want and "Specify Advanced Style" as a "Dialog Window (Modal)."
Make sure you test your script thourougly. It's easy to get yourself stuck with a window that you can't close! While testing, put a button on your layout that can close the window.
I would also recommend the Dialogue Plugin mentioned by ptrc, that's come in very hand for us in a few cases with some really complex screens. However, building your own using modal windows (assuming Filemaker 12) has the advantage of being supported in Filemaker Go. I would presume that (Filemaker Go) is also the case with a web viewer version, but I don't have any direct experience with that.
Thank you for your information!!
Thank you! i will test that ! Good to know the new feature in filemaker 12 !
Thank you for your more description on the issues!!
Good point. Just bear in mind, that in Go, a new window will always be full screen, displayed on top of the other window(s). So you will not be able to see info from the window "below" the frontmost one. Otherwise, the functionality is equal.