there is an old saying in Maine.. You can't get there from here".
In this context it means that the fine control over the presentation of the dialog box is not possible.
i would think that a popover button could be set up to behave as a custom dialog or you could spawn a new window and dress either up as you like.
Thanks for very valuable tips to work on popover instead of custom dialog.
I would suggest spawning a new window that is modal and remove the controls for sizing and closing. Size and place the window where you want but the user is forced to deal with this window. This way the layout can have your theme.