The options in the Show Custom Dialog script step are about the dialog box. Period.
What you want to happen afterwards is what you enter as subsequent script steps. You can check which button they entered by using If/ElseIf and evaluating Get ( Lastmessagechoice ) , which returns the number of the button they clicked. So if that equals 2, and the second button was "Cancel", you'd cancel, if it was 1 and the first button was "do stuff" you'd "do stuff", etc.
The backwards nature of the buttons used to get me; the right-most button returns 1. It is the default button. I get why it is that way, just makes it hard to remember what number goes with what button when you start using the dialog box.
FileMaker's custom dialog is indeed limited. Alternatives include: a popover; a modal new window; or a plugin.