Clicking on a blank area of the layout commits the records and commiting the record is needed for some validation rules. Clicking on a button does not commit the record, but the action that takes place when the button is clicked could commit the record. (changing layouts, changing records, or the commit record step all will commit the record.
Thus you could set up your button to perform a script where the first part of the script commits the record and exits without doing anything else if an error is detected after committing the record.
Set Error capture [on]
If [Get ( LastError ) //there was an error from the commit ]
Show Custom dialog ["There was a validation error']
Exit Script 
#Rest of your script goes here.
Note that you can also user the onObjectValidation trigger if you also include code in the script to test for the error instead of relying on the validation setting in field options.