In Layout Mode:
If the button object (which can be a field, a graphic object, a FileMaker button object...) is not grouped with other objects, you can double click it while in layout mode to open Button Setup.
You can also select the button and choose Button Setup from the Format menu.
In some cases, you may have to ungroup groups of objects to get to the button grouped with other objects. To keep from unintentionally changing the design of your layout, you may want to use Duplicate Layout to make a copy of your layout and then you can investigate button settings on the duplicate--ungrouping moviing objects around as needed without fear of messing something up on your working copy.