get(windowMode) will tell you if you're in browse mode (result = 0) or in find mode (result = 1)
tab order is by definition circular so when you tab out of the last field in tab order you will land on the first field in the tab order.
But you can put a trigger on the last field, firing on onObjectExit, consisting of just Exit Script (false), to stop the cycling.
If you include buttons in your tab order, they will be tabbed into and a CR will "click" them.
You can also use script triggers that perform scripts that put the focus on different layouts depending on conditions, such as windowmode, that you have your script check.
Both go to field and go to object can be used to set the focus on a layout object.
I would use onObjectKeystroke to check for specific keys inorder to intelligently change focus. That way clicking something doesn't trip the trigger. And you will need to use Ecut Script [False] to keep the tab/enter/return key that tripped the trigger from being processed or you won't get the focus where you want it.