Thanks for posting!
I was unable to replicate the behavior you described. I created a new database on a Windows 7 machine using FileMaker Pro 11v3 containing serveral fields. I then entered layout mode and went to Layouts -> Set Tab Order. Immediately, a small window pops up titled Set Tab Order with serveral options including an OK button which allows me to save the order I've set.
Are you able to reproduce this issue in a new database? What about on a completely different machine?