I'm not sure if it's a bug or incomplete documentation. I've just started playing with this feature today and so far can't get any layout object except a button to actually use these settings to change appearance when in browse mode.
You might want to report this in Report an Issue and see what the FileMaker Inc. folks say.
I am also having a problem with the tabs. I had a set of tabs on a tab in a database that was working fine. When I converted it, the color on the lettering was not right, so I went to the layout to change it and it will not change. Also, Filemaker 12 seems very slow to finish a task. This was my first opening of the program and I am not very happy yet.
The main tabs respond fine, but the tabset that is on the tab do not respond to formatting.
I increased cache size, but the Inspector is still extrememly unresponsive. It takes quite a while to change a color on anything. I am running Windows 7. I do not know who to report this to. I am running 12 Advanced. Although the new features look great, this is VERY DISAPPOINTING and I will have to go back to v11 until they can fix these issues.
I also tried this by creating a new database. When you have tabs on a tab, it works fines if the database was created in 12. But it does not work on my converted file.