I have systems with many times that number of layouts and go to layout does not have an issue for me.
Things to evaluate as a possible cause:
Is there a script trigger that might be tripped by the change in layouts? A large number of triggers can be tripped by a scripted layout change, both on exiting the current layout and upon entering the new layout.
What exact form of Go to Layout are you using? Might you be using either of the "by calculation" methods?
Problem solved. An errant script trigger like PhilModJunk suggested.