FileMaker 11 does not handle finds any differently from older versions.
Next time you click that button check to see if the word "browse" in the lower left corner of the window has changed to "find" as this is the indicator to tell you if the window is in find mode, browse mode, layout mode or preview mode.
For some reason it would appear that the window never entered find mode. Try selecting that option manually from the View menu and see if you can enter find mode that way. And is there any chance that buttons reading "cancel" or "continue" are visible in the status tool area at the time you can't perform this find?