I almost posted a thread earlier last week on this issue, but we were able to work the problem long enough to crank out our own answer.
Essentially what we were trying to accomplish was upgrade our FMS 10 running in Tiger to FMS 11 running in SnowLeopard. Aside from the initial difficulties experienced with the File Maker Console as a result of the Java update 51, see FM forum post, we were seeing some strange DB authentication behavior. The way we authenticate (for the most part) to our databases is by using local fms (FileMakerServer) groups populated with Open Directory Users, pretty common I'm sure. But once all our databases were copied to the new (SL) server, and the proper local groups had been setup and populated with users, we were not able to connect to the database using an Open Directory user accound specified in the local FileMaker Server group. We tried everything, from creating new local groups (rather than exporting/importing from the Tiger server). Using the modifying the Goup IDs on the SL server to match those of the Tiger server. We uninstalled and re-installed FMS. I built up a whole new SL Server on different hardware. But no matter what, we'd see the same behavior. As it turns out, in FMS 11, the external authentication groups defined in the hosted database file have to match the short name of the local file maker server access group! In Tiger, this was not the case. In that environment you could setup the external authentication group in the DB file to reference the group name (the easy reading/non-shortname version). Maybe this is a commonly known limitation of FMS 11.
See the attached png for the difference I'm talking about between Group Name and Goup Shortname. Again, USE THE GROUP SHORTNAME when referencing external authentication in the DB file.