I uninstalled FMSA 22.214.171.124 from the OS X 10.5.8 Server, reinstalled FMSA 10v2 and external authentication is now working again.
To test this, I installed FMSA 126.96.36.199 on a mac mini server with OS 10.6.2 server which, by the way, both the 10.5 and 10.6 servers are connected to an OS 10.6.2 Open Directory server, and external authentication fails to authenticate.
I have Client Authentication set to FileMaker and external server accounts selected in the Database Server->Security settings.
I am not new to FileMaker as we have been using it since version 3.
I found the issue.
Apparently my main admin user account to authenticate the filemaker databases to had become corrupt. Re-creating the account on the Open Directory server resolved the issue.