1) What kind of "computer" are you running this on? A desktop workstation, or a server?
2) What OS?
3) What kind of network do you have? How is it configured?
We've seen this issue before, and it had to do with a bad DNS server that was confusing FileMaker Server when attempting to make / break connections. My first instinct, therefore, is to suspect network issues.
The password it's needing is the one for the admin console.
1.) Mac Pro Late 2013 , 3.7 Ghz Quad core Xeon E5, 12GB RAM
2.) Mac OS X 10.10.5.
3.) Not 100% sure about all network details, but I use the cisco ASDM for ASA to configure our cisco router/firewall, and there is a juniper ethernet switch. I can get the exact cisco model number when I get into the office later this morning.
When you say a bad dns server, I'm not sure which one you mean... I'm using openDNS as the dns server in system preferences on the Mac Pro that hosts filemaker. We also have a mac mini server that is our open directory server and it's running a dns, but only locally for the lan, so I doubt it's that one...
Thanks for the details on the fmsadmin login, I was using the user name it showed in parenthesis in the console prompt instead of the admin console user name.
I'm not particularly concerned about the specific router model. If the DNS server gets confused, then FileMaker can't resolve the connections, which is why they hang. I'm really not an expert on network / DNS configurations (maybe one of the other forum members more familiar with server configuration, like maybe wimdecorte or taylorsharpe, can chime in).
Is there any way you can do some network analytics and figure out if you're getting a lot of dropped packets on port 5003 between the clients and the server?