FileMaker Server hung; would not respond to clients, would not restart, processes had to be killed manually
Operating system version
Windows Server 208 R2 Standard, Service Pack 1
Description of the issue
We have been seeing some slowness recently, but this problem appears to be unrelated.
Normally, if the server is responding sluggishly, all eight cores show signs of activity. In this case, there was minimal activity on the CPUs.
In other slowdowns we've seen, many clients will be sitting with elapsed times of 15 seconds, but others will be lower (or not appear at all). In this case, all the connected clients appeared to be having elapsed times of 15 seconds, except for one user who had 30. This appears to be because she had two processes.
Sometimes we see connections (and record locks) maintained even when the client goes away (because they drop off the network, or whatever). In those cases, closing and opening a database (even one with no connections) appears to force FileMaker to reassess its connection pool, and those phantom connections are dropped; but in normal cases slowness, closing and opening database is also slow. In this case, the server appeared to be very responsive to the admin console, closing and reopening the file quickly. This had no affect on the clients.
I also checked whether the MySQL server that is used for shadow tables was having issues (it was fine) and asked the room whether there were any network or web server problems (there weren't). (The web server is relevant because there are some web portals in our layouts; I would not expect that to affect the behaviour on the server in this way, but at this point I was willing to contemplate anything.)
With some trepidation, I hit the red "Stop Database Server" button. The admin console was disconnected, but the option to "Start Database Server" remained greyed out. Going to the Windows Services interface, I attempted to stop the FileMaker service -- this failed. Finally, I brought up the process manager and killed the FileMaker processes; this appeared to be enough to allow the Services window to give me the option to restart the FileMaker service, which I did. FileMaker server then came back up and, once the databases were verified and opened, everything appeared to be back to normal.
Steps to reproduce the problem
The Windows box is on a well-provisioned virtual machine, and is configured with two 2.70GHz 4-core Intel Xeon CPUs and 24Gb of ram.
There are generally between 50 and 100 clients connecting at any one time; they are encouraged to use ethernet, but sometimes have to use wifi. When switching between the two, they are told that they should stop and restart FileMaker.
The server hosts 75 databases, most of which contain multiple tables, and which range in size between 104k and 2.1G. Many of these database are rarely opened; some of them (that are frequently used) have connections to 20 or so other database files.
Web publishing is disabled.
There are ODBC connections to a MySQL server; there are a large number of shadow tables, but they are very rarely displayed directly to the users. Instead, relevant information is generally copied and cached to FileMaker tables on the server. Similarly, copies of information are regularly written to the MySQL server, and there are relationships set up so that records that are deleted in FileMaker delete the corresponding copy in MySQL.
The clients that are connecting to the server are running a mix of Mac and Windows OSes; they are generally using FileMaker Pro 11.0v3 (or v4 if on a Mac), though there is an occasional machine that Support hasn't caught running v1.
I don't know what else might be relevant, since I don't understand what went wrong.
Killing the processes and restarting the service seemed to work, but is not really an acceptable solution.