Do the sessions respond to disconnect commands via the admin console?
If so you can try:
fmsadmin restart adminserver
in command line to see if that clears it out. Otherwise a maintenance reboot is most likely in order.
I have seen this behavior with WebDirect sessions but not desktop clients.
By chance did this server have FMS14 installed before switching to 15?
Some sessions response and others do not respond to disconnect commands via the admin console.
The file is nearly 6GB, so it's hard to tell when it's just taking a long time to close or open and when it's actually stuck.
Did a reboot of the server and am now running a consistency check, but even if it comes through fine and is up and running again afterward, since I don't know what caused all the sessions to stack up like that, I'm not confident it won't happen again. Users were reporting freezing happening in a specific portion of the database which prompted the check on the server. I don't know if the server session stack up was causing the freezing or the freezing was causing the session stack up or if there is another issue causing both.
In WebDirect, it was caused by people not properly exiting. I’d imagine the same could hold true for the FMP clients as well. If the client is force-quitting out of filemaker pro it could be trailing it off.
6gb isn’t necessarily a large file size.
Any more hints as to what the function of that “slow” section was? Mass container data? Reporting on lots of records? There may be optimizations to speed up that section and avoid potential hung sessions.
The section that was freezing does involve a lot of container data (lots of smallish files) The RC_data_FMS folder for this one field is about 79GB. Right now I'm looking into whether or not there are any corruptions in there or if it's related to the progressive backup not being able to properly function right around the same time that the users started noticing the freezing. The server logged an error about not being able to delete a file (I think that's what it said - something like that anyway) and said to manually remove a specific file from the progressive backup folder. Could be all related. I definitely agree that if the users were doing a lot of force quitting yesterday that would explain the sever sessions looking like they did. I'll keep pursuing the file consistency and run a recover and compact the file and then once it's functional again, rework that part of the UI for better optimization.
I'm not sure why the progressive backup hit that error because the drive it's being backed up to has nearly 4TB of space on it, so if that jogged a thought for you, please share. :-)
I saw this happening yesterday as well... The server was on v15.0.2 on macOS 10.12 (Sierra) and the only clients were FileMaker Pro (Mac and Windows).
One of the database files hosted on it uses External SQL Data Source (ESS) connected to a remote MySQL database, and this file in particular was having issues with FMS reporting multiple instances of clients being connected to it and refusing to release them, and with the file freezing up when trying to be accessed by other clients. I checked the Actual Technologies Open Source ODBC drivers were all up to date, as well as checking the connection to the MySQL database, and they were all ok...
In the end I closed everything down, uninstalled FMServer and reinstalled using the full 15.0.3 installer, and brought everything back up. Since then has been running fine... fingers-crossed.
I just went back to the server log. the exact error about progressive backup was: Unable to read backup log. To free disk space, manually delete "filemac:/Promise RAID/FM-Progressive/Removed_by_FMS/Comp Links1003383840_980.fxl". (10709)
I've seen this as well. FMS 15.0.3, latest OS and desktop client. Just a handful of users at any one time, but occasionally they show up three or more times (when they only have one computer) and they report slowness of typical operations. Not using progressive backups.
Not sure, but it might be a similar problem I've seen in earlier versions (I called them "ghost connections" at the time). All my users use Filemaker clients, and a single workstation so it was obvious they showed up with multiple connections in the Admin console.
In some instances a server restart was required, "ghost" clients would not disconnect even using the command-line fmsadmin tool to initiate the disconnect. When I checked with the users they would tell me either "the Filemaker application was 'not responding' so I ended the task and relogged in." or "Filemaker crashed during an operation, and I had to restart the application", or "I let the script run for 6 hours and finally gave up and closed the program to start over"
In my case it was a well-meant script which was disk intensive and built to handle a specific "emergency" situation -- fast forward 6 months and 10 different users understood it to be the "process to use if they see this problem". Another common cause was ESS connected search that "hung" (easy to do if you turn Filemaker Find loose in an MS-SQL database with 10M rows).
Bad News is - at least in my case this was indicative of a long-running, resource intensive process that needed to be retooled. The fact it existed and would create performance issues for other connected users, who would then perceive their long running (other) tasks were hung and would restart them, snowballing to the point where the server was so busy servicing ghost requests that all new requests were "slower than normal", eventually leading to server shutdown requirement.
Temporary fix make sure you have the disk performance to handle the peak issues quickly, I watch these counters
- Remote Calls/sec
- Remote Calls in Progress
- Elapsed Time / Call
- Wait Time / Call
Since you are using FMS15, you have the added benefit of the Top Call Stats, maybe turn that on and share the details and someone might spot the problem.
Unfortunately, FMS15 is still only on my DEV environment, so I don't have adequate testing with 20-50 users.
I get a similar Progressive Backup message weekly? Guessing that's a different issue or simply intermittent issue Progressive Backup code itself.
Thanks, Robert. This is really helpful. I will check the stats on calls to see if I find the likely culprit (or culprits). Snowball is definitely an adequate description of the effects.