1 Reply Latest reply on Jan 18, 2017 9:37 AM by connollymp

    FMS 14.0.4 "Ghost" Connections and Server Chaos

    JeffHenry

      I've gone through the community and seen that other clients have ran into the same issue, named "Ghost Connections" in the past. I only found one thread that suggested doing a full re-install, but it was never mentioned of the cause or of a true fix. However, I think our issue a little bit more severe and doing a complete re-install of FMS yesterday night did not resolve the issue.

       

      Threads i've looked at:

      Ghost Users

      Filemaker Server not releasing client connections

      FileMaker 13 not releasing users

      Bug? Filemaker 'ghost' client session 

       

      Here are the symptoms when FileMaker Server is in its rebellious state:

      - WebDirect connections don't completely close on the server, eventually leading to exceeding concurrent licensing.

      - WebDirect does not completely load the layouts.

      - Regular FMP client connections get stuck opening and never completely load on the clients machine.

      - Terminal commands to close the connections or restart the WPE return a 10006/10007 error that the service is already running.

      - Ultimately, have to do a full stop/start of FileMaker services to get it back up.

       

      Here are the server machines specs:

      Mac Mini (OS X 10.10.5 Yosemite)

      3GHz Intel Core i7

      16 GB DDR3

      500 GB Flash storage (50% used)

      2TB External (50% used for Backups/Progressives)

       

      FileMaker Server Specs:

      FileMaker Server 14 v 14.0.412

      69 Databases with an average of 30 connections daily.

      FileMaker Pro 12-14 client connections, WebDirect connections, and PHP connections.

      PHP,XML, and ODBC all enabled.

      Standard SSL certificate from GoDaddy

      WebDirect session timeouts set to 20 minutes.

       

      I've been watching the client/server statistics and nothing seems to be putting a strain on the server. We have roughly 10 WebDirect connections during business hours. Console logs for the machine do not throw any crashes or errors when this behavior is going on.In the past few days we've had to restart the machine multiple times during business hours and we have roughly 20 different hosting clients whose businesses halt when we restart the server. I am unable to upgrade FileMaker Server to the latest 14.0.4b because we still have FileMaker 12 clients  using the server.

       

      Anyone else find an actual fix to this issue or what might be causing it?

       

      Thanks!

       

        • 1. Re: FMS 14.0.4 "Ghost" Connections and Server Chaos
          connollymp

          I've seen this issue occur with all versions of FMS from 13.0.3 up to the most recent 15 (running 15.0.1 in development, 13.0.5 in Production), all while running the exact same version of Windows Server, 2008 R2 w/ the same specs, 64GB RAM, 1TB of SSD storage. 

           

          The closest I can figure is causing it was postulated by ch0c0halicon page 2 of your top thread; to quote:

          Typically my Ghost users are a result of a client activity being performed on the server, such as a Find, where the Client has disconnected (force quit) before the operation has completed. This leaves a completed operation (FMS does NOT terminate the operation) with no responding client to report the results to. Note that a multiple record update, like a Replace, that terminates incorrectly is also an event being reported to clients. If a client is disconnected this may create a second locking condition. Terminating the original client may remove that condition allowing termination of the other clients (Ghosts).

           

          This theory is supported by the fact that the fmserver.exe process is not performing any disk activity in Resource Monitor, any numerous clients were locking up, freezing, force-quitting, then reconnecting, leading to a fresh ghost in the Client listing.

           

          The closest I got to solving the issue was by opening Sysinternal's Process Explorer utility, and sniping off individual threads reporting a status of Wait:WrQueue, or Wait:Suspended.  This got rid of certain ghosts, and generally improved performance until a point where clients safely disconnected, and the server was hard-reset for the evening.

           

          In case anyone from FMI is still investigating this issue for those of us stuck in Windows land (seriously, do you guys think we have any choice in the matter?  ENTERPRISE), here's a stack trace that might help, taken from one such suspended thread that turned out to be a ghost:

           

          ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a

          ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x712

          ntoskrnl.exe!KeWaitForSingleObject+0x19f

          ntoskrnl.exe!PoStartNextPowerIrp+0xbd4

          ntoskrnl.exe!PoStartNextPowerIrp+0x187d

          ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x91d

          ntoskrnl.exe!KeWaitForSingleObject+0x19f

          ntoskrnl.exe!NtWaitForSingleObject+0xde

          ntoskrnl.exe!KeSynchronizeExecution+0x3a23

          ntdll.dll!ZwWaitForSingleObject+0xa

          MSWSOCK.dll!WSPStartup+0x8495

          MSWSOCK.dll!WSPStartup+0x8577

          WS2_32.dll!select+0x15c

          WS2_32.dll!select+0xdd

          OmniORB4.dll!omni::tcpConnection::Recv+0x222

          OmniORB4.dll!omni::giopStream::inputMessage+0x27e

          OmniORB4.dll!omni::giopImpl12::inputNewServerMessage+0x3d

          OmniORB4.dll!omni::giopImpl12::inputMessageBegin+0xef

          OmniORB4.dll!omni::GIOP_S::dispatcher+0xf6

          OmniORB4.dll!omni::giopWorker::real_execute+0x6e3

          OmniORB4.dll!omni::giopWorkerInfo::run+0x55

          OmniORB4.dll!omni::giopWorker::execute+0x22

          OmniORB4.dll!omniAsyncWorker::real_run+0x441

          OmniORB4.dll!omniAsyncWorkerInfo::run+0x55

          FMSLIB.dll!Draco::FMWebServices::operator<<+0x1e957

          OmniORB4.dll!omniAsyncWorkerInfo::run+0x45

          OmniORB4.dll!omniAsyncWorker::run+0x27

          OmniThread.dll+0x10c5

          MSVCR110.dll!beginthreadex+0x107

          MSVCR110.dll!endthreadex+0x192

          kernel32.dll!BaseThreadInitThunk+0xd

          ntdll.dll!RtlUserThreadStart+0x21

           

           

          Let's keep track of this, guys.  For me it's certainly been one of the most debilitating issues I've ever had with FileMaker.

           

          --Michael Connolly

          1 of 1 people found this helpful