7 Replies Latest reply on May 23, 2013 3:27 PM by philmodjunk

    fmxdbc_listener.exe thread leak

    ssignorelli

      Summary

      fmxdbc_listener.exe thread leak

      Product

      FileMaker Server

      Version

      11.0.3.309

      Operating system version

      Windows 2008 R2

      Description of the issue

      The fmxdbc_listener.exe develops a thread leak when the number of "in progress" XDBC clients reaches 40. Windows Task Manager shows usually more than 75 threads and continues to climb and will not come back down even if using cleanmem. Clients are using the 11.3.81.0 (installer from website indicated 11.3.82). Approximately 700-950 concurrent connections(mostly fmapp with a range of 10-30 fmxdbc, and 3-15 php)  1 file about 2.8GB. When it "goes into the weeds", the fmserver.exe process stops processing (cpu monitor shows very little activity). fmxbc clients get stuck and will not release from the server even when disconnected using the Admin Console. We have tried compacting the file, adjusting the base priority of the fmserver.exe, fmadminserver.exe and the fmxdbc_listener.exe to either above normal or high and have been using cleanmem utility. The ODBC calls are from a VBS script on the client end to the Server using SQL inserts (We used to do a count but removed that after reading of a known memory leak).

      Steps to reproduce the problem

      this occurs daily and seems to get triggered when a large number of simultaneous xdbc clients connect near the same time. Once the in progress number hits 40, the problem is reproduced without fail.

      Expected result

      "In progress" count should not get stuck at 40, fmxdbc_listener.exe thread count should reduce once work is complete, server should continue processing requests.

      Actual result

      The fmxdbc_listener.exe develops a thread leak when the number of "in progress" XDBC clients reaches 40. Threads climb without reduction beyond 75. Server stops processing requests.

      Exact text of any error message(s) that appear

      no errors during issue, however when attempting to pause the files, there is always a server event indicating the pause could not complete due to an active request from one of the xdbc clients (This would not be the case if I could successfully disconnect the clients in the first place)

      Configuration information

      10 terminal servers with FileMaker Pro 11.3 clients using a vbs script with ODBC SQL inserts all connected to a central server hosting 1 file. Also there is a PHP website connecting to this server.

      Workaround

      none fully successful. compacting helped for 1-2 days, Disconnecting idle sessions seems to delay the failure a little(30-90minutes), cleanmem also seems to delay the failure a little (30-90minutes)

        • 1. Re: fmxdbc_listener.exe thread leak
          philmodjunk

          This would appear to be a known issue.

          For More Information see:     FileMaker Server

          This is one of many acknowledged bugs that can be found in the Known Bug List thread here in the Report an Issue section of the forum.

          It can also be downloaded as a database file from:    https://www.dropbox.com/s/jt09b82i0xijbu3/FMP%20Bugs.zip

           

          Wonder if it is fixed in Server 12?

          • 2. Re: fmxdbc_listener.exe thread leak
            hschlossberg

            Phil -- the thread you pointed to above was in regard to 11.0.2.217.  This current report is in regard to 11.3.82.

            The readme for 11.3.82 xDBC update for FileMaker Server 11.0v3, dated 27-Apr-2011, includes these items:

            • Addressed a performance issue caused by a memory leak when executing certain SQL queries.
            • Addressed an issue where the xDBC listener would leak memory when a client disconnected unexpectedly.

            I happen to be up on this because I've just started running into ODBC issues after upgrading a client's server from 11.0v2 to 11.0v3, though my issues have to do with xDBC clients no longer disconnecting, even after hours of no use.  I don't think my particular case is a memory leak issue, but I'm following these things just in case.

            • 3. Re: fmxdbc_listener.exe thread leak
              ssignorelli

              One additional observation is that using the Windows Resource monitor I was able to see that the fmxdbc_listener.exe seems to create a tmp file in C:\Windows\Temp\ for each ODBC connection. Viewing the Write (B/sec) in the Resource monitor shows the combined total of this process has more disk writes than any other process on the server/any other file including the Windows Page File.

              When the XDBC "In Progress" reaches 40 in the FM Admin console, the fmxdbc_listener.exe process drops off the list of Disk I/O completely (indicating it has stopped actually performing work).

              fmserver.exe still has decent I/O. I am wondering if FileMaker cannot handle that many tmp files at once? Any thoughts...?

              -Steve

              • 4. Re: fmxdbc_listener.exe thread leak
                gdewey

                     Steve, did you got a response on this? we being trying to relay on the JDBC services but end up doing something else to get info out of FM (like using the WP XML engine. The fmxdbc_listerner.exe in our case just crashes after 4 to 5 connections.

                     I did some tweeking on the client connector and was able to perform 10-13 sim connections.. after some minutes of heavy load it just crashes..

                     I've reproduce this on mac server, and windows server.. same exact thing

                • 5. Re: fmxdbc_listener.exe thread leak
                  ssignorelli

                       we have worked around it for now.

                       \We setup a separate server that only gets connections when a special VBS script makes the ODBC call. This server only gets 5-20 connections at once instead of 1000+ and at the end of every day we import the created records into the main system centralized DB, it makes it so the client can see the entry the next day instead of real time.

                       Even then, we are averaging 3 crashes per day on the dedicated server. I am not sure how Apple expects to compete with SQL with that level of unrealiability.

                  • 6. Re: fmxdbc_listener.exe thread leak
                    gdewey

                         in some way you are making a datawarehouse to query againts other db.. we are doing just that with a scheduler that export to mysql where we can run our BI server and website.

                         Its clear for me that FM is meant for small user groups. I would love to see in wich planet FMS12-Adv JDBC service can handle 50 connections, maybe they connect but they will never get query results LOL! Not to mention the service will crash even on a fresh OS installed with12 Core 64 RAM, SSD HD  server, mac or windows.

                         Any ways we are stick in a situacion where we really need a stable JDBC. I wont walk away just like that. We have tons of work on a proyect that is just missing this JDBC service to work stable so we can close things and turn the page.

                         I've made a formal report on this. I will post in here in case I get something.

                         by the way the 360works guys have a JDBC Driver (its not on their site) that queries on the XML ENGINE instead of the JDBC service. I am trying it as we speak. I belive it will be a way more slow but stable solution to get data our of FMS

                         Guillermo Dewey

                         http://www.ofik.com

                    • 7. Re: fmxdbc_listener.exe thread leak
                      philmodjunk

                           Please note that since the initial post to this thread is more than 3 months old, comments posted to it no longer automatically make it appear in the Recent Items list. And since no TS personnel have subscribed to this thread, it is unlikely that any will notice that you continue to post to it.

                           If you want to raise the visibility of this issue, you might want to make a new issue report and include a link to this one...