3 Replies Latest reply on Jan 21, 2013 4:36 PM by philmodjunk

    Complex search blocks server 100% EVEN AFTER Cancel, Client Disconnect and File Close

      Summary

      Complex search blocks server 100% EVEN AFTER Cancel, Client Disconnect and File Close & REQUIRES SERVER RESTART

      Product

      FileMaker Server

      Version

      12.0.3.327

      Operating system version

      Mac OSX 10.8.2

      Description of the issue

      Canceling a complicated search does not seem stop the thread on the server, which is performing the search and using 100% CPU, and the only way to stop it (after ~4 hours) was to stop and restart the fmserver service using the console.

      This entry covers multiple issues:

      1. The cancel button in the search progress dialog seems to have no effect, other than disappearing when you click it.

      2. Using the Admin console to close the file: closes the file, but does not interrupt or kill the thread performing the search, which continues to use 100% CPU.

      3. Using the Admin console to disconnect the client: disconnects the client, but does not interrupt or kill the thread performing the search, which continues to use 100% CPU.

      4. The only method I found to kill the thread performing the search was to use the Admin console to shut all files and stop and restart the FM Server demon.

      5. Additionally, using the Admin console to reopen the file closed in step 2 only *appears* to reopen the file - it is however not visible to the client.

      6. Additionally, using the Admin console to test the consistency of the file closed in step 2 produces an error.

      7. There is also the question of whether the search itself is faulty, causing the search-thread to work for ~4 hours before being killed.

      Steps to reproduce the problem

      1. Perform a complicated CPU-intensive search on a large dataset (e.g. search ~1/4 Million records over a relation with 7 criterion of which 5 have ≤ operators*)
      2. Analyse the server process using the activity monitor and note the thread id of the thread performing the search.

      3. Try to cancel the search from the client by clicking the cancel button
      4. Analyse the server process using the activity monitor and check if the search-thread is still running

      5. Try to cancel the search from the server in the admin console by closing the file
      6. Analyse the server process using the activity monitor and check if the search-thread is still running

      [Aside 1: reopening the file doesn't work:
      7. Reopen the file in the admin console
      8. In the client open remote and select the server
      ]

      [Aside 2: performing a consistency check on the file doesn't work:
      9. Try to do a consistency check of the file in the admin console
      ]

      10. Try to cancel the search from the server in the admin console by disconnecting the client
      11. Analyse the server process using the activity monitor and check if the search-thread is still running

      12. Close all files, stop and restart the server using the buttons in the admin console.
      13. Analyse the server process using the activity monitor and check if the search-thread is still running

      * For example, my database CrossCheckHelper to find layout objects with broken tool tips in fmp12, because they are hidden behind other objects.

      Expected result

      1. The search should not take longer than 1 hour

      3. Clicking cancel in the client should cancel the search within a reasonable timeframe and cease to block client and server.
      4. The search-thread should not be still running and CPU-usage should be low or zero.

      5. The file should be stopped
      6. The search-thread should not be still running and CPU-usage should be low or zero.

      [Aside 1: reopening the file doesn't work:
      7. The file should be marked as Normal in the admin console
      8. The file should be visible in the client's open remote dialog
      ]

      [Aside 2: performing a consistency check on the file doesn't work:
      9. File should be OK
      ]

      10. The client should disappear
      11. The search-thread should not be still running and CPU-usage should be low or zero.

      12. The server should restart (but I don't expect to have to do this in the first place!)
      13. The search-thread should not exist and CPU-usage should be low or zero.

      Actual result

      1. :-( The search took ~1 hour before clicking cancel and the thread was still running after 4 hours.

      3. :-( The search progress dialog remains on the screen and the client remains blocked..
      4. :-( The search-thread is still running and using 100% CPU.

      5. :-) The file is marked as stopped in the admin console
      6. :-( The search-thread is still running and using 100% CPU.

      [Aside 1: reopening the file doesn't work:
      7. :-) The file is marked as Normal in the admin console
      8. :-( The file is not visible in the client's open remote dialog
      ]

      [Aside 2: performing a consistency check on the closed and reopened file doesn't work:
      9. :-( The consistency check causes an error in the server log
      ]

      10. :-) The client disappears from the client list in the admin console
      11. :-( The search-thread is still running and using 100% CPU.

      12. :-) The server restarts - but also :-( because I still don't expect to have to do this in the first place!
      13. :-) The search-thread does not exist and CPU-usage is low.

      Exact text of any error message(s) that appear

      Server log entry in step 9 (german):

      Das Sicherungsprotokoll kann nicht gelesen werden. Um Festplattenspeicher freizugeben, löschen Sie „filemac:/Dasi/advanter_progressive/Removed_by_FMS/CrossCheckHelper10479108_39075.fxl“ manuell. (10709)

      approx:

      "The log cannot be read. To free-up disc-space please delete ..."

      Configuration information

      I have a further 3 screenshots and the database with which this problem occurred for you.

      Workaround

      none

      - or just don't perform complex searches
      - but how do you know what is complex?

      BUG_1_fms12_100pc_busy.png

        • 1. Re: Complex search blocks server 100% EVEN AFTER Cancel, Client Disconnect and File Close & REQUIRES...
          philmodjunk

               Being unable to kill a Find Request after initiating it is very frustrating. I suspect but can't prove that this happens because the original request spawns a process intended to build an index for a non-indexed/non-indexable file--which can be a very lengthy process on tables with large numbers of records. The cancel then kills off the find request, but the "index building" process spawned by it continues onward...

               Just my guess on why this happens and we really do need the ability to halt these.

               Of course, it's also a good idea to design some interface modifications for layouts based on large tables to keep users from performing such finds in the first place, but that does not elminate the need for correcting this issue.

          • 2. Re: Complex search blocks server 100% EVEN AFTER Cancel, Client Disconnect and File Close & REQUIRES...

                 Hello Phil,

                 thanks for the interesting information about spawned indexing processes.

                  

                 (...'fraid I can't find the Logs at the mo' for more information - oh yes, there they are :-) ... So here is an anaylsis of the process that was blocking. )

                 I don't think it was a spawned indexing thread blocking the server, because it is the same thread that called Draco::DBRemoteQueryProgram that is  blocking. IFF that is the case, then what you say - "cancel then kills off the find request" - appears not to be the case (BUG 1).

                  

                 From your answer I was not able to understand the Bug-Status of the various issues, I would thus be very grateful if you could you confirm and forward the following bugs to FileMaker Inc., :

            BUG 1. The cancel button in the search progress dialog of the client does not cancel the search thread on the server.

            BUG 2. Using the Admin console to close the file: closes the file, but does not interrupt or kill the thread performing the search, which continues to use 100% CPU.

            BUG 3. Using the Admin console to disconnect the client: disconnects the client, but does not interrupt or kill the thread performing the search, which continues to use 100% CPU.

            USABILITY ISSUE 4. One user performing an overly complex search can lead to the server having to be shutdown for all users - because there is no other way for either the user or an admin to kill the thread performing the search except for using the Admin console to stop and restart the FM Server demon.

            BUG 5. The Admin console does not correctly reopen the file closed in step 2. It *appears* to reopen the file - it is however not visible to the client.

            BUG 6.The Admin console to test the consistency of the file closed in step 2 produces an error.

            BUG 7. There is also the question of whether the search itself is faulty, causing the search-thread to work for ~4 hours before being killed.

                  

            I can send an example Database if necessary.

                 Thanking you again,

                 MrWatson in Hamburg

            • 3. Re: Complex search blocks server 100% EVEN AFTER Cancel, Client Disconnect and File Close & REQUIRES...
              philmodjunk

                   I can't forward to FileMaker. By posting here, you have already posted the report where FileMaker Inc. techs are supposed to look.