12 Replies Latest reply on Dec 5, 2016 2:19 PM by TSGal

    FMS Server 14 Crashes with multiple PSOS and Complex Searches

    lmagnan

      Product and version (e.g. FileMaker Pro 14.0.3)

      Filemaker Server 14.04.412

      OS and version

      Windows Server 2012 Standard

      Browser and version (for WebDirect only)

      N/A

      Hardware

      16 Core Intel Xeon CPU E5-2630 @ 2.4 Ghz, 8 GB RAM, 64-bit Operating System

      Description

      I was trying to run a complex search into which I was looking for the levenshtein distance between a field and a local variable. I am importing new entries in my database once in a while and the idea is to compare imported entries with existing entries in two different tables.

       

      My search is run from the table I call the approval document table, where I set a local variable from a value list and compare certain entries of that table (from a found set previously established) to the local variable using the Levenshtein Custom Function in a first find request. In the second find request, I use the same CF to look for differences between another local variable and a field in the related table I call modification table. The search is run against an unindexed calculation field that contains the levenshtein function pointed at another field on that record and to the local variables previously set by the script. I ask FM to get anything that is within a certain distance that is different for each of the imported entry I run the scirpt against, a little like saying "give me everything that is within 20% similar to this string".

       

      Since the CF takes a while to resolve, I thought I had a great idea by splitting my list in pieces and having multiple instances of the same script running on the server at the same time, each doing their part of the task. However, as soon as I run more than one instance, I start seeing DMP files in the Log Folder of my FMServer. I basically get one DMP per search action. Since the script is a loop that looks for around 100 entries total (in between the various instances) each time I run it, I end up with quite a few different DMP files that are most likely containing the same thing.

       

      When I analyse the DMP file using WinDebug, I get an C0000005 exception against FMServer Support.DLL. One thing I did notice during my trials is that I was getting super high CPU frequency (around 120-130%) for a somewhat low CPU load (around 12-18%). The script goes through without any issues if I run only a single instance of it. FileMaker Server doesn't actually crash and I get expected search results. However, I do get some sort of admin console lock-up for certain user, and I am incapable of properly closing the database or turn off the server without using power shell and restarting the server machine.

       

      The load on the server remains low at all times, and I had somewhere around 3GB of memory available for cache when I was running the scripts, if this info can be of any use.

       

      Levenshtein CF: FileMaker Custom Function:Levenshtein ( str1 ; str2 ; damerau ; caseSensitive )

      Community Discussion related to this issue: PSOS Complex Search Script generating a dmp file

       

      How to replicate

      Run more than one scripts at the same time performing the Levenshtein function.

       

      Workaround (if any)

      Run only one instance of the script.

        • 1. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
          TSGal

          lmagnan:

           

          Thank you for your post.

           

          Looking at the Community discussion, you mention that it takes 10-15 minutes to run the script.

           

          What is the length of the string and the length of the Field?  Multiplying those numbers together would give you the number of evaluations performed to obtain the Levenshtein distance for one record.  The Script Engine will be focusing all of its resources on these evaluations, so running multiple scripts could time out or stop the Script Engine.

           

          After a crash, does the Admin Console show any process that stopped?

           

          What does the Event.log show?

           

          TSGal

          FileMaker, Inc.

          • 2. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
            lmagnan

            Hello TSGal,

             

            Length of the tring can be anywhere between 5 or 6 chars and 20 chars on both sides of the function, as they are both supposed to be fairly similar. It is looking at 500 different records that are listed as approval documents, although I thought the way I had it structured (the calculation in an unstored field for each record) it would be considered a separate calculation for each and every document. I take the time to strip out non alpha numeric chars before running the function to ensure I do no waste time looking for spaces or special chars which are useless for what I aim to do anyway.

             

            Event.Log shows nothing. Script runs through and seems to give expected results. Only thing that made me notice the issue was that sometimes I would get a script that would hang, and would refuse to disconnect even if I "forced-disconnect" it from the admin console. From that point (when the script hangs), the admin console is unable to fully close the DB on its own, even though it is able to disconnect other clients just as easily as before.

             

            It is after troubleshooting that I noticed the DMP file which I had never heard of before a couple of days ago. This is the only sign of an actual crash that I get, as the FM log files show nothing and the DB continues to perform as usual, even though there's that one script that is hanging there doing nothing that I can't disconnect...

             

            The script takes 15 minutes for now, but will probably take longer at some point as we keep adding more and more approval documents everyday.

            • 3. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
              TSGal

              lmagnan:

               

              Thank you for the additional information.

               

              When the script hung, was there any indication what records had been processed or what record the hang occurred?  After restarting the script engine (and/or server), are you able to run the script again with that same found set?  Since the database "continues to perform as usual", I'm trying to determine if the script engine ran out of memory or if the cause is data related.

               

              TSGal

              FileMaker, Inc.

              • 4. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                lmagnan

                TSGal,

                 

                I think I am getting all the matches I should be getting as the matches were all going to a common field in the record where the matches were related to. I made sure that the script took record locking into account, it basically loops trying to set the common field until the field is available. At the end of the scripts, I would get the right number of "match lines" compared to what was expected, but I can't know for sure if the script is delivering all the possible matches or just a section of it, stopping to look for matches when the scripting engine runs out of memory.

                 

                I also took a backup copy of the DB and ran a recovery against it, to see if there was any corruption or problems associated with the file, since some of these C0000005 exceptions seem to be related to corruption. The Recovery completed and did find some structure issues. However, they seemed unrelated to what I was doing and I think were negligible as they referred to Custom Themes (for which you have a KB article) and script folder markers (that one I couldn't find anything about...). The recovery operation also seems to create two empty tables when it starts looking at data to store recovered items, but they both end up empty. I tried running the recovery against a clone file and I got the exact same result.

                 

                I was indeed able to run the script again on the same found set after the restart, but as a matter of fact I didn't really need to restart the server as I could just start more instances of the same script without even restarting the server. It is the fact that I was looking at the admin console that made me notice what was happening. The scripts don't actually always hang, sometimes they'll go through, but they still produce DMP files for each and every search action they took (somewhere around 100 DMP files for every time I lunch the scripts). I think the scripts were hanging pretty consistently when I was running something like 5 or 6 at the same time... I was probably getting a little too optimistic for the amount of resources my server has available!

                 

                Basically, the bottom line is if I lunch one script on its own, I get no DMP file, no issues, and the same expected results. If I launch more than one, I get DMP files systematically, and scripts will sometimes hang, but not all the time. I do get a non-responsive admin console pretty consistently though.

                 

                Thanks for your support,

                 

                LM

                • 5. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                  TSGal

                  lmagnan:

                   

                  Thank you again for the additional information.

                   

                  A corrupt file may or may not be the cause of this issue, but make sure you fix the file.  Since you also got error messages when running Recover on the clone, run Recover again on the recovered clone file.  This will let you know if there is other damage in the file that FileMaker cannot fix.  Once you have a clean file, then import the data from the original file into the clean file.

                   

                  The c0000005 errors are access violations, so this may be caused by another user accessing the same record while changes are trying to be made.

                   

                  Since you mentioned "it basically loops trying to set the common field until the field is available", make sure a user hasn't edited a record and moved to some other task without committing the data.  This would cause your script to be in a loop until the user returns to the file, and it might appear this process has hung.  Therefore, you may want to put a timer or loop counter.  If this fails after "x" amount of time or iterations, make note of it in another table, and proceed to the next record and continue with the script.

                   

                  With the information you provided, I have sent everything (including the DMP file) to our Development and Testing departments for review.  When I receive any feedback, I will let you know.

                   

                  TSGal

                  FileMaker, Inc.

                  • 6. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                    lmagnan

                    TSGal,

                     

                    Thanks for the advice.

                     

                    I think I already know the answer to that one, but can I run the recover against a closed copy of the complete database instead? This way, I make sure the data is not corrupted as well, and I don't have to run a gigantic import...? I know it is probably not the best way to do this, but I have run the recovery against a backup copy of the database and I know there is no corruption in the data itself.

                     

                    However, I thought we were never supposed to re-use a recovered copy in production?

                     

                    I ran the recovery again, and I still get two errors:

                    8473

                    0

                    8476

                    8495

                     

                    0

                    0

                    0

                    0

                    0

                    0

                    0

                    Created table 'Recovered 3'

                    Recovering: theme 'com.filemaker.theme.custom.D804E372_9830_FD47_8EF0_0F711A1E453D' (2)

                    This item changed

                    WARNING: problems were detected while recovering the database.  The recovered file should NOT be used going forward; copy only the most recent work from it into a backup copy of the original file.

                    File blocks: scanned and rebuilt 2909 block(s), dropped 0 invalid data block(s)

                    Schema: scanned fields and tables; some problems were found...

                    fields created to match record data: 0

                    field values deleted due to invalid ID or repetition: 0

                    records deleted due to invalid ID: 0

                    Structure: scanned; 2 item(s) modified

                    File size after recovery is 11968512 bytes

                     

                    I will try the script again without writing to any fields later on, just running the calculations to see what happens, but if my memory serves me right, I wasn't working either as I think this was part of my first trials. I do get some 301 errors with the script, but I thought they were normal since the script was looping until the field became available for edit?

                     

                    What you are suggesting is that the script ends up accessing the memory even thought it hasn't quite been release by the other client?

                    • 7. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                      TSGal

                      lmagnan:

                       

                      Yes, you can run a Recover on the original file.  Since you got the error, and FileMaker cleaned up the file, the recovery does say "The recovered file should NOT be used going forward."  Make the corrections in the file, make sure everything looks fine, and then run another Recover on the recovered file.  The message from the second Recover may reveal more information about the file, and it may tell you it is okay to use the file.

                       

                      There's a history over the years that recovered files should never be used.  However, most of this is based on the results of the Recover.  Specifically, if the Recovery says to NOT use, then something in the file has been deleted, rebuilt, etc.  You should go back to your original file and save the data.  That was the reasoning for my initial post.

                       

                      However, if the Recovery found no errors, and the Recover.log file doesn't display any errors, then the recovered file is good to use.

                       

                      If the script is in a loop waiting for a user to commit the data, this will hold on to the current resources, and slow down other scripts trying to use the same resources.  That is the reason for trying to update the record, and if the record can't be accessed after "x" amount of time or iterations, log it and move to the next record.  This will ensure the script completes and frees up the resource for the next script.

                       

                      TSGal

                      FileMaker, Inc.

                      1 of 1 people found this helpful
                      • 8. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                        lmagnan

                        TSGal,

                         

                        Just to make sure were clear here before I start playing around with the recovery:

                        • I get 4 (structure) errors when I recover the file at first, no matter if I run against an empty clone or against the full file.
                        • If I run against the recovered file, again no matter clone or not, I still get two errors.

                         

                        Plan is to backup the full DB, use the backup to run the recover. Once I get my "new", full, recovered file (same one as before less two errors), I put it into service and take the "old" one offline.

                         

                        Am I getting this right? Just want to be a little cautious with the recovery thing there... Is it still worth doing the recovery even though I get the two errors I copied you in my previous message?

                         

                        I get your script point there, but I am not sure it all makes sense in my head. The idea behind the script was to enter a loop, set a field, commit, and then exit loop if no error was found on the last two steps. Basically if two scripts try to do the same thing at once, one is going to have to wait until the other one is done. At some point, they should get "out of phase" and have less 301 errors.

                         

                        That's the way I thought about it, but there's probably far more to understand behind the scenes that what I can think about!

                         

                        Thanks for all your help!

                        • 9. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                          TSGal

                          lmagnan:

                           

                          If you still get two errors after recovering the recovered file, then the file is definitely damaged, and you should fine a recent backup that doesn't show any errors when recovered.

                           

                          As far as the script, consider a client changes the value in a field and leaves the cursor in the field and goes to lunch, or goes home for the evening.  The record is currently locked for that client.  During that time, you decide to run the script.  As the script gets to that locked record, the script will continue looping, and this may be a long wait.  Does this help?

                           

                          TSGal

                          FileMaker Inc.

                          • 10. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                            TSGal

                            lmagnan:

                             

                            Our Testing department would like to get a copy of your file.  I have sent you a private message with instructions where to send the file.

                             

                            TSGal

                            FileMaker, Inc.

                            • 11. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                              TSGal

                              lmagnan:

                               

                              Private messaging is temporarily not working for me.  In essence, feel free to send a clone of the solution.  Be sure to include the steps generally taken that causes the issue.

                               

                              TSGal

                              • 12. Re: FMS Server 14 Crashes with multiple PSOS and Complex Searches
                                TSGal

                                lmagnan:

                                 

                                Development has looked again at your files, and noticed a crashed thread that may be causing a conflict.  Development mentioned that changes in FileMaker Server 15 should not permit this type of crash and ask that you download a trial version of FileMaker Server 15 to see if the issue occurs.  At this time, there are no plans to revisit the FileMaker Server 14 code.

                                 

                                TSGal

                                FileMaker, Inc.