13 Replies Latest reply on Oct 24, 2016 10:18 AM by TSGal

    FMSE crashes intermittently

    DamianKelly

      Product and version FileMaker Server 15.0.1.137

      OS and version OS X 10.11.5

      Hardware  Mac Pro (Late 2013) 2.7GHz 12 Core Xeon 64GB RAM

      Description FMSE crashes randomly every 2-3 days

      How to replicate  Unable to predict exactly when

       

       

       

      We are seeing the FMSE crashing:

      Oct 10, 2016 11:07:51 AM

      Error

      FileMaker Script Engine process has terminated abnormally.

      from the filemaker log and

      10/10/2016 11:07:52.182 ReportCrash[16966]: Saved crash report for fmsased[89321] version 0 to /Library/Logs/DiagnosticReports/fmsased_2016-10-10-110752_TOT-PRODUCTION-SERVER.crash

      from the system log. I have attached the crash report.

       

      We make fairly heavy use of PSoS. The PSoS scripts use a plugin, the DoSQL plugin from FM Butler. There are a couple of possible scripts that are run on the server but all execute the vast majority of the time successfully. I am not seeing any errors in the scripts with regard to server side compatibility.

        • 1. Re: FMSE crashes intermittently
          TSGal

          DamianKelly:

           

          Thank you for your post and crash report.

           

          In the crash report, I see reference to authentication and buffering, which generally means you are opening a file.  Since you mentioned the DoSQL plug-in, are you connecting to an external SQL data source?  If so, is this a MySQL, SQL Server, or Oracle data source?

           

          Do you know at what point the crash occurs in the script?

           

          Although you mention "FMSE crashes randomly every 2-3 days", how often does the script run during the day?

           

          Do you notice how much memory is being used after restarting and after the crash occurs?  Specifically, I'm looking to see if memory is decreasing each time the script is run.

           

          Is the DoSQL plug-in essential for the script?  Is it possible to run the script without the plug-in enabled?

           

          Any other information you can provide may be helpful.

           

          TSGal

          FileMaker, Inc.

          • 2. Re: FMSE crashes intermittently
            DamianKelly

            Hi TSGal,

             

            The guys at SH Partners have asked me to update the plugin to the latest version so I will do this between the day shift and night shift tonight. I am usually pretty conservative with updates, so only apply them when I see a benefit.

             

            It might be worth waiting to see if this helps although there isnt anything specific in the version notes for the plugin. Maybe the "Fixed an Issue with 64bit" will help. We were on 2.0.0.26 before which looks like it was still 32bit.

            http://wiki.myfmbutler.com/index.php/DoSQL_2#Version_History

             

            I almost put we have not noticed any memory or load related issues in my original post. I didnt because the evidence was anecdotal rather than from a log.

             

            The DoSQL plugin updates the FileMaker file itself. It is essential for the script,  we us DoSQL, combined with ExecuteSQL, because the data necessary to work out what we are trying to achieve comes from a variety of tables and its easier to generate list of joblines in SQL than it is in FileMaker finds.

             

            There are two UPDATE statements generated like this:

             

            Let([

            tbl_JobLineStatus = SQL_TableName ( JobLineStatus::ID_JobLineStatus );

             

            fld_ID_JobLine = SQL_FieldName_Unqualified ( JobLineStatus::ID_JobLine );

            fld_d_Status_15_stock_AvailableToAllocate_tern = SQL_FieldName_Unqualified ( JobLineStatus::d_Status_15_stock_AvailableToAllocate_tern );

            fld_d_Status_20_stock_Allocated_tern = SQL_FieldName_Unqualified ( JobLineStatus::d_Status_20_stock_Allocated_tern );

            fld_x_NUM_quantity_toBeAllocated = SQL_FieldName_Unqualified ( JobLineStatus::x_NUM_quantity_toBeAllocated )

             

            ];

            "UPDATE " & tbl_JobLineStatus & "

            SET " & fld_d_Status_15_stock_AvailableToAllocate_tern & " = 1

            WHERE " & fld_ID_JobLine & " IN (" & Substitute ( $id_jobline_list; ¶ ; ",") & ")

            AND " & fld_d_Status_20_stock_Allocated_tern & " < 2

            AND " & fld_x_NUM_quantity_toBeAllocated & " > " & $d_quantity

            )

             

            and

             

            Let([

            tbl_JobLineStatus = SQL_TableName ( JobLineStatus::ID_JobLineStatus );

             

            fld_ID_JobLine = SQL_FieldName_Unqualified ( JobLineStatus::ID_JobLine );

            fld_d_Status_15_stock_AvailableToAllocate_tern = SQL_FieldName_Unqualified ( JobLineStatus::d_Status_15_stock_AvailableToAllocate_tern );

            fld_d_Status_20_stock_Allocated_tern = SQL_FieldName_Unqualified ( JobLineStatus::d_Status_20_stock_Allocated_tern );

            fld_x_NUM_quantity_toBeAllocated = SQL_FieldName_Unqualified ( JobLineStatus::x_NUM_quantity_toBeAllocated )

             

            ];

            "UPDATE " & tbl_JobLineStatus & "

            SET " & fld_d_Status_15_stock_AvailableToAllocate_tern & " = 2

            WHERE " & fld_ID_JobLine & " IN (" & Substitute ( $id_jobline_list; ¶ ; ",") & ")

            AND " & fld_d_Status_20_stock_Allocated_tern & " < 2

            AND " & fld_x_NUM_quantity_toBeAllocated & " <= " & $d_quantity"

            )

             

            The field JobLineStatus::d_Status_20_stock_Allocated_tern only has values 0f 0, 1 and 2 so I am wondering if using IN (0, 1) would be quicker.

             

            The rest of the script is pretty straight forward, It basically calls a few subscripts each one returning either a value or a list of values. I would have uploaded a PDF but the image upload doesnt seem to like that format.....

            • 3. Re: FMSE crashes intermittently
              TSGal

              DamianKelly:

               

              Thank you for the additional information.  Since the SQL is integral to the script, there is probably no need to see your entire script.

               

              Please keep me updated after the DoSQL update.  I realize it takes a few days before a crash occurs, so after installing the update, look to see how memory is allocated.  If the crash occurs, look again to see how memory is allocated as this may help pinpoint the cause.

               

              TSGal

              FileMaker, Inc.

              1 of 1 people found this helpful
              • 4. Re: FMSE crashes intermittently
                DamianKelly

                Thanks TSGal,

                 

                Ok the memory definitely climbs over time it was 110MB when I started it yesterday at 11AM, 250MB when I left at 5:30 to over 350MB this morning. However we are at our busiest first thing in the morning and last thing at night, so it could be that we are using it a lot. This morning the fmserverd process was over 260% and fmsased was between 0 and 140% I restarted the FMSE between the shifts this morning and will watch what happens to it during the day.

                 

                I have talked to Peter at SH Partners, he doesnt think its the plugin and has asked the same questions as you. He suggested I reduce any query size. So I am going to optimise the UPDATE today. At the moment it updates about 180 ish records a time and is called pretty often. I think I can reduce that number so will see of that helps.

                 

                Damian

                • 5. Re: FMSE crashes intermittently
                  DamianKelly

                  It crashed again. The memory definitely grows. It looks like somewhere above 348MB the process crashes.

                   

                  For the moment I have turned off the PSoS and returned it to the clients side. Its slower but solid. The issue is we need to look at a lot of records to work out what to update, so having it happen on the server reduced my network traffic.

                   

                  I am now wondering about trying to make a queue structure and a robot, plus go through the whole thing and make absolutely sure nothing superfluous is happening.

                  • 6. Re: FMSE crashes intermittently
                    TSGal

                    DamianKelly:

                     

                    Thank you for the additional information and your observations.

                     

                    I have sent all information, including the crash report, to our Development and Testing department for review.  When I receive any feedback, I will let you know.

                     

                    TSGal

                    FileMaker, Inc.

                    1 of 1 people found this helpful
                    • 7. Re: FMSE crashes intermittently
                      DamianKelly

                      Thanks, sorry I can not narrow it down further.

                       

                      We can not recreate the problem on our dev or test systems but neither of those come anywhere near the load of the production.

                       

                      Damian

                      • 8. Re: FMSE crashes intermittently
                        TSGal

                        DamianKelly:

                         

                        Our Testers would like more information about the plug-in. 

                         

                        Is this crashing issue new to FileMaker Server 15?  That is, did you previously have FileMaker Server 14 (or earlier) with the plug-in enabled?

                         

                        Please have myFMButler contact me via private messaging as Testing wants to make sure the plug-in was compiled and built with the FIleMaker Server 15 SDK.  Testing would also like to get the DoSQL plug-in and its source code.

                         

                        TSGal

                        FileMaker, Inc.

                        • 9. Re: FMSE crashes intermittently
                          DamianKelly

                          We have been on server 15.0.1 for quite a while with the plugin and with a few PSoS scripts. We increased the load we placed on the server in an attempt to reduce some network and client load. That was the change. It actually worked quite well up to a point but our wait times on the server climbed a lot. I think, because we removed the constraint of individual clients doing one of these updates at a time, we started log jamming the server. The memory use of the script engine climbed, although I only specifically observed the last time we let the process crash.

                           

                          With the scripts performing locally the server seems to be less busy and the script engine seems fine.

                           

                          The scripts are very heavily used as they control the availability status of all our SKUs. So almost any activity in the factory will cause the script to run. The script looks at the quantity we have in stock and then updates the joblines as to whether they can be picked and allocated. The result of all this is we were seeing maybe 60 PSoS sessions on the server.

                           

                          I have emailed Peter at myFMButler with a link to this thread and asked him to contact you directly.

                           

                          If you need any logs or anything else let me know.

                          Thanks

                          Damian

                          • 10. Re: FMSE crashes intermittently
                            TSGal

                            DamianKelly:

                             

                            Thank you for the additional information.

                             

                            Yes, please send in all of the FileMaker Server log files as well as the tomcat files.  Also include one of the most recent crash reports, as this will tell me the time of the crash, and I can look in the logs to see what occurred just prior to the crash.

                             

                            I am temporarily unable to send a private message, so an email from our Technical Support contact is being sent to you with instructions where to send the files.  If you don't receive an email within 60 minutes, please let me know, and I'll make sure the email is sent immediately.

                             

                            TSGal

                            FileMaker, Inc.

                            • 11. Re: FMSE crashes intermittently
                              Peter Wagemans

                              Hi Damian,

                               

                              The plug-in leaks memory, I noticed that while testing a few 100,000 records. That was just on my client, and I guess it also plays on fmse. I definitely have to fix that. It's not causing any crashes I believe, but it would be a good idea to restart fmse in an overnight shedule so the memory gets freed. I'm working together with our c++ guru Kris here to get the code play nicer and free up these small chunks of memory. I'm actually playing with the new SDK that FileMaker is making avaiable, and a lot of code is being optimized to work with the new data types they introduce.

                              This should result in a new generation plug-in, but it will also give me enough insights to fix these issue and do a (last) 2.2 release.

                              So for the time being, it is a very good idea to restart fmse automatically every night to keep the memory from building up. But it is not the reason for the crashes I think.

                              PS: I hope you're not using the SaveQuery function without using DeleteQuery afterwards. That would cause massive memory leaks of course.

                              1 of 1 people found this helpful
                              • 12. Re: FMSE crashes intermittently
                                DamianKelly

                                Thanks Peter. For the moment I have moved everything back onto client so we are okay for the moment.

                                 

                                We have a few more records, probably 500000 plus. Although our query should only update a few records at a time.The only function I use in the plug in is the one that executes SQL. I don't use any of the extras.

                                 

                                I can test any new version on our development and test server if you like

                                 

                                 

                                Thanks

                                Damian

                                • 13. Re: FMSE crashes intermittently
                                  TSGal

                                  DamianKelly:

                                   

                                  I received your server and web log files.  Thank you.

                                   

                                  The files have been sent to Development and Testing for review.  They will check to see if there is any memory leak caused by FileMaker.  Continue working with Peter Wagemans to hopefully find a resolution.

                                   

                                  TSGal

                                  FileMaker, Inc.

                                  1 of 1 people found this helpful