1 2 Previous Next 17 Replies Latest reply on Jul 12, 2017 2:19 PM by TSGal

    FileMaker Server 13 crash - MSVCR110.dll

    Smef

      Hello,

       

      We're getting infrequent but very problematic crashes of FileMaker Server 13.0.10.1003. I'm not seeing anything triggering this, and the crashes seem to happen late at night when nobody is in the system. Has anyone seen this before, or know what may be causing it? There are no plugins installed on the server.

       

      Amazon t2.Large

      Windows Server 2012 R2 Standard x64

      8 GB RAM

      Xeon E-2676 v3

       

      Here's are the four events from the event viewer:

       

       

      Faulting application name: fmserver.exe, version: 13.0.10.1003, time stamp: 0x55529c1e

      Faulting module name: MSVCR110.dll, version: 11.0.50727.1, time stamp: 0x5011a017

      Exception code: 0xc0000409

      Fault offset: 0x0000000000073904

      Faulting process id: 0xff0

      Faulting application start time: 0x01d2ed65434185b9

      Faulting application path: C:\Program Files\FileMaker\FileMaker Server\Database Server\fmserver.exe

      Faulting module path: C:\Program Files\FileMaker\FileMaker Server\Database Server\MSVCR110.dll

      Report Id: 6bde5d9f-5c8b-11e7-814a-127003b6b617

      Faulting package full name:

      Faulting package-relative application ID:

       

      -------

      next event

      -------

       

       

      Fault bucket , type 0

      Event Name: BEX64

      Response: Not available

      Cab Id: 0

       

      Problem signature:

      P1: fmserver.exe

      P2: 13.0.10.1003

      P3: 55529c1e

      P4: MSVCR110.dll

      P5: 11.0.50727.1

      P6: 5011a017

      P7: 0000000000073904

      P8: c0000409

      P9: 0000000000000007

      P10:

       

      Attached files:

       

      These files may be available here:

      C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_fmserver.exe_956f4eb22d5d6148d65655ec2f242526cc798b_a129d647_1540ef75

       

      Analysis symbol:

      Rechecking for solution: 0

      Report Id: 6bde5d9f-5c8b-11e7-814a-127003b6b617

      Report Status: 4100

      Hashed bucket:

       

      --------

      next event

      --------

       

       

       

      Fault bucket , type 0

      Event Name: BEX64

      Response: Not available

      Cab Id: 0

       

      Problem signature:

      P1: fmserver.exe

      P2: 13.0.10.1003

      P3: 55529c1e

      P4: MSVCR110.dll

      P5: 11.0.50727.1

      P6: 5011a017

      P7: 0000000000073904

      P8: c0000409

      P9: 0000000000000007

      P10:

       

      Attached files:

       

      These files may be available here:

      C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_fmserver.exe_956f4eb22d5d6148d65655ec2f242526cc798b_a129d647_1540ef75

       

      Analysis symbol:

      Rechecking for solution: 0

      Report Id: 6bde5d9f-5c8b-11e7-814a-127003b6b617

      Report Status: 0

      Hashed bucket:

       

       

      -------

      next event

      -------

       

       

      Database Server process has terminated abnormally.

        • 1. Re: FileMaker Server 13 crash - MSVCR110.dll
          TSGal

          Smef:

           

          Thank you for your post.

           

          Can you be a little more specific on "infrequent"?  Once a week?  Once a month?

           

          Has this issue always occurred?  Or, has it just started fairly recently?

           

          Does the Event.log file show anything out of the ordinary near the time of the termination?

           

          Have you tried an uninstall and reinstall of FileMaker Server?  This would write all new files, including MSVCR110.dll.

           

          Any other information you can provide may be helpful in narrowing down possible causes.

           

          TSGal

          FileMaker, Inc.

          • 2. Re: FileMaker Server 13 crash - MSVCR110.dll
            Smef

            This seems to happen once every 1-2 months. It doesn't appear to be on any sort of schedule, unfortunately.

             

            We deployed the server a number of months ago, and seems to have been happening with the same infrequency the entire time of the deployment. The server is running on a t2.Large instance on AWS.

             

            The Event.log just says "2017-06-29 01:25:57.381 -0400 Error 701 FMS Database Server process has terminated abnormally."

             

            I had to uninstall and reinstall it once a while ago for an unrelated reason. The crashes don't seem more or less frequent before or after.

             

            One important thing which has happened each time, and why this is much more of a problem than just the server going down and needing to be turned back on, is that it appears that the FileMaker Server Script Engine seems to also hang or crash when this happens. The FMSE process doesn't come back up when the server is turned back on through the FMS Admin Console web interface. This is a problem because the client will turn the server back on through the Admin Console, but any scheduled scripts or PSOS script steps won't work properly.

             

            When PSOS is called it hits the script step, but no sessions or script is ever started on the server. The PSOS script step doesn't return any error either, and it thinks it has worked properly. This is a significant issue because we're unable to detect that PSOS hasn't worked correctly as FMP thinks it triggered that script step without an error. When users hit these script steps when submitting jobs they don't get an error message and think that the job has processed on the server correctly, when in fact it hasn't been processed at all.

             

            To resolve this issues of the FMSE not starting we either need to start the process through the command line by stopping and then starting the FMSE using "fmsadmin stop fmse" and "fmsadmin start fmse," or by restarting the machine completely, which is what we normally do to make sure that everything comes back up properly.

            • 3. Re: FileMaker Server 13 crash - MSVCR110.dll
              TSGal

              Smef:

               

              Thank you for the additional information.

               

              What scheduled scripts do you have scheduled on the server?  How often does it/they run?  What actions does it/they perform?

               

              What actions occur when you execute Perform Script on Server?  With the Perform Script on Server, is Wait for completion turned on?

               

              Do you have progressive backups enabled?  If so, how often do the backups run?

               

              Are there any messages in the Event.log that show backups (regular and progressive) starting but not completing?

               

              Looking at reported issues for FileMaker Server 13 that may apply, I do see one issue where a customer performs Save as PDF with a large data file under Windows, but our Testing department was unable to reproduce the issue.

               

              TSGal

              FileMaker, Inc.

              • 4. Re: FileMaker Server 13 crash - MSVCR110.dll
                Smef

                Until fairly recently we only had the default Daily and Weekly backup scripts running. We recently added a new nightly one which does an average of a set of records and records the result in another table. It's very fast and doesn't seem to be related to this crash which happened before we added the new scheduled script.

                 

                When a users steps through the PSOS script step it runs the script step (though the script doesn't actually run on the server) as if everything was fine and get(lasterror) returns 0. Wait is not enabled. I believe it will normally report an error with get(lasterror) if it's unable to trigger it to run at all, like in the case when you're using an offline file and there's no server for PSOS to run on.

                 

                Yes, progressive backups are enabled with the default 5 minutes. It shows the service starting and stopping in the log at times when we have restarted the server, but the individual progressive backups don't appear to be logged.

                 

                As far as regular backups and scripts go, this crash happened at 1:25 AM and we didn't have any schedules scheduled to run until 11pm when the backups run, so there aren't any attempts to run any schedules in the log. We manually started the FMSE process and then restarted the server before 11pm, so everything was already back up and running by the time they fired off.

                • 5. Re: FileMaker Server 13 crash - MSVCR110.dll
                  TSGal

                  Smef:

                   

                  Thank you for the information.

                   

                  After discussing your posts with another Support Technician, we would like to look at your crash dump (.dmp) files.  You will find these in the FileMaker Server Logs folder at:

                  C:\Program Files\FileMaker\FileMaker Server\Logs

                  I have sent you a private message with instructions where to send these files.

                   

                  TSGal

                  FileMaker, Inc.

                  • 6. Re: FileMaker Server 13 crash - MSVCR110.dll
                    Smef

                    There don't appear to be any dump files at that location.Screen Shot 2017-07-03 at 4.39.31 PM.png

                    • 7. Re: FileMaker Server 13 crash - MSVCR110.dll
                      TSGal

                      Smef:

                       

                      Since the crash appears to be "silent", you can safely force the output of a crash dump by modifying the registry with a .reg file.  I have sent files and instructions to our Technical Support contact, so you should receive an email from him shortly.  If you don't receive the files within one hour, please let me know, and I'll make sure the files are sent to you immediately.

                       

                      TSGal

                      FileMaker, Inc.

                      • 8. Re: FileMaker Server 13 crash - MSVCR110.dll
                        Smef

                        He sent the files which seem that they'll enable the dump the next time the crash happens, though the registry change won't force it to output anything right now. I probably need to restart the server for it to take effect as well.

                        • 9. Re: FileMaker Server 13 crash - MSVCR110.dll
                          TSGal

                          Smef:

                           

                          I received your email.  Thank you.

                           

                          What application did you use?  The "&" is a valid character in CMD.  In Windows, Run CMD.  You will then get the command prompt.  Enter the exact same line, and the information will be displayed in a Notepad document.  This may take a few minutes to complete.

                           

                          A restart of the server is not required.

                           

                          TSGal

                          FileMaker, Inc.

                          • 10. Re: FileMaker Server 13 crash - MSVCR110.dll
                            Smef

                            Ah - it was in PowerShell. It looks like it's working in Command Prompt.

                            • 11. Re: FileMaker Server 13 crash - MSVCR110.dll
                              TSGal

                              Smef:

                               

                              I did receive your latest email.  Thank you.

                               

                              As mentioned in your details, there are no FileMaker Server .dmp dump files on the server.  Therefore, run the command to update the registry.  The next time a crash occurs, check the Logs folder, or run the CMD statement again to verify the .dmp file is created.

                               

                              TSGal

                              FileMaker, Inc.

                              • 12. Re: FileMaker Server 13 crash - MSVCR110.dll
                                TSGal

                                Smef:

                                 

                                I received your .dmp file.  Thank you.

                                 

                                One of our developers requests that you use Windows Explorer to look at the properties of the following two files:

                                FileMaker Server/Database Server/DBEngine.DLL

                                FileMaker Server/Database Server/OmniORB4.DLL

                                 

                                The developer needs the information from both the General and Details panels for both .DLL files.

                                 

                                TSGal

                                FileMaker, Inc.

                                • 13. Re: FileMaker Server 13 crash - MSVCR110.dll
                                  Smef

                                  Here are the screenshots of those files.

                                   

                                  FYI - I reinstalled the server over the weekend. Originally the server was installed with a 13.0.1 installer and then upgraded to the latest version. The updater files were having issued running after the reinstall, so I removed FMS again and downloaded a fresh installer to install 13.0.10 directly.

                                   

                                  DBEngine-details.png

                                  DBEngine-properties.png

                                  OmniORB4-details.png

                                  OmniORB4-properties.png

                                  • 14. Re: FileMaker Server 13 crash - MSVCR110.dll
                                    TSGal

                                    Smef:

                                     

                                    Thanks for the screen shots and information.  I have sent this back to the tester and developer for review.

                                     

                                    TSGal

                                    FileMaker, Inc.

                                    1 2 Previous Next