10 Replies Latest reply on May 9, 2017 12:32 PM by jonathanofik

    Database Server process has terminated abnormally

    jonathanofik

      I have installed FileMaker Server 15.0.3 on Windows Server 2008 R2 machine. This has happened about 30 times, sometimes it's more than once a day

       

      Event Log

       

      2017-04-19 15:09:23.130 -0500    Error    701    FM15    Database Server process has terminated abnormally.

        • 1. Re: Database Server process has terminated abnormally
          mikebeargie

          There are numerous things that could cause this, layout corruption and independent java updates are two possible causes.

           

          Have you run java updates on the server outside of the filemaker updaters?

           

          Also, what does the windows application log say about what crashed (not the filemaker log, but the system/OS log)?

          • 2. Re: Database Server process has terminated abnormally
            wimdecorte

            Make sure you enable the 'top call stats log' whenever you restart FMS, that may give you additional clues as to what the users are doing at the time of the crashes.  Also make sure the regular 'usage statistics' log is turned on.

            Look for performance bottlenecks.  Look for DMP files in the FMS logs folder; if there are any, open a support ticket with FMI.

            • 3. Re: Database Server process has terminated abnormally
              jonathanofik

              Mike Beargie wrote:

               

              There are numerous things that could cause this, layout corruption and independent java updates are two possible causes.

               

              Have you run java updates on the server outside of the filemaker updaters?

               

              Also, what does the windows application log say about what crashed (not the filemaker log, but the system/OS log)?

               

              We checked for layout corruption 2 weeks ago and fixed that. Today, we run undependent java updates after Filemaker crashing again.

               

              We checked the Crash dumps and we got this from the debugger:

               

               

              FAULTING_IP: 
              ntdll!RtlFreeHeap+d0
              00000000`77ca32d0 4c8b6308        mov     r12,qword ptr [rbx+8]
              
              EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
              ExceptionAddress: 0000000077ca32d0 (ntdll!RtlFreeHeap+0x00000000000000d0)
                 ExceptionCode: c0000005 (Access violation)
                ExceptionFlags: 00000000
              NumberParameters: 2
                 Parameter[0]: 0000000000000000
                 Parameter[1]: 000003c38c05f438
              Attempt to read from address 000003c38c05f438
              
              CONTEXT:  0000000000000000 -- (.cxr 0x0;r)
              rax=000000005cac5050 rbx=0000000000000000 rcx=0000000000000030
              rdx=0000000000000000 rsi=00000000ffffffff rdi=0000000000000ad0
              rip=0000000077ca135a rsp=000000005825c908 rbp=0000000000009670
               r8=000000005825c960  r9=000000005825c478 r10=0000000000000000
              r11=0000000000000020 r12=0000000000000000 r13=000000005825ea30
              r14=000000005825d030 r15=000000000035af3a
              iopl=0         nv up ei pl zr na po nc
              cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
              ntdll!ZwWaitForSingleObject+0xa:
              00000000`77ca135a c3              ret
              
              PROCESS_NAME:  fmserver.exe
              
              ERROR_CODE: (NTSTATUS) 0xc0000005 - La instrucci n en 0x%08lx hace referencia a la memoria en 0x%08lx. La memoria no se pudo %s.
              
              EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - La instrucci n en 0x%08lx hace referencia a la memoria en 0x%08lx. La memoria no se pudo %s.
              
              EXCEPTION_PARAMETER1:  0000000000000000
              
              EXCEPTION_PARAMETER2:  000003c38c05f438
              
              READ_ADDRESS:  000003c38c05f438 
              
              • 4. Re: Database Server process has terminated abnormally
                wimdecorte

                Any plugins that you use on FMS?  Scripts that use plugins that you call through PSoS or server-side schedules?

                • 5. Re: Database Server process has terminated abnormally
                  jonathanofik

                  Make sure you enable the 'top call stats log' whenever you restart FMS, that may give you additional clues as to what the users are doing at the time of the crashes.  Also make sure the regular 'usage statistics' log is turned on.

                  Look for performance bottlenecks.  Look for DMP files in the FMS logs folder; if there are any, open a support ticket with FMI.

                  Event.log

                   

                  2017-05-08 12:20:38.710 -0500    Warning    661    FM15    SECURITY: Client "gramirez (admin’s MacBook Pro) [10.30.30.118]" authentication failed on database "Produccion.fmp12" using "Admin [fmapp]".
                  2017-05-08 12:20:42.267 -0500    Warning    661    FM15    SECURITY: Client "gramirez (admin’s MacBook Pro) [10.30.30.118]" authentication failed on database "Produccion.fmp12" using "gramirez [fmapp]".
                  2017-05-08 12:41:33.982 -0500    Error    701    FM15    Database Server process has terminated abnormally.
                  2017-05-08 12:48:08.912 -0500    Information    745    FM15    Stopping FileMaker Server processes...
                  2017-05-08 12:48:08.912 -0500    Information    704    FM15    Stopping FileMaker Script Engine process..
                  
                  

                   

                   

                  I only have the Stats.log

                  Stats.log

                  2017-05-08 12:38:43.349 -0500    4    121    140    19    100    0    30    29    0    0    0    26    1    39289    1079    135    02017-05-08 12:39:13.208 -0500    5    10    255    65    100    0    30    29    0    0    0    45    0    12189    63    28    0
                  2017-05-08 12:39:43.097 -0500    96    33    17598    287    99    0    30    29    0    0    0    38    2    15847    2332    215    0
                  2017-05-08 12:40:13.049 -0500    101    123    813    273    100    0    30    29    0    0    0    21    1    55747    3354    21    0
                  2017-05-08 12:40:43.188 -0500    103    47    9    289    100    0    30    29    0    0    0    66    0    16845    956    4    0
                  2017-05-08 12:41:14.014 -0500    26    76    41    60    100    0    30    29    0    0    0    20    4    4584    343    2    0
                  2017-05-08 12:49:43.458 -0500    0    0    470    1    79    0    0    29    0    0    0    0    0    0    0    0    0
                  2017-05-08 12:50:13.379 -0500    0    0    8915    0    54    0    0    29    0    0    0    0    0    4044    26    4    0
                  2017-05-08 12:50:43.238 -0500    0    0    37998    1    65    0    0    29    0    0    0    0    0    29364    0    2609    0
                  2017-05-08 12:51:13.096 -0500    0    0    10490    1    52    0    0    29    0    0    0    0    0    972    71    0    0
                  2017-05-08 12:51:43.485 -0500    0    0    0    0    50    0    0    29    0    0    0    0    0    0    0    0    0
                  2017-05-08 12:52:13.343 -0500    0    0    46786    0    54    0    0    29    0    0    0    0    0    0    0    0    0
                  2017-05-08 12:52:43.202 -0500    0    8    396340    0    54    0    1    29    0    0    0    1    0    3137    0    2869    0
                  2017-05-08 12:53:13.060 -0500    1    72    265    3    96    0    0    29    0    0    0    6    0    41240    40    26820    0
                  

                  Export it to Excel to see it better

                   

                  Screenshot 2017-05-08 17.56.24.png

                  • 6. Re: Database Server process has terminated abnormally
                    wimdecorte

                    You have to manually enable the top call stats log whenever you start FMS, it turns itself off.

                     

                    Your server did not seem under a lot of stress at the time of the crash, but clearly at least one client (or schedule) was doing something really intensive given the amount of network and disk activity.

                    But without the top call stats log it is impossible to say what was going on.

                     

                    Something to check: any backups running at the time of the crash?

                    • 7. Re: Database Server process has terminated abnormally
                      jonathanofik

                      You have to manually enable the top call stats log whenever you start FMS, it turns itself off.

                       

                      Your server did not seem under a lot of stress at the time of the crash, but clearly at least one client (or schedule) was doing something really intensive given the amount of network and disk activity.

                      But without the top call stats log it is impossible to say what was going on.

                       

                      Something to check: any backups running at the time of the crash?

                      We already enabled it but some day  the machine crashed and restarted itself,  but the log was disabled after the crash by the server itself. and we didn't knew it, but three days ago that we had to re enable it (the top call stats log).

                      • 8. Re: Database Server process has terminated abnormally
                        jonathanofik

                        Any plugins that you use on FMS?  Scripts that use plugins that you call through PSoS or server-side schedules?

                         

                        Yes we are  using the MBS plugin  by MonkeyBread Software.

                        Monkeybread Software - MBS FileMaker Plugin: index

                         

                        and nowadays we're using the plugin version:

                        MBS Plug-In;6.1.0.6

                         

                        And although  we have scripts running at the server's side. We aren't  running schedulers that  perform actions at

                        certain times of the day.

                        • 9. Re: Database Server process has terminated abnormally
                          james_quiggins

                          Hi Jonathanofik,

                           

                          Would you be willing to share the .dmp in a downloadable format?

                           

                          Thanks

                          • 10. Re: Database Server process has terminated abnormally
                            jonathanofik

                            These are the last 2 dumps

                            I uploaded them to google drive

                             

                            DMP FM15 - Google Drive