3 Replies Latest reply on Jan 23, 2013 3:36 PM by svend

    FM 11 Advanced Server crashing (heap_corruption mentioned in the dump file)

    svend

      We've got a virtual machine running FM Server Advanced, v11.03.309. Yesterday evening, the server process (but not the admin console) crashed; when the server was restarted, one of the files was deemed to be corrupt, and had to be recovered. Once recovered, the day's changes were copied into the previous day's backup, and that composite file with put back on the server. We did not reboot the box, or do anything else to any of the other files being served.

       

      The next day, in the morning, the server crashed again. This time, none of the files reported any corruption.

       

      There's nothing in the FileMaker logs in either case to indicate the problem -- they go straight from people logging in and out, to restarting the server/reopening the database files.

       

      Using the Windows debugger, the crash files talk about the faulting module being called "heap_corruption"; I'll include the full output below. Our Systems team reports that they don't see anything unusual on the filer/network/etc end.

       

      Tonight, we're going to reboot the virtual machine, clone all the 40-odd files (and re-import their data), and replace the existing files with their clones. Is there anything else obvious we should do.

       

      The web server isn't running, as we do not provide any web services.

       

      The virtual machine is running Windows Server 2008 R2 Standard with Service Pack 1. It's running as a 4-core 2.70GHz machine with 24GB of ram; I believe that this memory is actually reserved for it on the underlying box. This box provides no other services. The virtual machine lives in VMWare VSphere 5, on top of a 6 node cisco UCS cluster, with its back end datastores on NetApp. There's no evidence that it's starved for resources.

       

      The crash report for the first crash said:

       

       

      Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64

      Copyright (c) Microsoft Corporation. All rights reserved.

       

       

      Loading Dump File [C:AppCrash_fmserver.exe_44bfcdd01d5652a9774efea35dbf3021eed4_cab_710e130eWER3CD.tmp.hdmp]

      User Mini Dump File: Only registers, stack and portions of memory are available

       

      Symbol search path is: *** Invalid ***

      ****************************************************************************

      * Symbol loading may be unreliable without a symbol search path. *

      * Use .symfix to have the debugger choose a symbol path. *

      * After setting your symbol path, use .reload to refresh symbol locations. *

      ****************************************************************************

      Executable search path is:

      Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x86 compatible

      Product: Server, suite: TerminalServer SingleUserTS

      Machine Name:

      Debug session time: Mon Jan 21 18:17:12.000 2013 (UTC + 13:00)

      System Uptime: 23 days 20:31:51.059

      Process Uptime: 0 days 14:15:26.000

      ................................................................

      .............................

      Loading unloaded module list

      .......

      This dump file has an exception of interest stored in it.

      The stored exception information can be accessed via .ecxr.

      (680.dfc): Access violation - code c0000005 (first/second chance not available)

      eax=00000000 ebx=3d7cee08 ecx=00000000 edx=00000000 esi=00000002 edi=00000000

      eip=7750013d esp=3d7cedb8 ebp=3d7cee54 iopl=0 nv up ei pl zr na pe nc

      cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246

      *** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -

      ntdll!ZwWaitForMultipleObjects+0x15:

      7750013d 83c404 add esp,4

      0:100> !analyze -v

       

       

      GetPageUrlData failed, server returned HTTP status 404

      URL requested: http://watson.microsoft.com/StageOne/fmserver_exe/11_0_1_86/4b6b3007/ntdll_dll/6_1_7601_17725/4ec49b8f/c0000005/000343d0.htm?Retriage=1

       

      FAULTING_IP:

      ntdll!RtlImageNtHeader+124c

      775143d0 8b4904 mov ecx,dword ptr [ecx+4]

       

      EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)

      ExceptionAddress: 775143d0 (ntdll!RtlImageNtHeader+0x0000124c)

      ExceptionCode: c0000005 (Access violation)

      ExceptionFlags: 00000000

      NumberParameters: 2

      Parameter[0]: 00000000

      Parameter[1]: 00000004

      Attempt to read from address 00000004

       

      PROCESS_NAME: fmserver.exe

       

      FAULTING_MODULE: 757a0000 kernel32

       

      DEBUG_FLR_IMAGE_TIMESTAMP: 0

       

      MODULE_NAME: heap_corruption

       

      ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

       

      EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

       

      EXCEPTION_PARAMETER1: 00000000

       

      EXCEPTION_PARAMETER2: 00000004

       

      READ_ADDRESS: 00000004

       

      FOLLOWUP_IP:

      ntdll!RtlImageNtHeader+124c

      775143d0 8b4904 mov ecx,dword ptr [ecx+4]

       

      MOD_LIST: <ANALYSIS/>

       

      ADDITIONAL_DEBUG_TEXT:

      Use '!findthebuild' command to search for the target build information.

      If the build information is available, run '!findthebuild -s ; .reload' to set symbol path and load symbols. ; Enable Pageheap/AutoVerifer

       

      FAULTING_THREAD: 00000dfc

       

      DEFAULT_BUCKET_ID: HEAP_CORRUPTION

       

      PRIMARY_PROBLEM_CLASS: HEAP_CORRUPTION

       

      BUGCHECK_STR: APPLICATION_FAULT_HEAP_CORRUPTION_NULL_CLASS_PTR_DEREFERENCE_NULL_POINTER_READ_INVALID_POINTER_READ_WRONG_SYMBOLS

       

      LAST_CONTROL_TRANSFER: from 775135a7 to 775143d0

       

      STACK_TEXT:

      WARNING: Stack unwind information not available. Following frames may be wrong.

      3d7cf464 775135a7 010d0000 47847568 3d7cf52c ntdll!RtlImageNtHeader+0x124c

      3d7cf55c 77513492 47847568 47847570 47847570 ntdll!RtlImageNtHeader+0x423

      3d7cf57c 757b14dd 010d0000 00000000 47847570 ntdll!RtlImageNtHeader+0x30e

      3d7cf590 74a63c1b 010d0000 00000000 47847570 kernel32!HeapFree+0x14

      3d7cf5dc 005429cb 47847570 47847570 47847570 msvcr90!free+0xcd

      3d7cf5f0 00542b02 47847570 361ddf3c 00000017 OmniORB4+0x429cb

      3d7cf608 00542e78 ffffffff 005abfd0 0000036a OmniORB4+0x42b02

      3d7cf6ac 00536122 01349610 3d7cf6c4 00091141 OmniORB4+0x42e78

      3d7cf6c0 0053676c 361ddf3c 00000000 01349610 OmniORB4+0x36122

      3d7cf6e4 005271c8 361ddf3c 00536de0 00000000 OmniORB4+0x3676c

      3d7cf784 00545918 00000001 00000001 73ec0621 OmniORB4+0x271c8

      3d7cf85c 005452b3 3d7cf878 3d7cf884 3d7cf884 OmniORB4+0x45918

      3d7cf86c 0054535d 39deb170 005acb04 39deb170 OmniORB4+0x452b3

      3d7cf884 0054b986 00000000 00000000 2d54c190 OmniORB4+0x4535d

      3d7cf8f0 0054ac53 3d7cf924 73b14a0b 3d7cf918 OmniORB4+0x4b986

      3d7cf900 704c04e0 0054ac43 3d7cf924 3d7cf924 OmniORB4+0x4ac53

      3d7cf918 0054bb5d 32864870 005ad6d8 32864870 DBEngine+0x404e0

      3d7cf930 00091e5f 00000000 3d7cf988 74a238b3 OmniORB4+0x4bb5d

      3d7cf950 74a23433 32864870 58bb0a21 00000000 OmniThread+0x1e5f

      3d7cf988 74a234c7 00000000 3d7cf9a0 757b339a msvcr90!endthreadex+0x44

      3d7cf994 757b339a 2d54c190 3d7cf9e0 77519ef2 msvcr90!endthreadex+0xd8

      3d7cf9a0 77519ef2 2d54c190 3de8048d 00000000 kernel32!BaseThreadInitThunk+0x12

      3d7cf9e0 77519ec5 74a2345e 2d54c190 ffffffff ntdll!RtlInitializeExceptionChain+0x63

      3d7cf9f8 00000000 74a2345e 2d54c190 00000000 ntdll!RtlInitializeExceptionChain+0x36

       

       

      SYMBOL_NAME: heap_corruption!heap_corruption

       

      FOLLOWUP_NAME: MachineOwner

       

      IMAGE_NAME: heap_corruption

       

      STACK_COMMAND: ~100s; .ecxr ; kb

       

      FAILURE_BUCKET_ID: HEAP_CORRUPTION_c0000005_heap_corruption!heap_corruption

       

      BUCKET_ID: APPLICATION_FAULT_HEAP_CORRUPTION_NULL_CLASS_PTR_DEREFERENCE_NULL_POINTER_READ_INVALID_POINTER_READ_WRONG_SYMBOLS_heap_corruption!heap_corruption

       

      WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/fmserver_exe/11_0_1_86/4b6b3007/ntdll_dll/6_1_7601_17725/4ec49b8f/c0000005/000343d0.htm?Retriage=1

       

      Followup: MachineOwner

      ---------

       

      Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64

      Copyright (c) Microsoft Corporation. All rights reserved.

       

       

      Loading Dump File [C:AppCrash_fmserver.exe_44bfcdd01d5652a9774efea35dbf3021eed4_cab_710e130eWER1274.tmp.mdmp]

      User Mini Dump File: Only registers, stack and portions of memory are available

       

      Symbol search path is: *** Invalid ***

      ****************************************************************************

      * Symbol loading may be unreliable without a symbol search path. *

      * Use .symfix to have the debugger choose a symbol path. *

      * After setting your symbol path, use .reload to refresh symbol locations. *

      ****************************************************************************

      Executable search path is:

      Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x86 compatible

      Product: Server, suite: TerminalServer SingleUserTS

      Machine Name:

      Debug session time: Mon Jan 21 18:17:15.000 2013 (UTC + 13:00)

      System Uptime: 23 days 20:31:52.075

      Process Uptime: 0 days 14:15:29.000

      ................................................................

      .............................

      Loading unloaded module list

      .......

      This dump file has an exception of interest stored in it.

      The stored exception information can be accessed via .ecxr.

      (680.dfc): Access violation - code c0000005 (first/second chance not available)

      eax=00000000 ebx=3d7cee08 ecx=00000000 edx=00000000 esi=00000002 edi=00000000

      eip=7750013d esp=3d7cedb8 ebp=3d7cee54 iopl=0 nv up ei pl zr na pe nc

      cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246

      *** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -

      ntdll!ZwWaitForMultipleObjects+0x15:

      7750013d 83c404 add esp,4

      0:100> !analyze -v

       

      FAULTING_IP:

      ntdll!RtlImageNtHeader+124c

      775143d0 8b4904 mov ecx,dword ptr [ecx+4]

       

      EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)

      ExceptionAddress: 775143d0 (ntdll!RtlImageNtHeader+0x0000124c)

      ExceptionCode: c0000005 (Access violation)

      ExceptionFlags: 00000000

      NumberParameters: 2

      Parameter[0]: 00000000

      Parameter[1]: 00000004

      Attempt to read from address 00000004

       

      PROCESS_NAME: fmserver.exe

       

      FAULTING_MODULE: 757a0000 kernel32

       

      DEBUG_FLR_IMAGE_TIMESTAMP: 0

       

      MODULE_NAME: heap_corruption

       

      ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

       

      EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

       

      EXCEPTION_PARAMETER1: 00000000

       

      EXCEPTION_PARAMETER2: 00000004

       

      READ_ADDRESS: 00000004

       

      FOLLOWUP_IP:

      ntdll!RtlImageNtHeader+124c

      775143d0 8b4904 mov ecx,dword ptr [ecx+4]

       

      MOD_LIST: <ANALYSIS/>

       

      ADDITIONAL_DEBUG_TEXT:

      Use '!findthebuild' command to search for the target build information.

      If the build information is available, run '!findthebuild -s ; .reload' to set symbol path and load symbols. ; Enable Pageheap/AutoVerifer

       

      FAULTING_THREAD: 00000dfc

       

      DEFAULT_BUCKET_ID: HEAP_CORRUPTION

       

      PRIMARY_PROBLEM_CLASS: HEAP_CORRUPTION

       

      BUGCHECK_STR: APPLICATION_FAULT_HEAP_CORRUPTION_NULL_CLASS_PTR_DEREFERENCE_NULL_POINTER_READ_INVALID_POINTER_READ_WRONG_SYMBOLS

       

      LAST_CONTROL_TRANSFER: from 775135a7 to 775143d0

       

      STACK_TEXT:

      WARNING: Stack unwind information not available. Following frames may be wrong.

      3d7cf464 775135a7 010d0000 47847568 3d7cf52c ntdll!RtlImageNtHeader+0x124c

      3d7cf55c 77513492 47847568 47847570 47847570 ntdll!RtlImageNtHeader+0x423

      3d7cf57c 757b14dd 010d0000 00000000 47847570 ntdll!RtlImageNtHeader+0x30e

      3d7cf590 74a63c1b 010d0000 00000000 47847570 kernel32!HeapFree+0x14

      3d7cf5dc 005429cb 47847570 47847570 47847570 msvcr90!free+0xcd

      3d7cf5f0 00542b02 47847570 361ddf3c 00000017 OmniORB4+0x429cb

      3d7cf608 00542e78 ffffffff 005abfd0 0000036a OmniORB4+0x42b02

      3d7cf6ac 00536122 01349610 3d7cf6c4 00091141 OmniORB4+0x42e78

      3d7cf6c0 0053676c 361ddf3c 00000000 01349610 OmniORB4+0x36122

      3d7cf6e4 005271c8 361ddf3c 00536de0 00000000 OmniORB4+0x3676c

      3d7cf784 00545918 00000001 00000001 73ec0621 OmniORB4+0x271c8

      3d7cf85c 005452b3 3d7cf878 3d7cf884 3d7cf884 OmniORB4+0x45918

      3d7cf86c 0054535d 39deb170 005acb04 39deb170 OmniORB4+0x452b3

      3d7cf884 0054b986 00000000 00000000 2d54c190 OmniORB4+0x4535d

      3d7cf8f0 0054ac53 3d7cf924 73b14a0b 3d7cf918 OmniORB4+0x4b986

      3d7cf900 704c04e0 0054ac43 3d7cf924 3d7cf924 OmniORB4+0x4ac53

      3d7cf918 0054bb5d 32864870 005ad6d8 32864870 DBEngine+0x404e0

      3d7cf930 00091e5f 00000000 3d7cf988 74a238b3 OmniORB4+0x4bb5d

      3d7cf950 74a23433 32864870 58bb0a21 00000000 OmniThread+0x1e5f

      3d7cf988 74a234c7 00000000 3d7cf9a0 757b339a msvcr90!endthreadex+0x44

      3d7cf994 757b339a 2d54c190 3d7cf9e0 77519ef2 msvcr90!endthreadex+0xd8

      3d7cf9a0 77519ef2 2d54c190 3de8048d 00000000 kernel32!BaseThreadInitThunk+0x12

      3d7cf9e0 77519ec5 74a2345e 2d54c190 ffffffff ntdll!RtlInitializeExceptionChain+0x63

      3d7cf9f8 00000000 74a2345e 2d54c190 00000000 ntdll!RtlInitializeExceptionChain+0x36

       

       

      SYMBOL_NAME: heap_corruption!heap_corruption

       

      FOLLOWUP_NAME: MachineOwner

       

      IMAGE_NAME: heap_corruption

       

      STACK_COMMAND: ~100s; .ecxr ; kb

       

      FAILURE_BUCKET_ID: HEAP_CORRUPTION_c0000005_heap_corruption!heap_corruption

       

      BUCKET_ID: APPLICATION_FAULT_HEAP_CORRUPTION_NULL_CLASS_PTR_DEREFERENCE_NULL_POINTER_READ_INVALID_POINTER_READ_WRONG_SYMBOLS_heap_corruption!heap_corruption

       

      WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/fmserver_exe/11_0_1_86/4b6b3007/ntdll_dll/6_1_7601_17725/4ec49b8f/c0000005/000343d0.htm?Retriage=1

       

      Followup: MachineOwner

      ---------

        • 1. Re: FM 11 Advanced Server crashing (heap_corruption mentioned in the dump file)
          ch0c0halic

          "Access violation - code c0000005"

           

          I haven't seen this in quite awhile. But, I've not been using any of the current Windows OS's so it may still be there.

           

          In the past this was a problem when file access was allowed through both FMS and from the network. Make sure all network access paths to the databases are closed. No one should be able to access the files from the network. If they can then this is the result.

           

          Review article 2696; "Troubleshooting Windows NT Access Violation (C0000005) Errors"

          http://help.filemaker.com/app/answers/detail/a_id/2696

          • 2. Re: FM 11 Advanced Server crashing (heap_corruption mentioned in the dump file)
            wimdecorte

            It could also be a memory access problem.  Can be caused by a faulty RAM module, or by another application / service not playing nice with the memory.  Is the physical  host running many other virtual instances?

            • 3. Re: FM 11 Advanced Server crashing (heap_corruption mentioned in the dump file)
              svend

              Checking with out windows admins -- they confirm that no-one should be get at the .fp7 files via filesharing. Indeed, they say that there are no shares or printers on the server.  They do say, however:

              "Only error log i'm seeing is one in regards to the local admin account.

               

              "It is trying to run the 'rename administrator to admin' GPO and failing because admin already exists. This is due to an old GPO creating a 2nd local admin account on 2008 servers. I've removed the admin account and gpupdate. Now only one local admin account as 'admin'.

               

              "Not sure this will fix the issue with fmserver crashing but worth a shot."

              So, we took the 3-4 hours to rebuild all the databases, and... so far, it's a couple of days later, and we haven't had another crash.  So... maybe it was the weird account rename, maybe it was something in one of the files, but I'm very much hoping it won't happen again.