svend

FM 11 Advanced Server crashing (heap_corruption mentioned in the dump file)

Discussion created by svend on Jan 21, 2013
Latest reply on Jan 23, 2013 by svend

We've got a virtual machine running FM Server Advanced, v11.03.309. Yesterday evening, the server process (but not the admin console) crashed; when the server was restarted, one of the files was deemed to be corrupt, and had to be recovered. Once recovered, the day's changes were copied into the previous day's backup, and that composite file with put back on the server. We did not reboot the box, or do anything else to any of the other files being served.

 

The next day, in the morning, the server crashed again. This time, none of the files reported any corruption.

 

There's nothing in the FileMaker logs in either case to indicate the problem -- they go straight from people logging in and out, to restarting the server/reopening the database files.

 

Using the Windows debugger, the crash files talk about the faulting module being called "heap_corruption"; I'll include the full output below. Our Systems team reports that they don't see anything unusual on the filer/network/etc end.

 

Tonight, we're going to reboot the virtual machine, clone all the 40-odd files (and re-import their data), and replace the existing files with their clones. Is there anything else obvious we should do.

 

The web server isn't running, as we do not provide any web services.

 

The virtual machine is running Windows Server 2008 R2 Standard with Service Pack 1. It's running as a 4-core 2.70GHz machine with 24GB of ram; I believe that this memory is actually reserved for it on the underlying box. This box provides no other services. The virtual machine lives in VMWare VSphere 5, on top of a 6 node cisco UCS cluster, with its back end datastores on NetApp. There's no evidence that it's starved for resources.

 

The crash report for the first crash said:

 

 

Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64

Copyright (c) Microsoft Corporation. All rights reserved.

 

 

Loading Dump File [C:AppCrash_fmserver.exe_44bfcdd01d5652a9774efea35dbf3021eed4_cab_710e130eWER3CD.tmp.hdmp]

User Mini Dump File: Only registers, stack and portions of memory are available

 

Symbol search path is: *** Invalid ***

****************************************************************************

* Symbol loading may be unreliable without a symbol search path. *

* Use .symfix to have the debugger choose a symbol path. *

* After setting your symbol path, use .reload to refresh symbol locations. *

****************************************************************************

Executable search path is:

Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x86 compatible

Product: Server, suite: TerminalServer SingleUserTS

Machine Name:

Debug session time: Mon Jan 21 18:17:12.000 2013 (UTC + 13:00)

System Uptime: 23 days 20:31:51.059

Process Uptime: 0 days 14:15:26.000

................................................................

.............................

Loading unloaded module list

.......

This dump file has an exception of interest stored in it.

The stored exception information can be accessed via .ecxr.

(680.dfc): Access violation - code c0000005 (first/second chance not available)

eax=00000000 ebx=3d7cee08 ecx=00000000 edx=00000000 esi=00000002 edi=00000000

eip=7750013d esp=3d7cedb8 ebp=3d7cee54 iopl=0 nv up ei pl zr na pe nc

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246

*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -

ntdll!ZwWaitForMultipleObjects+0x15:

7750013d 83c404 add esp,4

0:100> !analyze -v

 

 

GetPageUrlData failed, server returned HTTP status 404

URL requested: http://watson.microsoft.com/StageOne/fmserver_exe/11_0_1_86/4b6b3007/ntdll_dll/6_1_7601_17725/4ec49b8f/c0000005/000343d0.htm?Retriage=1

 

FAULTING_IP:

ntdll!RtlImageNtHeader+124c

775143d0 8b4904 mov ecx,dword ptr [ecx+4]

 

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)

ExceptionAddress: 775143d0 (ntdll!RtlImageNtHeader+0x0000124c)

ExceptionCode: c0000005 (Access violation)

ExceptionFlags: 00000000

NumberParameters: 2

Parameter[0]: 00000000

Parameter[1]: 00000004

Attempt to read from address 00000004

 

PROCESS_NAME: fmserver.exe

 

FAULTING_MODULE: 757a0000 kernel32

 

DEBUG_FLR_IMAGE_TIMESTAMP: 0

 

MODULE_NAME: heap_corruption

 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

 

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

 

EXCEPTION_PARAMETER1: 00000000

 

EXCEPTION_PARAMETER2: 00000004

 

READ_ADDRESS: 00000004

 

FOLLOWUP_IP:

ntdll!RtlImageNtHeader+124c

775143d0 8b4904 mov ecx,dword ptr [ecx+4]

 

MOD_LIST: <ANALYSIS/>

 

ADDITIONAL_DEBUG_TEXT:

Use '!findthebuild' command to search for the target build information.

If the build information is available, run '!findthebuild -s ; .reload' to set symbol path and load symbols. ; Enable Pageheap/AutoVerifer

 

FAULTING_THREAD: 00000dfc

 

DEFAULT_BUCKET_ID: HEAP_CORRUPTION

 

PRIMARY_PROBLEM_CLASS: HEAP_CORRUPTION

 

BUGCHECK_STR: APPLICATION_FAULT_HEAP_CORRUPTION_NULL_CLASS_PTR_DEREFERENCE_NULL_POINTER_READ_INVALID_POINTER_READ_WRONG_SYMBOLS

 

LAST_CONTROL_TRANSFER: from 775135a7 to 775143d0

 

STACK_TEXT:

WARNING: Stack unwind information not available. Following frames may be wrong.

3d7cf464 775135a7 010d0000 47847568 3d7cf52c ntdll!RtlImageNtHeader+0x124c

3d7cf55c 77513492 47847568 47847570 47847570 ntdll!RtlImageNtHeader+0x423

3d7cf57c 757b14dd 010d0000 00000000 47847570 ntdll!RtlImageNtHeader+0x30e

3d7cf590 74a63c1b 010d0000 00000000 47847570 kernel32!HeapFree+0x14

3d7cf5dc 005429cb 47847570 47847570 47847570 msvcr90!free+0xcd

3d7cf5f0 00542b02 47847570 361ddf3c 00000017 OmniORB4+0x429cb

3d7cf608 00542e78 ffffffff 005abfd0 0000036a OmniORB4+0x42b02

3d7cf6ac 00536122 01349610 3d7cf6c4 00091141 OmniORB4+0x42e78

3d7cf6c0 0053676c 361ddf3c 00000000 01349610 OmniORB4+0x36122

3d7cf6e4 005271c8 361ddf3c 00536de0 00000000 OmniORB4+0x3676c

3d7cf784 00545918 00000001 00000001 73ec0621 OmniORB4+0x271c8

3d7cf85c 005452b3 3d7cf878 3d7cf884 3d7cf884 OmniORB4+0x45918

3d7cf86c 0054535d 39deb170 005acb04 39deb170 OmniORB4+0x452b3

3d7cf884 0054b986 00000000 00000000 2d54c190 OmniORB4+0x4535d

3d7cf8f0 0054ac53 3d7cf924 73b14a0b 3d7cf918 OmniORB4+0x4b986

3d7cf900 704c04e0 0054ac43 3d7cf924 3d7cf924 OmniORB4+0x4ac53

3d7cf918 0054bb5d 32864870 005ad6d8 32864870 DBEngine+0x404e0

3d7cf930 00091e5f 00000000 3d7cf988 74a238b3 OmniORB4+0x4bb5d

3d7cf950 74a23433 32864870 58bb0a21 00000000 OmniThread+0x1e5f

3d7cf988 74a234c7 00000000 3d7cf9a0 757b339a msvcr90!endthreadex+0x44

3d7cf994 757b339a 2d54c190 3d7cf9e0 77519ef2 msvcr90!endthreadex+0xd8

3d7cf9a0 77519ef2 2d54c190 3de8048d 00000000 kernel32!BaseThreadInitThunk+0x12

3d7cf9e0 77519ec5 74a2345e 2d54c190 ffffffff ntdll!RtlInitializeExceptionChain+0x63

3d7cf9f8 00000000 74a2345e 2d54c190 00000000 ntdll!RtlInitializeExceptionChain+0x36

 

 

SYMBOL_NAME: heap_corruption!heap_corruption

 

FOLLOWUP_NAME: MachineOwner

 

IMAGE_NAME: heap_corruption

 

STACK_COMMAND: ~100s; .ecxr ; kb

 

FAILURE_BUCKET_ID: HEAP_CORRUPTION_c0000005_heap_corruption!heap_corruption

 

BUCKET_ID: APPLICATION_FAULT_HEAP_CORRUPTION_NULL_CLASS_PTR_DEREFERENCE_NULL_POINTER_READ_INVALID_POINTER_READ_WRONG_SYMBOLS_heap_corruption!heap_corruption

 

WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/fmserver_exe/11_0_1_86/4b6b3007/ntdll_dll/6_1_7601_17725/4ec49b8f/c0000005/000343d0.htm?Retriage=1

 

Followup: MachineOwner

---------

 

Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64

Copyright (c) Microsoft Corporation. All rights reserved.

 

 

Loading Dump File [C:AppCrash_fmserver.exe_44bfcdd01d5652a9774efea35dbf3021eed4_cab_710e130eWER1274.tmp.mdmp]

User Mini Dump File: Only registers, stack and portions of memory are available

 

Symbol search path is: *** Invalid ***

****************************************************************************

* Symbol loading may be unreliable without a symbol search path. *

* Use .symfix to have the debugger choose a symbol path. *

* After setting your symbol path, use .reload to refresh symbol locations. *

****************************************************************************

Executable search path is:

Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x86 compatible

Product: Server, suite: TerminalServer SingleUserTS

Machine Name:

Debug session time: Mon Jan 21 18:17:15.000 2013 (UTC + 13:00)

System Uptime: 23 days 20:31:52.075

Process Uptime: 0 days 14:15:29.000

................................................................

.............................

Loading unloaded module list

.......

This dump file has an exception of interest stored in it.

The stored exception information can be accessed via .ecxr.

(680.dfc): Access violation - code c0000005 (first/second chance not available)

eax=00000000 ebx=3d7cee08 ecx=00000000 edx=00000000 esi=00000002 edi=00000000

eip=7750013d esp=3d7cedb8 ebp=3d7cee54 iopl=0 nv up ei pl zr na pe nc

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246

*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -

ntdll!ZwWaitForMultipleObjects+0x15:

7750013d 83c404 add esp,4

0:100> !analyze -v

 

FAULTING_IP:

ntdll!RtlImageNtHeader+124c

775143d0 8b4904 mov ecx,dword ptr [ecx+4]

 

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)

ExceptionAddress: 775143d0 (ntdll!RtlImageNtHeader+0x0000124c)

ExceptionCode: c0000005 (Access violation)

ExceptionFlags: 00000000

NumberParameters: 2

Parameter[0]: 00000000

Parameter[1]: 00000004

Attempt to read from address 00000004

 

PROCESS_NAME: fmserver.exe

 

FAULTING_MODULE: 757a0000 kernel32

 

DEBUG_FLR_IMAGE_TIMESTAMP: 0

 

MODULE_NAME: heap_corruption

 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

 

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

 

EXCEPTION_PARAMETER1: 00000000

 

EXCEPTION_PARAMETER2: 00000004

 

READ_ADDRESS: 00000004

 

FOLLOWUP_IP:

ntdll!RtlImageNtHeader+124c

775143d0 8b4904 mov ecx,dword ptr [ecx+4]

 

MOD_LIST: <ANALYSIS/>

 

ADDITIONAL_DEBUG_TEXT:

Use '!findthebuild' command to search for the target build information.

If the build information is available, run '!findthebuild -s ; .reload' to set symbol path and load symbols. ; Enable Pageheap/AutoVerifer

 

FAULTING_THREAD: 00000dfc

 

DEFAULT_BUCKET_ID: HEAP_CORRUPTION

 

PRIMARY_PROBLEM_CLASS: HEAP_CORRUPTION

 

BUGCHECK_STR: APPLICATION_FAULT_HEAP_CORRUPTION_NULL_CLASS_PTR_DEREFERENCE_NULL_POINTER_READ_INVALID_POINTER_READ_WRONG_SYMBOLS

 

LAST_CONTROL_TRANSFER: from 775135a7 to 775143d0

 

STACK_TEXT:

WARNING: Stack unwind information not available. Following frames may be wrong.

3d7cf464 775135a7 010d0000 47847568 3d7cf52c ntdll!RtlImageNtHeader+0x124c

3d7cf55c 77513492 47847568 47847570 47847570 ntdll!RtlImageNtHeader+0x423

3d7cf57c 757b14dd 010d0000 00000000 47847570 ntdll!RtlImageNtHeader+0x30e

3d7cf590 74a63c1b 010d0000 00000000 47847570 kernel32!HeapFree+0x14

3d7cf5dc 005429cb 47847570 47847570 47847570 msvcr90!free+0xcd

3d7cf5f0 00542b02 47847570 361ddf3c 00000017 OmniORB4+0x429cb

3d7cf608 00542e78 ffffffff 005abfd0 0000036a OmniORB4+0x42b02

3d7cf6ac 00536122 01349610 3d7cf6c4 00091141 OmniORB4+0x42e78

3d7cf6c0 0053676c 361ddf3c 00000000 01349610 OmniORB4+0x36122

3d7cf6e4 005271c8 361ddf3c 00536de0 00000000 OmniORB4+0x3676c

3d7cf784 00545918 00000001 00000001 73ec0621 OmniORB4+0x271c8

3d7cf85c 005452b3 3d7cf878 3d7cf884 3d7cf884 OmniORB4+0x45918

3d7cf86c 0054535d 39deb170 005acb04 39deb170 OmniORB4+0x452b3

3d7cf884 0054b986 00000000 00000000 2d54c190 OmniORB4+0x4535d

3d7cf8f0 0054ac53 3d7cf924 73b14a0b 3d7cf918 OmniORB4+0x4b986

3d7cf900 704c04e0 0054ac43 3d7cf924 3d7cf924 OmniORB4+0x4ac53

3d7cf918 0054bb5d 32864870 005ad6d8 32864870 DBEngine+0x404e0

3d7cf930 00091e5f 00000000 3d7cf988 74a238b3 OmniORB4+0x4bb5d

3d7cf950 74a23433 32864870 58bb0a21 00000000 OmniThread+0x1e5f

3d7cf988 74a234c7 00000000 3d7cf9a0 757b339a msvcr90!endthreadex+0x44

3d7cf994 757b339a 2d54c190 3d7cf9e0 77519ef2 msvcr90!endthreadex+0xd8

3d7cf9a0 77519ef2 2d54c190 3de8048d 00000000 kernel32!BaseThreadInitThunk+0x12

3d7cf9e0 77519ec5 74a2345e 2d54c190 ffffffff ntdll!RtlInitializeExceptionChain+0x63

3d7cf9f8 00000000 74a2345e 2d54c190 00000000 ntdll!RtlInitializeExceptionChain+0x36

 

 

SYMBOL_NAME: heap_corruption!heap_corruption

 

FOLLOWUP_NAME: MachineOwner

 

IMAGE_NAME: heap_corruption

 

STACK_COMMAND: ~100s; .ecxr ; kb

 

FAILURE_BUCKET_ID: HEAP_CORRUPTION_c0000005_heap_corruption!heap_corruption

 

BUCKET_ID: APPLICATION_FAULT_HEAP_CORRUPTION_NULL_CLASS_PTR_DEREFERENCE_NULL_POINTER_READ_INVALID_POINTER_READ_WRONG_SYMBOLS_heap_corruption!heap_corruption

 

WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/fmserver_exe/11_0_1_86/4b6b3007/ntdll_dll/6_1_7601_17725/4ec49b8f/c0000005/000343d0.htm?Retriage=1

 

Followup: MachineOwner

---------

Outcomes