user24882

Filemaker 14 crashing with ESS (maybe)

Discussion created by user24882 on Mar 21, 2016
Latest reply on Apr 14, 2016 by user24882

Product and version: Filemaker Server 14.04.414

OS and version: OS X 10.11.3

Hardware: Mac mini 2012 quad core, 2x SSD drives

Description: See below

How to replicate No idea, wait for the next one...

Workaround: None found

 

I have a filemaker solution that has been crashing very regularly recently. It is a fairly complex solution that has evolved over the past 7 years. It connects to two different MySQL ESS sources using the ActualTech plugin (server licence). Up until recently it has been very stable, and any crashing issues have been easy to pin down.

 

Recently it has been crashing on a frequent basis, now getting to more than once a day. Until last week the server was running FMS12 on OS X 10.8. The crashing prompted me to finally upgrade to FMS14. I used a clean install os OS X 10.11 and FMS.

It doesn't seem to be linked to any particular scheduled script, as they run repeatedly without issue, sometimes for days. There are a couple of scripts that run regularly, but they always completed before the server falls over.

 

My current hunch is that it might be related to ActualTech, but I can't decode the logs to figure out how.

The ESS servers are running MySQL 5.5.29 on a local ubuntu server, and 5.6.23 on an Amazon RDS instance.

 

I will attached the appropriate crashlog header, followed by the crashed thread from a number of recent crashlogs (from newest to oldest). Perhaps someone can shed more light on it for me?

I also have crashlogs from the previous FMS 12 backed up, so can bring those out if helpful...

 

Process:               fmserverd [267]

Path:                  /Users/USER/*/fmserverd

Identifier:            fmserverd

Version:               0

Code Type:             X86-64 (Native)

Parent Process:        launchd [1]

Responsible:           fmserverd [267]

User ID:               502

 

 

Date/Time:             2016-03-22 11:38:51.431 +1100

OS Version:            Mac OS X 10.11.3 (15D21)

Report Version:        11

 

----

 

Crashed Thread:        65

 

 

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)

Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000038

Exception Note:        EXC_CORPSE_NOTIFY

 

 

VM Regions Near 0x38:

-->

    __TEXT                 0000000105741000-000000010575e000 [  116K] r-x/rwx SM=COW  /Users/USER/*

 

 

Global Trace Buffer (reverse chronological seconds):

231.872130   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799a2ae00] Releasing session

231.873318   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799e09000] Releasing session

231.875053   AppleJPEG                 0x00007fff9b6f311c [0x7fc799e09000] Decoding completed without errors

231.878818   AppleJPEG                 0x00007fff9b6f10fc [0x7fc799e09000] Options: 2x-1 [FFFFFFFF,FFFFFFFF] 0007D060

231.878818   AppleJPEG                 0x00007fff9b6f0fae [0x7fc799e09000] Decoding: C0 0x009A013F 0x0014464A 0x11111111 0x00000000 803088

231.881187   AppleJPEG                 0x00007fff9b6f005e [0x7fc799e09000] Created session

231.888631   AppleJPEG                 0x00007fff9b6f005e [0x7fc799a2ae00] Created session

280.877641   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799798c00] Releasing session

281.053349   AppleJPEG                 0x00007fff9b6f311c [0x7fc799798c00] Decoding completed without errors

281.056297   AppleJPEG                 0x00007fff9b6f10fc [0x7fc799798c00] Options: 1192x842 [FFFFFFFF,FFFFFFFF] 00025060

281.056297   AppleJPEG                 0x00007fff9b6f0fae [0x7fc799798c00] Decoding: C0 0x04A8034A 0x0000354A 0x22111100 0x00000000 305272

281.056297   AppleJPEG                 0x00007fff9b6f005e [0x7fc799798c00] Created session

818.307140   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799484200] Releasing session

818.307140   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799536200] Releasing session

818.307140   AppleJPEG                 0x00007fff9b6f311c [0x7fc799536200] Decoding completed without errors

818.307140   AppleJPEG                 0x00007fff9b6f10fc [0x7fc799536200] Options: 4x-1 [FFFFFFFF,FFFFFFFF] 00025060

818.307140   AppleJPEG                 0x00007fff9b6f0fae [0x7fc799536200] Decoding: C0 0x0145023B 0x0000304A 0x22111100 0x00000000 3650

818.307140   AppleJPEG                 0x00007fff9b6f005e [0x7fc799536200] Created session

818.307140   AppleJPEG                 0x00007fff9b6f005e [0x7fc799484200] Created session

821.622722   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799484200] Releasing session

821.622722   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799536200] Releasing session

821.622722   AppleJPEG                 0x00007fff9b6f311c [0x7fc799536200] Decoding completed without errors

821.622722   AppleJPEG                 0x00007fff9b6f10fc [0x7fc799536200] Options: 2x-1 [FFFFFFFF,FFFFFFFF] 00025060

821.622722   AppleJPEG                 0x00007fff9b6f0fae [0x7fc799536200] Decoding: C0 0x0145023B 0x0000304A 0x22111100 0x00000000 3650

821.622722   AppleJPEG                 0x00007fff9b6f005e [0x7fc799536200] Created session

821.622722   AppleJPEG                 0x00007fff9b6f005e [0x7fc799484200] Created session

821.627713   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799484200] Releasing session

821.627713   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc799536200] Releasing session

821.627713   AppleJPEG                 0x00007fff9b6f311c [0x7fc799536200] Decoding completed without errors

821.627713   AppleJPEG                 0x00007fff9b6f10fc [0x7fc799536200] Options: 2x-1 [FFFFFFFF,FFFFFFFF] 00025060

821.627713   AppleJPEG                 0x00007fff9b6f0fae [0x7fc799536200] Decoding: C0 0x01EA00FB 0x0000304A 0x22111100 0x00000000 2609

821.627713   AppleJPEG                 0x00007fff9b6f005e [0x7fc799536200] Created session

821.627713   AppleJPEG                 0x00007fff9b6f005e [0x7fc799484200] Created session

930.548251   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc79748ae00] Releasing session

930.548702   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc797499e00] Releasing session

930.548821   AppleJPEG                 0x00007fff9b6f311c [0x7fc797499e00] Decoding completed without errors

930.549251   AppleJPEG                 0x00007fff9b6f10fc [0x7fc797499e00] Options: 2x-1 [FFFFFFFF,FFFFFFFF] 00025060

930.549251   AppleJPEG                 0x00007fff9b6f0fae [0x7fc797499e00] Decoding: C0 0x0145023B 0x0000304A 0x22111100 0x00000000 3650

930.549401   AppleJPEG                 0x00007fff9b6f005e [0x7fc797499e00] Created session

930.549484   AppleJPEG                 0x00007fff9b6f005e [0x7fc79748ae00] Created session

930.551189   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc797486c00] Releasing session

930.551727   AppleJPEG                 0x00007fff9b6f0a11 [0x7fc797d79200] Releasing session

930.551858   AppleJPEG                 0x00007fff9b6f311c [0x7fc797d79200] Decoding completed without errors

930.552084   AppleJPEG                 0x00007fff9b6f10fc [0x7fc797d79200] Options: 2x-1 [FFFFFFFF,FFFFFFFF] 00025060

930.552084   AppleJPEG                 0x00007fff9b6f0fae [0x7fc797d79200] Decoding: C0 0x01EA00FB 0x0000304A 0x22111100 0x00000000 2609

930.552326   AppleJPEG                 0x00007fff9b6f005e [0x7fc797d79200] Created session

930.552452   AppleJPEG                 0x00007fff9b6f005e [0x7fc797486c00] Created session

........

Thread 65 Crashed:

0   atopnsrc.so                   0x000000010d6de094 SQLTablesW + 6030

1   atopnsrc.so                   0x000000010d6a0c00 SQLAllocHandle + 1360

2   com.filemaker.dbengine.framework 0x0000000105deceab SQLAllocStmt_Internal + 523

3   com.filemaker.dbengine.framework 0x0000000105ded3e2 SQLAllocStmt + 162

4   com.filemaker.dbengine.framework 0x000000010610f63a FMODBC::ODBCQuery::SetupStatement() + 106

5   com.filemaker.dbengine.framework 0x000000010610ca18 FMODBC::ODBCQuery::ExecuteDirect(Draco::unistring const&, long long) + 468

6   com.filemaker.dbengine.framework 0x0000000106115c7c FMODBC::RCODBCQuery::PerformOperation() + 1304

7   com.filemaker.dbengine.framework 0x0000000106172063 Draco::RCUploadDownload::Perform() + 625

8   com.filemaker.dbengine.framework 0x0000000106181d1f Draco::RCNetworkStack::DispatchTransaction(Draco::RCConnection*, unsigned int, unsigned char*, unsigned int) + 1395

9   com.filemaker.dbengine.framework 0x00000001061815f1 RPO_i::Perform(unsigned int, char const*, unsigned int, OctetSeq&, OctetSeq_out) + 173

10  com.filemaker.dbengine.framework 0x000000010617d26b _0RL_lcfn_07aabdf5dad7309d_70000000(omniCallDescriptor*, omniServant*) + 176

11  com.filemaker.omniorb4.framework 0x00000001066f8cef omniCallHandle::upcall(omniServant*, omniCallDescriptor&) + 1099

12  com.filemaker.dbengine.framework 0x000000010617e69f _impl_RCFMP12::_dispatch(omniCallHandle&) + 1871

13  com.filemaker.omniorb4.framework 0x00000001067ea5bf omni::omniOrbPOA::dispatch(omniCallHandle&, omniLocalIdentity*) + 715

14  com.filemaker.omniorb4.framework 0x00000001067c27b2 omniLocalIdentity::dispatch(omniCallHandle&) + 94

15  com.filemaker.omniorb4.framework 0x0000000106752ba6 omni::GIOP_S::handleRequest() + 330

16  com.filemaker.omniorb4.framework 0x0000000106752805 omni::GIOP_S::dispatcher() + 323

17  com.filemaker.omniorb4.framework 0x000000010676fad6 omni::giopWorker::real_execute() + 1378

18  com.filemaker.omniorb4.framework 0x000000010676fcc6 omni::giopWorker::execute() + 46

19  com.filemaker.omniorb4.framework 0x00000001067751d7 omniAsyncWorker::real_run() + 657

20  com.filemaker.dbengine.framework 0x0000000105e17fbe Draco::threadCreateInterceptor(omni::omniInterceptors::createThread_T::info_T&) + 22

21  com.filemaker.omniorb4.framework 0x0000000106775482 omniAsyncWorker::run(void*) + 46

22  com.filemaker.omniorb4.framework 0x00000001067f9e49 omni_thread_wrapper + 148

23  libsystem_pthread.dylib       0x00007fff89314c13 _pthread_body + 131

24  libsystem_pthread.dylib       0x00007fff89314b90 _pthread_start + 168

25  libsystem_pthread.dylib       0x00007fff89312375 thread_start + 13

 

*****next log

 

Crashed Thread:        66

 

 

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)

Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000038

Exception Note:        EXC_CORPSE_NOTIFY

 

 

VM Regions Near 0x38:

-->

    __TEXT                 000000010c3c6000-000000010c3e3000 [  116K] r-x/rwx SM=COW  /Users/USER/*

 

 

Global Trace Buffer (reverse chronological seconds):

455.058961   AppleJPEG                 0x00007fff9fecba11 [0x7fea97b5a000] Releasing session

455.058961   AppleJPEG                 0x00007fff9fecba11 [0x7fea97b71000] Releasing session

455.070109   AppleJPEG                 0x00007fff9fece11c [0x7fea97b71000] Decoding completed without errors

455.080090   AppleJPEG                 0x00007fff9fecc0fc [0x7fea97b71000] Options: 8x-1 [FFFFFFFF,FFFFFFFF] 0001D060

455.080090   AppleJPEG                 0x00007fff9fecbfae [0x7fea97b71000] Decoding: C0 0x0CC00990 0x00CC304A 0x22111100 0x00000000 1457690

455.080090   AppleJPEG                 0x00007fff9fecb05e [0x7fea97b71000] Created session

455.080090   AppleJPEG                 0x00007fff9fecb05e [0x7fea97b5a000] Created session

455.162790   AppleJPEG                 0x00007fff9fecba11 [0x7fea97b59000] Releasing session

455.162790   AppleJPEG                 0x00007fff9fecba11 [0x7fea97b61000] Releasing session

455.169334   AppleJPEG                 0x00007fff9fece11c [0x7fea97b61000] Decoding completed without errors

455.180972   AppleJPEG                 0x00007fff9fecc0fc [0x7fea97b61000] Options: 8x-1 [FFFFFFFF,FFFFFFFF] 0001D060

455.180972   AppleJPEG                 0x00007fff9fecbfae [0x7fea97b61000] Decoding: C0 0x0CC00990 0x00CC304A 0x22111100 0x00000000 1463074

455.180972   AppleJPEG                 0x00007fff9fecb05e [0x7fea97b61000] Created session

455.180972   AppleJPEG                 0x00007fff9fecb05e [0x7fea97b59000] Created session

455.187213   AppleJPEG                 0x00007fff9fecba11 [0x7fea7a02b400] Releasing session

455.187530   AppleJPEG                 0x00007fff9fecba11 [0x7fea977e2a00] Releasing session

455.190033   AppleJPEG                 0x00007fff9fece11c [0x7fea977e2a00] Decoding completed without errors

455.207201   AppleJPEG                 0x00007fff9fecc0fc [0x7fea977e2a00] Options: 8x-1 [FFFFFFFF,FFFFFFFF] 0001D060

455.207201   AppleJPEG                 0x00007fff9fecbfae [0x7fea977e2a00] Decoding: C0 0x0CC00990 0x00CC304A 0x22111100 0x00000000 1383648

455.207201   AppleJPEG                 0x00007fff9fecb05e [0x7fea977e2a00] Created session

455.207201   AppleJPEG                 0x00007fff9fecb05e [0x7fea7a02b400] Created session

455.214391   AppleJPEG                 0x00007fff9fecba11 [0x7fea7b2c9800] Releasing session

455.214391   AppleJPEG                 0x00007fff9fecba11 [0x7fea7b245200] Releasing session

455.215889   AppleJPEG                 0x00007fff9fece11c [0x7fea7b245200] Decoding completed without errors

455.235885   AppleJPEG                 0x00007fff9fecc0fc [0x7fea7b245200] Options: 8x-1 [FFFFFFFF,FFFFFFFF] 0001D060

.....

Thread 66 Crashed:

0   atopnsrc.so                   0x000000011425f094 SQLTablesW + 6030

1   atopnsrc.so                   0x0000000114221c00 SQLAllocHandle + 1360

2   com.filemaker.dbengine.framework 0x000000010ca72eab SQLAllocStmt_Internal + 523

3   com.filemaker.dbengine.framework 0x000000010ca733e2 SQLAllocStmt + 162

4   com.filemaker.dbengine.framework 0x000000010cd9563a FMODBC::ODBCQuery::SetupStatement() + 106

5   com.filemaker.dbengine.framework 0x000000010cd92a18 FMODBC::ODBCQuery::ExecuteDirect(Draco::unistring const&, long long) + 468

6   com.filemaker.dbengine.framework 0x000000010cd9bc7c FMODBC::RCODBCQuery::PerformOperation() + 1304

7   com.filemaker.dbengine.framework 0x000000010cdf8063 Draco::RCUploadDownload::Perform() + 625

8   com.filemaker.dbengine.framework 0x000000010ce07d1f Draco::RCNetworkStack::DispatchTransaction(Draco::RCConnection*, unsigned int, unsigned char*, unsigned int) + 1395

9   com.filemaker.dbengine.framework 0x000000010ce075f1 RPO_i::Perform(unsigned int, char const*, unsigned int, OctetSeq&, OctetSeq_out) + 173

10  com.filemaker.dbengine.framework 0x000000010ce0326b _0RL_lcfn_07aabdf5dad7309d_70000000(omniCallDescriptor*, omniServant*) + 176

11  com.filemaker.omniorb4.framework 0x000000010d37ccef omniCallHandle::upcall(omniServant*, omniCallDescriptor&) + 1099

12  com.filemaker.dbengine.framework 0x000000010ce0469f _impl_RCFMP12::_dispatch(omniCallHandle&) + 1871

13  com.filemaker.omniorb4.framework 0x000000010d46e5bf omni::omniOrbPOA::dispatch(omniCallHandle&, omniLocalIdentity*) + 715

14  com.filemaker.omniorb4.framework 0x000000010d4467b2 omniLocalIdentity::dispatch(omniCallHandle&) + 94

15  com.filemaker.omniorb4.framework 0x000000010d3d6ba6 omni::GIOP_S::handleRequest() + 330

16  com.filemaker.omniorb4.framework 0x000000010d3d6805 omni::GIOP_S::dispatcher() + 323

17  com.filemaker.omniorb4.framework 0x000000010d3f3ad6 omni::giopWorker::real_execute() + 1378

18  com.filemaker.omniorb4.framework 0x000000010d3f3cc6 omni::giopWorker::execute() + 46

19  com.filemaker.omniorb4.framework 0x000000010d3f91d7 omniAsyncWorker::real_run() + 657

20  com.filemaker.dbengine.framework 0x000000010ca9dfbe Draco::threadCreateInterceptor(omni::omniInterceptors::createThread_T::info_T&) + 22

21  com.filemaker.omniorb4.framework 0x000000010d3f9482 omniAsyncWorker::run(void*) + 46

22  com.filemaker.omniorb4.framework 0x000000010d47de49 omni_thread_wrapper + 148

23  libsystem_pthread.dylib       0x00007fff8daefc13 _pthread_body + 131

24  libsystem_pthread.dylib       0x00007fff8daefb90 _pthread_start + 168

25  libsystem_pthread.dylib       0x00007fff8daed375 thread_start + 13

 

 

****Next Log

 

Crashed Thread:        66

 

 

Exception Type:        EXC_CRASH (SIGABRT)

Exception Codes:       0x0000000000000000, 0x0000000000000000

Exception Note:        EXC_CORPSE_NOTIFY

 

 

Application Specific Information:

abort() called

*** error for object 0x7f7f571a55e0: incorrect checksum for freed object - object was probably modified after being freed.

 

 

 

Global Trace Buffer (reverse chronological seconds):

1662.308649  AppleJPEG                 0x00007fff9c732a11 [0x7f7f381b0800] Releasing session

1662.308649  AppleJPEG                 0x00007fff9c732a11 [0x7f7f381a4000] Releasing session

1662.312764  AppleJPEG                 0x00007fff9c73511c [0x7f7f381a4000] Decoding completed without errors

1662.312764  AppleJPEG                 0x00007fff9c7330fc [0x7f7f381a4000] Options: 4x-1 [FFFFFFFF,FFFFFFFF] 0001D060

1662.312764  AppleJPEG                 0x00007fff9c732fae [0x7f7f381a4000] Decoding: C0 0x037C037C 0x0038304A 0x22111100 0x00000000 58207

 

...

 

 

Thread 66 Crashed:

0   libsystem_kernel.dylib         0x00007fff8f576002 __pthread_kill + 10

1   libsystem_pthread.dylib       0x00007fff8a3585c5 pthread_kill + 90

2   libsystem_c.dylib             0x00007fff889756e7 abort + 129

3   libsystem_malloc.dylib         0x00007fff93a68e66 szone_error + 626

4   libsystem_malloc.dylib         0x00007fff93a5f2e8 tiny_free_list_remove_ptr + 289

5   libsystem_malloc.dylib         0x00007fff93a5dfc4 szone_free_definite_size + 3094

6   atopnsrc.so                   0x0000000108fc0cc3 SQLGetDiagRecW + 14411

7   atopnsrc.so                   0x0000000108fab176 SQLCopyDesc + 70528

8   atopnsrc.so                   0x0000000108f9f0ea SQLCopyDesc + 21236

9   atopnsrc.so                   0x0000000108e64fc5 0x108e57000 + 57285

10  atopnsrc.so                   0x0000000108f99270 SQLFreeHandle + 858

11  com.filemaker.dbengine.framework 0x00000001022f2866 SQLFreeStmt_Internal + 294

12  com.filemaker.dbengine.framework 0x00000001022fa87e SQLFreeHandle + 814

13  com.filemaker.dbengine.framework 0x000000010260e74e FMODBC::ODBCQuery::Reset() + 32

14  com.filemaker.dbengine.framework 0x0000000102614865 FMODBC::ODBCQuery::~ODBCQuery() + 21

15  com.filemaker.dbengine.framework 0x000000010261b090 FMODBC::RCODBCQuery::PerformOperation() + 2348

16  com.filemaker.dbengine.framework 0x0000000102677063 Draco::RCUploadDownload::Perform() + 625

17  com.filemaker.dbengine.framework 0x0000000102686d1f Draco::RCNetworkStack::DispatchTransaction(Draco::RCConnection*, unsigned int, unsigned char*, unsigned int) + 1395

18  com.filemaker.dbengine.framework 0x00000001026865f1 RPO_i::Perform(unsigned int, char const*, unsigned int, OctetSeq&, OctetSeq_out) + 173

19  com.filemaker.dbengine.framework 0x000000010268226b _0RL_lcfn_07aabdf5dad7309d_70000000(omniCallDescriptor*, omniServant*) + 176

20  com.filemaker.omniorb4.framework 0x0000000102bfecef omniCallHandle::upcall(omniServant*, omniCallDescriptor&) + 1099

21  com.filemaker.dbengine.framework 0x000000010268369f _impl_RCFMP12::_dispatch(omniCallHandle&) + 1871

22  com.filemaker.omniorb4.framework 0x0000000102cf05bf omni::omniOrbPOA::dispatch(omniCallHandle&, omniLocalIdentity*) + 715

23  com.filemaker.omniorb4.framework 0x0000000102cc87b2 omniLocalIdentity::dispatch(omniCallHandle&) + 94

24  com.filemaker.omniorb4.framework 0x0000000102c58ba6 omni::GIOP_S::handleRequest() + 330

25  com.filemaker.omniorb4.framework 0x0000000102c58805 omni::GIOP_S::dispatcher() + 323

26  com.filemaker.omniorb4.framework 0x0000000102c75ad6 omni::giopWorker::real_execute() + 1378

27  com.filemaker.omniorb4.framework 0x0000000102c75cc6 omni::giopWorker::execute() + 46

28  com.filemaker.omniorb4.framework 0x0000000102c7b1d7 omniAsyncWorker::real_run() + 657

29  com.filemaker.dbengine.framework 0x000000010231cfbe Draco::threadCreateInterceptor(omni::omniInterceptors::createThread_T::info_T&) + 22

30  com.filemaker.omniorb4.framework 0x0000000102c7b482 omniAsyncWorker::run(void*) + 46

31  com.filemaker.omniorb4.framework 0x0000000102cffe49 omni_thread_wrapper + 148

32  libsystem_pthread.dylib       0x00007fff8a356c13 _pthread_body + 131

33  libsystem_pthread.dylib       0x00007fff8a356b90 _pthread_start + 168

34  libsystem_pthread.dylib       0x00007fff8a354375 thread_start + 13

Outcomes