1 2 Previous Next 15 Replies Latest reply on Jan 28, 2014 6:02 AM by EdwardMcPikeJr

    Why do our servers keep crashing?

    pthomas

      We are having all sorts of problems with our FileMaker servers lately.

       

      Specifically the fmserverd process is crashing - this is happening across 3 different servers!

       

      I have been investigating but am not making much progress so was hoping someone here will be able to have a look at the attached crash reports and be able to point me in the right direction?

       

      Each of the attached reports are from a different server - the guins006 file is from the server that we are having the most trouble with (it has crashed every night for the last week and twice during the day).

       

      If there is any more information I can provide that will help please let me know.

       

      Cheers,

       

      Paul.

       

      Update:

       

      Server version: 12.0.4.405

      OS: MAC OS X 10.8.4

       

      Message was edited by: Paul Thomas

        • 1. Re: Why do our servers keep crashing?
          BruceRobertson

          You shoud provide much more information in the body of your message about your server configuration.

           

          What version of server? What operating system, what version?

          • 2. Re: Why do our servers keep crashing?
            pthomas

            Thanks Bruce - I've updated the post to include that information.

            • 3. Re: Why do our servers keep crashing?
              steve_ssh

              Hello Paul,

               

              FWIW:  You do not appear to be alone.

               

              Here are two similar reports:

               

              http://fmforums.com/forum/topic/89373-regular-crashing-of-fms-server/

               

              and

               

              http://forums.filemaker.com/posts/315bf4bd0d

               

               

              Of note is that in each of the crash logs posted (your three, and those from the other two posts), the crash is always occurring in thread #11, right at the time that FMS appears to be attempting to update stored (plist) preferences via a standard Apple API.

               

              Apologies if this was already apparent to you -- I figured it would be good to mention it, just in case this jogs someone's recollection of the appropriate diagnosis/remedy.

               

              Very best & good luck,

               

              -steve

              1 of 1 people found this helpful
              • 4. Re: Why do our servers keep crashing?
                mbraendle

                Because the same happened at different users (installations), I would recommend to open a case at FM Inc. support so that this can be investigated.

                • 5. Re: Why do our servers keep crashing?
                  TSGal

                  Paul,

                   

                  This does look similar to an OS X issue that has been reported to Apple.  However, we encourage you to contact our Technical Support department so a case number can be assigned.  You can contact our Technical Support department at toll-free 800-325-2747 (if in North America).  Please provide your full system configuration.  We would also like to see your system.log (or system.log.0.bz2 or similar), and the FileMaker Server Event.log from the three servers from around the time of the crashes.  If you kept the system resource utilization statistics, we would also like to review that.

                   

                  TSGal

                  FileMaker, Inc.

                  • 6. Re: Why do our servers keep crashing?
                    pthomas

                    Thanks for the replies and suggestions everyone.

                     

                    I have opened an official support request here:

                     

                    http://forums.filemaker.com/posts/a68b5b24d8

                     

                    And will update this post when I know more.

                     

                    Cheers,

                     

                    Paul.

                    • 7. Re: Why do our servers keep crashing?
                      pthomas

                      Well our servers are still crashing and I haven't heard anything since posting the issue report.

                       

                      Last night two of our servers (in different countries) both crashed with basically the same error, seems to be that the crash happens when FileMaker is attempting to update some sort of preference file when a schedule has finished running (presumably to update the last complted and status details):

                       

                      Our system admin has asked if anyone knows the location of the preference files? Plist, Property List, PrefsArchiver or PreferencesManager?

                       

                      GUINS00K – NZFM

                       

                      Process: fmserverd [275]

                      Path: /Library/FileMaker Server/*/fmserverd

                      Identifier: fmserverd

                      Version: ???

                      Code Type: X86-64 (Native)

                      Parent Process: launchd [1]

                      User ID: 502

                       

                      Date/Time: 2013-09-02 23:04:21.169 +1200

                      OS Version: Mac OS X 10.8.4 (12E55)

                      Report Version: 10

                       

                      Crashed Thread: 11

                       

                      Exception Type: EXC_BAD_ACCESS (SIGSEGV)

                      Exception Codes: EXC_I386_GPFLT

                       

                      Thread 11 Crashed:

                      0 libobjc.A.dylib 0x00007fff8c7e0250 objc_msgSend + 16

                      1 com.apple.CoreFoundation 0x00007fff89aa4f81 __CFPropertyListIsValidAux + 65

                      2 com.apple.CoreFoundation 0x00007fff89ac4dd7 __CFPropertyListIsDictPlistAux + 247

                      3 com.apple.CoreFoundation 0x00007fff89b5c986 __CFDictionaryApplyFunction_block_invoke_0 + 22

                      4 com.apple.CoreFoundation 0x00007fff89a867b0 CFBasicHashApply + 128

                      5 com.apple.CoreFoundation 0x00007fff89aa431e CFDictionaryApplyFunction + 158

                      6 com.apple.CoreFoundation 0x00007fff89aa509d __CFPropertyListIsValidAux + 349

                      7 com.apple.CoreFoundation 0x00007fff89ad44dd CFPropertyListIsValid + 93

                      8 com.apple.CoreFoundation 0x00007fff89adf0e3 CFPropertyListCreateDeepCopy + 83

                      9 com.apple.CoreFoundation 0x00007fff89bedc20 -[CFPrefsPlistSource setValue:forKey:] + 128

                      10 com.apple.CoreFoundation 0x00007fff89beafb4 +[CFPrefsSource withSourceForIdentifier:user:byHost:container:perform:] + 884

                      11 com.apple.CoreFoundation 0x00007fff89be98fe _CFPreferencesSetValueWithContainer + 238

                      12 com.filemaker.FMS.framework 0x0000000100148fa0 MacPrefs::FinalCloseStorage(Draco::unistring const&, __CFDictionary*&) + 80

                      13 com.filemaker.FMS.framework 0x000000010014cace PrefsArchiver::PutTree(FMSTypes::PreferenceList const&, FMSTypes::PreferenceList&) + 894

                      14 com.filemaker.PrefsMgr.framework 0x0000000100350635 PreferencesMgr::SetRequest(FMS::EventHelper&) + 133

                      15 com.filemaker.PrefsMgr.framework 0x00000001003514b9 PreferencesMgr::ProcessEvent(FMS::EventHelper&) + 137

                      16 com.filemaker.FMS.framework 0x00000001001b8d18 FMS::ComponentBaseThread::Main() + 232

                      17 com.filemaker.PrefsMgr.framework 0x000000010034d8ff PreferencesMgr::Main() + 383

                      18 com.filemaker.support.framework 0x00000001016f94eb Draco::Thread::RunThread() + 43

                      19 com.filemaker.support.framework 0x00000001016fa7f9 Draco::Thread::ThreadUnixProc(void*) + 57

                      20 libsystem_c.dylib 0x00007fff816dc7a2 _pthread_start + 327

                      21 libsystem_c.dylib 0x00007fff816c91e1 thread_start + 13

                       

                       

                      MLBNS002 - FMAU

                       

                      Process: fmserverd [165]

                      Path: /Library/FileMaker Server/*/fmserverd

                      Identifier: fmserverd

                      Version: ???

                      Code Type: X86-64 (Native)

                      Parent Process: launchd [1]

                      User ID: 502

                       

                      Date/Time: 2013-09-02 22:31:43.586 +1000

                      OS Version: Mac OS X 10.8.4 (12E55)

                      Report Version: 10

                       

                      Crashed Thread: 11

                       

                      Exception Type: EXC_BAD_ACCESS (SIGSEGV)

                      Exception Codes: EXC_I386_GPFLT

                       

                      Thread 11 Crashed:

                      0 libobjc.A.dylib 0x00007fff88bce250 objc_msgSend + 16

                      1 com.apple.CoreFoundation 0x00007fff8d1caf81 __CFPropertyListIsValidAux + 65

                      2 com.apple.CoreFoundation 0x00007fff8d1eadd7 __CFPropertyListIsDictPlistAux + 247

                      3 com.apple.CoreFoundation 0x00007fff8d282986 __CFDictionaryApplyFunction_block_invoke_0 + 22

                      4 com.apple.CoreFoundation 0x00007fff8d1ac7b0 CFBasicHashApply + 128

                      5 com.apple.CoreFoundation 0x00007fff8d1ca31e CFDictionaryApplyFunction + 158

                      6 com.apple.CoreFoundation 0x00007fff8d1cb09d __CFPropertyListIsValidAux + 349

                      7 com.apple.CoreFoundation 0x00007fff8d1fa4dd CFPropertyListIsValid + 93

                      8 com.apple.CoreFoundation 0x00007fff8d2050e3 CFPropertyListCreateDeepCopy + 83

                      9 com.apple.CoreFoundation 0x00007fff8d313c20 -[CFPrefsPlistSource setValue:forKey:] + 128

                      10 com.apple.CoreFoundation 0x00007fff8d310fb4 +[CFPrefsSource withSourceForIdentifier:user:byHost:container:perform:] + 884

                      11 com.apple.CoreFoundation 0x00007fff8d30f8fe _CFPreferencesSetValueWithContainer + 238

                      12 com.filemaker.FMS.framework 0x0000000100148fa0 MacPrefs::FinalCloseStorage(Draco::unistring const&, __CFDictionary*&) + 80

                      13 com.filemaker.FMS.framework 0x000000010014cace PrefsArchiver::PutTree(FMSTypes::PreferenceList const&, FMSTypes::PreferenceList&) + 894

                      14 com.filemaker.PrefsMgr.framework 0x0000000100350635 PreferencesMgr::SetRequest(FMS::EventHelper&) + 133

                      15 com.filemaker.PrefsMgr.framework 0x00000001003514b9 PreferencesMgr::ProcessEvent(FMS::EventHelper&) + 137

                      16 com.filemaker.FMS.framework 0x00000001001b8d18 FMS::ComponentBaseThread::Main() + 232

                      17 com.filemaker.PrefsMgr.framework 0x000000010034d8ff PreferencesMgr::Main() + 383

                      18 com.filemaker.support.framework 0x00000001016f94eb Draco::Thread::RunThread() + 43

                      19 com.filemaker.support.framework 0x00000001016fa7f9 Draco::Thread::ThreadUnixProc(void*) + 57

                      20 libsystem_c.dylib 0x00007fff8533d7a2 _pthread_start + 327

                      21 libsystem_c.dylib 0x00007fff8532a1e1 thread_start + 13

                      • 8. Re: Why do our servers keep crashing?
                        steve.winter

                        Hi

                         

                        Thanks for your email. I'm currently on leave until Monday 9th of September with limited access to

                        email.

                         

                        If your message is urgent, please forward it to support@msdev.co.uk or call +44 777 852 4776.

                         

                        In all other cases I'll reply as soon as I'm able.

                         

                        Cheers

                        Steve

                        • 9. Re: Why do our servers keep crashing?
                          TSGal

                          Paul,

                           

                          I just saw your update, and I responded to your post on the FileMaker forum.

                           

                          TSGal

                          FileMaker, Inc.

                          • 10. Re: Why do our servers keep crashing?
                            TSGal

                            Paul,

                             

                            Your files were received, and it is definitely associated to a CrashTracer issue reported to Apple Computer.  Apple is still analyzing this issue, and as of July 24, 2013, Apple has received 2,730 crash reports related to this issue.

                             

                            I will post again when more information becomes available.

                             

                            TSGal

                            FileMaker, Inc.

                            • 11. Re: Why do our servers keep crashing?
                              user10272

                              Any news on this?

                               

                              We are also experiencing this on a couple of our servers.

                               

                              Thanks!

                               

                              I also checked in at:

                              http://forums.filemaker.com/posts/a68b5b24d8

                              • 12. Re: Why do our servers keep crashing?
                                ericjungemann

                                We reported this to FileMaker Support over a year ago . . . did some work with them that didn't go anywhere.

                                 

                                For us, it happens with a full FMS process crash sometimes or just taking server offline.

                                 

                                It still happens . . . generally at the end of a backup, presumably when the plist is being updated(?).  Fortunately, it generally doesn't damage the database and comes back up with a reboot or just starting the server depending on the version of it.

                                 

                                It is 10.8.  Up to date FMS 12.  Doesn't happen in all our server installations.

                                 

                                Our plan is to move folks briskly to FMS 13.  In our testing, it doesn't seem to happen there but, unfortunately, the truth will probably emerge only when we have it widely deployed (and we are always mindful of 'the problems you know, vs. the problems you don't know).

                                • 13. Re: Why do our servers keep crashing?
                                  user10272

                                  Ok, we have the problem only on two servers and did not manage to replicate the problem on an identical server (ghosted from same image - macmini) with the same files that made fms crash. Now I wonder it it's the "save multiple copies of " the backup that is the culprit, since on the crashing server, the first crash happened right after the x:th backup was done, where x is how many copies of the backup to keep.

                                   

                                  Unfortunately our test server is no longer set up. It could be easily tested with just keeping 2 copies and let it run i think.

                                   

                                  Worth to notice is that we have several more more or less identical servers with same setup that do not have this issue. The main differenace beeing that there is also a local FM-client running on the fms machine. (The ones crashing lacks the running fm client.)

                                   

                                  I will probably try to set up our test server again.

                                  • 14. Re: Why do our servers keep crashing?
                                    user10272

                                    Ok, major breakthrough for us! We have it fixed!

                                     

                                    We do not know why it was faulty in the first place, but it seems that javas plist had the wrong permissions. When my collegue ran a "Repair Disk Persmissions" with the disk utility this was found!

                                     

                                    Group differs on “Library/Java”; should be 0; group is 80.

                                    Permissions differ on “Library/Java”; should be drwxr-xr-x ; they are drwxrwxr-x .

                                    Repaired “Library/Java”

                                    Group differs on “Library/Preferences/com.apple.alf.plist”; should be 80; group is 0.

                                    Repaired “Library/Preferences/com.apple.alf.plist”

                                     

                                     

                                    And now it seems that the backups and schedules are working as intended again. Maybe we should let it cool for some time before dancing all crazy, but it feels logical, since it was thread 11 - plist stuff, that was the crashing thread.

                                     

                                    Hope this helps someone!

                                     

                                    I will try to update you with more info if I find it.

                                    1 2 Previous Next