1 2 Previous Next 21 Replies Latest reply on Sep 12, 2014 9:46 AM by PointInSpace

    Second FM Server crash.

    bigtom

      This is twice in 3 months that FMS has just crapped out. The admin panel was there but nothing worked. When selecting the things on the left side menu I got an icon of red cirlce with exclamaiotn mark inside.

       

      I checked all the logs and I did not find anything that would have caused this or anything at all except for the backups had an error 20232 for inconsistency.

       

      How can I best get to the root of why this is happening. This happened overnight when no one was using the server.

        • 1. Re: Second FM Server crash.
          taylorsharpe

          There are always possibilities of hardware failures like hard drives or bad memory.  You can always test the memory and the drives as a first step.  How is your storage setup (e.g., internal drives, external drives, RAID, SAN, types of connections)?  Have you had any power outages or possible lightning strikes?  Do you have a backup power supply?

           

          Regarding the software side of things, how about some more details.  What operating System are you using and what version of FileMaker Server software?  Do you have any plugins installed?  If so, which ones and what versions?

           

          Have you run consistency checks and maybe recovery of the databases to see if there are any errors?

           

          Those are some starting points.  There are other tools like FMDIFF and Base Elements and looking through the DDR if these don't find something to start with.  Let us know and we'll help with ideas to check on. 

          • 2. Re: Second FM Server crash.
            bigtom

            Thanks for the reply.

             

            Setup is a latest Mac Mini Server. 2x256 Apple SSD RAID1 internal. Memory is 16GB installed by Apple. Backups are external USB drives, progressive and 2x scheduled backups. Have UPS. No power outages. No lightning strikes. OSX 10.9.2, no server side plugins, FileMaker Server 13.

             

            To get the Server running again I tried starting and stopping FMS from the command line. This resulted in the same behavior of the admin panel. I had to shut the server machine down and restart. FMS launched on restart and then was working again.

             

            The hardware seemed to checkout so far, but I will see if there is anything else I might be able to check them with. I will keep you posted and I appreciate the help. I really do not need this going down in the middle of the day. Thankfully, both times were over night and I was able to get it up and going by working hours.

            • 3. Re: Second FM Server crash.
              wimdecorte

              Fusion drive in that Mac Mini?

               

              The fact that the admin console does not respond properly does not mean that the database engine has crashed.  The Admin Server and the Database Engine run in separate processes and do not affect each other.  So check the FMS logs and the OSX system logs for clues, that's the best place to start.

               

              Anything else running on that machine?  Anti-virus?  Online backups? ...

              • 4. Re: Second FM Server crash.
                taylorsharpe

                Probably not a Fusion drive since that would be 1TB.  256 GB SSDs are offered in the Apple store and I would prefer them over Fusion drives in a server even though they lack storage.

                 

                Wim brings up some good ideas on if other services are scanning the live FileMaker databases.  The common Apple culprits are Spotlight and Time Machine.  Make sure those services are excluding the live FileMaker database folder.  They can scan the backups, just not live FileMaker databases.  Also true on any anti-virus software if you have any.  Just make sure other services (or youself) don't ever touch live open FM databases. 

                 

                It sounds like you have a fairly good setup with UPS, Mavericks, RAID storage and 16 Gigs of RAM.  You might make sure you are up to FileMaker 13.0.2.295 update if you haven't already. 

                 

                Once a any of the FileMaker services starts to act weird, I usually do a complete system reboot to clear out any problems.  I know... probably habit from working on so many Windows machines.  But it seems to work most of the time. 

                 

                There is always a weird chance you have bad memory.  You could use the Apple Hardware test software to test it although its not as good as most 3rd party ones like Memtest86 (free download and you setup a bootable USB drive to test the RAM). 

                 

                If you're doing hardware tests... does your UPS give stats on power.  Is your voltage running high low or anything else weird like that?

                • 5. Re: Second FM Server crash.
                  bigtom

                  Just crashed again. Got the message that the admin console was limited to Status and Logs.

                   

                  The one thing I cannot seem to turn off is the Notification center.

                   

                  I was able to find a failuer in the console this tim and it looks like some sort of memory allocation problem. Maybe I just need to allocate more memory or the memory is bad? Below are the three entries from the OSX Console related to this.

                   

                   

                  fmserverd[210]: fmserverd(210,0x118ae7000) malloc: *** mach_vm_map(size=59074891485184) failed (error code=3)

                  *** error: can't allocate region

                  *** set a breakpoint in malloc_error_break to debug

                   

                  ReportCrash[19546]: DebugSymbols was unable to start a spotlight query: spotlight is not responding or disabled.

                   

                  ReportCrash[19546]: Saved crash report for fmserverd[210] version ??? to /Library/Logs/DiagnosticReports/fmserverd_2014-06-04-122925_Server-Mac-mini.crash

                  • 6. Re: Second FM Server crash.
                    taylorsharpe

                    So its not just FileMaker service crashing... Spotlight crashed too.  Hmmmmm.... how about trying an upgrade to Mac OS X 10.9.3 and see if that helps.  I know 10.9.2 had some weird things that even kept it from booting in Safe mode.  But then again, it could be the memory. 

                     

                    I would turn your cache down in the FileMake Admin console since that is how much info can be stored in RAM before being dumped back to the drive since you are having crashing issues.  Keep that number small so you don't loose much at least until you figure out what is going on. 

                     

                    Actually, if upgrading to 10.9.3 doesn't work, I would boot into the restore mode (boot holding down Apple-R) and restore the operating system.  It won't remove FileMaker, but will reinstall all OS components.  Then run the upgrades to the OS.  That would clear out any corruption in the OS. 

                    • 7. Re: Second FM Server crash.
                      bigtom

                      Here is the logged crash file. I just have no idea how to read it to discover the problem.

                      • 8. Re: Second FM Server crash.
                        bigtom

                        Spotlight crashed? It should be disbaled.

                         

                        Looks like the previous crash was some sort of Spin problem with finder.

                        • 9. Re: Second FM Server crash.
                          taylorsharpe

                          Mavericks has problems if you disable spotlight.  So you don't want to disable it, just make sure it doesn't scan live FileMaker files. 

                           

                          What other services or applications are running at the same time? 

                           

                          But I seriously would be considering the OS recovery at this point. 

                          • 10. Re: Second FM Server crash.
                            bigtom

                            I also had these two errors two minutes before this most recent crash. I have really tried to set this up properly.

                             

                            secd[298]:  SecErrorGetOSStatus unknown error domain: com.apple.security.sos.error for error: The operation couldn’t be completed. (com.apple.security.sos.error error 2 - Public Key not available - failed to register before call)

                             

                             

                            secd[298]:  securityd_xpc_dictionary_handler EscrowSecurityAl[3580] DeviceInCircle The operation couldn’t be completed. (com.apple.security.sos.error error 2 - Public Key not available - failed to register before call)

                            • 11. Re: Second FM Server crash.
                              bigtom

                              No other services or apps are open except Safari and Console.

                               

                              There are a lot of these in the console. Do you know what this is about.

                               

                              kernel[0]: tcp_connect: inp=0x98c84df5952d8351 err=EADDRINUSE

                               

                              I will do an OS restore and see what happens.

                              • 12. Re: Second FM Server crash.
                                taylorsharpe

                                FYI, if this is a production system and you really need it working, you might want to contact a Mac OS X Server business consultant.  There are a lot of Apple Certified Consultants, but most just do consumer level stuff.  But there are a group of ones that are certified and work on Mac OS X Server installations for business.  So for my serious clients that use a Mac server, while I provide the FileMaker support, I pass on the OS management to one of these consultants.  You'll want to find one in your area, but the one I use in the Dallas, Texas area is www.mysupportpad.com.  These guys are the ones that usually bypass the Server.app and go right into terminal to manage things.  They can be very impressive.  I have had them recover and repair BIND, server mail and Open Directory that I would have just wiped out and started over.  This is analgous to the Microsoft Windows Server certified consultants with credentials like MCSA or MCSE, for which I refer my clients using Windows servers. 

                                • 13. Re: Second FM Server crash.
                                  bigtom

                                  Crashed thread data.

                                   

                                  Thread 59 Crashed:

                                  0   com.apple.CoreGraphics                  0x00007fff8e9657e1 CGDataProviderCopyData + 331

                                  1   com.apple.ImageIO.framework             0x00007fff86fa5182 CGImageReadCreateWithProvider + 229

                                  2   com.apple.ImageIO.framework             0x00007fff86fa5009 CGImageSourceCreateWithDataProvider + 202

                                  3   com.filemaker.support.framework          0x000000010d8c94aa Draco::Image2D::CreateImageFromData(Draco::ByteBuffer const*) + 40

                                  4   com.filemaker.support.framework          0x000000010d8c93e8 Draco::Image2D::Image2D(Draco::ByteBuffer const*, bool, Draco::QuadChar const&, Draco::UniversalPath const*, Draco::PhotoOrientation, Draco::PListMetadata*) + 304

                                  5   com.filemaker.support.framework          0x000000010d8c974e Draco::Image2D::LoadImage(Draco::ByteBuffer const&, Draco::QuadChar const&, Draco::UniversalPath const*, Draco::PhotoOrientation, Draco::PListMetadata*, bool) + 108

                                  6   com.filemaker.dbengine.framework          0x000000010c758757 Draco::DBBinaryData::GetExactThumbnailOnHost(Draco::CoordSize*, bool, Draco::CacheItem**, bool, unsigned int&) const + 1935

                                  7   com.filemaker.dbengine.framework          0x000000010c907713 Draco::RCDownloadThumbnail::Perform() + 609

                                  8   com.filemaker.dbengine.framework          0x000000010c90ffde Draco::RCNetworkStack::DispatchTransaction(Draco::RCConnection*, unsigned int, unsigned char*, unsigned int) + 1220

                                  9   com.filemaker.dbengine.framework          0x000000010c90fa1f RPO_i::Perform(unsigned int, char const*, unsigned int, OctetSeq&, OctetSeq_out) + 173

                                  10  com.filemaker.dbengine.framework          0x000000010c90bd19 _0RL_lcfn_07aabdf5dad7309d_70000000(omniCallDescriptor*, omniServant*) + 180

                                  11  com.filemaker.omniorb4.framework          0x000000010d1bcd4f omniCallHandle::upcall(omniServant*, omniCallDescriptor&) + 1099

                                  12  com.filemaker.dbengine.framework          0x000000010c90cf01 _impl_RCFMP12::_dispatch(omniCallHandle&) + 1697

                                  13  com.filemaker.dbengine.framework          0x000000010c90d278 non-virtual thunk to _impl_RCFMP12::_dispatch(omniCallHandle&) + 16

                                  14  com.filemaker.omniorb4.framework          0x000000010d2ae5cf omni::omniOrbPOA::dispatch(omniCallHandle&, omniLocalIdentity*) + 715

                                  15  com.filemaker.omniorb4.framework          0x000000010d2867c2 omniLocalIdentity::dispatch(omniCallHandle&) + 94

                                  16  com.filemaker.omniorb4.framework          0x000000010d216c06 omni::GIOP_S::handleRequest() + 330

                                  17  com.filemaker.omniorb4.framework          0x000000010d216865 omni::GIOP_S::dispatcher() + 323

                                  18  com.filemaker.omniorb4.framework          0x000000010d233ae6 omni::giopWorker::real_execute() + 1378

                                  19  com.filemaker.omniorb4.framework          0x000000010d233cd6 omni::giopWorker::execute() + 46

                                  20  com.filemaker.omniorb4.framework          0x000000010d2391e7 omniAsyncWorker::real_run() + 657

                                  21  com.filemaker.dbengine.framework          0x000000010c61b056 Draco::threadCreateInterceptor(omni::omniInterceptors::createThread_T::info_T&) + 22

                                  22  com.filemaker.omniorb4.framework          0x000000010d239492 omniAsyncWorker::run(void*) + 46

                                  23  com.filemaker.omniorb4.framework          0x000000010d2bde59 omni_thread_wrapper + 148

                                  24  libsystem_pthread.dylib                 0x00007fff90061899 _pthread_body + 138

                                  25  libsystem_pthread.dylib                 0x00007fff9006172a _pthread_start + 137

                                  26  libsystem_pthread.dylib                 0x00007fff90065fc9 thread_start + 13

                                   

                                   

                                  Thread 59 crashed with X86 Thread State (64-bit):

                                    rax: 0x0000000000000000  rbx: 0x00001485bb01d000  rcx: 0x00007fff8baf3901  rdx: 0x0000000000001000

                                    rdi: 0x0000000000000103  rsi: 0x000000010d5bd000  rbp: 0x0000000118ae5ae0  rsp: 0x0000000118ae5a60

                                     r8: 0x0000000000000010   r9: 0x00000000fffffff0  r10: 0x00007fff8af5dca6  r11: 0x0000000000000206

                                    r12: 0x00007fa9a01df600  r13: 0x00002134b6ea3000  r14: 0x0000000000000000  r15: 0x000035bba9fc9000

                                    rip: 0x00007fff8e9657e1  rfl: 0x0000000000010206  cr2: 0x0000000000000000

                                   

                                  Logical CPU:     0

                                  Error Code:      0x00000006

                                  Trap Number:     14

                                  • 14. Re: Second FM Server crash.
                                    taylorsharpe

                                    The log files like that don't mean much to me.  I would reinstall the OS from recovery mode and reinstall the Filemaker server.  Hopefully you have good backups to just make sure.  But I would not consider this machine stable anymore. 

                                    1 2 Previous Next