There are always possibilities of hardware failures like hard drives or bad memory. You can always test the memory and the drives as a first step. How is your storage setup (e.g., internal drives, external drives, RAID, SAN, types of connections)? Have you had any power outages or possible lightning strikes? Do you have a backup power supply?
Regarding the software side of things, how about some more details. What operating System are you using and what version of FileMaker Server software? Do you have any plugins installed? If so, which ones and what versions?
Have you run consistency checks and maybe recovery of the databases to see if there are any errors?
Those are some starting points. There are other tools like FMDIFF and Base Elements and looking through the DDR if these don't find something to start with. Let us know and we'll help with ideas to check on.
Thanks for the reply.
Setup is a latest Mac Mini Server. 2x256 Apple SSD RAID1 internal. Memory is 16GB installed by Apple. Backups are external USB drives, progressive and 2x scheduled backups. Have UPS. No power outages. No lightning strikes. OSX 10.9.2, no server side plugins, FileMaker Server 13.
To get the Server running again I tried starting and stopping FMS from the command line. This resulted in the same behavior of the admin panel. I had to shut the server machine down and restart. FMS launched on restart and then was working again.
The hardware seemed to checkout so far, but I will see if there is anything else I might be able to check them with. I will keep you posted and I appreciate the help. I really do not need this going down in the middle of the day. Thankfully, both times were over night and I was able to get it up and going by working hours.
Fusion drive in that Mac Mini?
The fact that the admin console does not respond properly does not mean that the database engine has crashed. The Admin Server and the Database Engine run in separate processes and do not affect each other. So check the FMS logs and the OSX system logs for clues, that's the best place to start.
Anything else running on that machine? Anti-virus? Online backups? ...
Probably not a Fusion drive since that would be 1TB. 256 GB SSDs are offered in the Apple store and I would prefer them over Fusion drives in a server even though they lack storage.
Wim brings up some good ideas on if other services are scanning the live FileMaker databases. The common Apple culprits are Spotlight and Time Machine. Make sure those services are excluding the live FileMaker database folder. They can scan the backups, just not live FileMaker databases. Also true on any anti-virus software if you have any. Just make sure other services (or youself) don't ever touch live open FM databases.
It sounds like you have a fairly good setup with UPS, Mavericks, RAID storage and 16 Gigs of RAM. You might make sure you are up to FileMaker 18.104.22.1685 update if you haven't already.
Once a any of the FileMaker services starts to act weird, I usually do a complete system reboot to clear out any problems. I know... probably habit from working on so many Windows machines. But it seems to work most of the time.
There is always a weird chance you have bad memory. You could use the Apple Hardware test software to test it although its not as good as most 3rd party ones like Memtest86 (free download and you setup a bootable USB drive to test the RAM).
If you're doing hardware tests... does your UPS give stats on power. Is your voltage running high low or anything else weird like that?
Just crashed again. Got the message that the admin console was limited to Status and Logs.
The one thing I cannot seem to turn off is the Notification center.
I was able to find a failuer in the console this tim and it looks like some sort of memory allocation problem. Maybe I just need to allocate more memory or the memory is bad? Below are the three entries from the OSX Console related to this.
fmserverd: fmserverd(210,0x118ae7000) malloc: *** mach_vm_map(size=59074891485184) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
ReportCrash: DebugSymbols was unable to start a spotlight query: spotlight is not responding or disabled.
ReportCrash: Saved crash report for fmserverd version ??? to /Library/Logs/DiagnosticReports/fmserverd_2014-06-04-122925_Server-Mac-mini.crash
So its not just FileMaker service crashing... Spotlight crashed too. Hmmmmm.... how about trying an upgrade to Mac OS X 10.9.3 and see if that helps. I know 10.9.2 had some weird things that even kept it from booting in Safe mode. But then again, it could be the memory.
I would turn your cache down in the FileMake Admin console since that is how much info can be stored in RAM before being dumped back to the drive since you are having crashing issues. Keep that number small so you don't loose much at least until you figure out what is going on.
Actually, if upgrading to 10.9.3 doesn't work, I would boot into the restore mode (boot holding down Apple-R) and restore the operating system. It won't remove FileMaker, but will reinstall all OS components. Then run the upgrades to the OS. That would clear out any corruption in the OS.
Spotlight crashed? It should be disbaled.
Looks like the previous crash was some sort of Spin problem with finder.
Mavericks has problems if you disable spotlight. So you don't want to disable it, just make sure it doesn't scan live FileMaker files.
What other services or applications are running at the same time?
But I seriously would be considering the OS recovery at this point.
I also had these two errors two minutes before this most recent crash. I have really tried to set this up properly.
secd: SecErrorGetOSStatus unknown error domain: com.apple.security.sos.error for error: The operation couldn’t be completed. (com.apple.security.sos.error error 2 - Public Key not available - failed to register before call)
secd: securityd_xpc_dictionary_handler EscrowSecurityAl DeviceInCircle The operation couldn’t be completed. (com.apple.security.sos.error error 2 - Public Key not available - failed to register before call)
No other services or apps are open except Safari and Console.
There are a lot of these in the console. Do you know what this is about.
kernel: tcp_connect: inp=0x98c84df5952d8351 err=EADDRINUSE
I will do an OS restore and see what happens.
FYI, if this is a production system and you really need it working, you might want to contact a Mac OS X Server business consultant. There are a lot of Apple Certified Consultants, but most just do consumer level stuff. But there are a group of ones that are certified and work on Mac OS X Server installations for business. So for my serious clients that use a Mac server, while I provide the FileMaker support, I pass on the OS management to one of these consultants. You'll want to find one in your area, but the one I use in the Dallas, Texas area is www.mysupportpad.com. These guys are the ones that usually bypass the Server.app and go right into terminal to manage things. They can be very impressive. I have had them recover and repair BIND, server mail and Open Directory that I would have just wiped out and started over. This is analgous to the Microsoft Windows Server certified consultants with credentials like MCSA or MCSE, for which I refer my clients using Windows servers.
Crashed thread data.
Thread 59 Crashed:
0 com.apple.CoreGraphics 0x00007fff8e9657e1 CGDataProviderCopyData + 331
1 com.apple.ImageIO.framework 0x00007fff86fa5182 CGImageReadCreateWithProvider + 229
2 com.apple.ImageIO.framework 0x00007fff86fa5009 CGImageSourceCreateWithDataProvider + 202
3 com.filemaker.support.framework 0x000000010d8c94aa Draco::Image2D::CreateImageFromData(Draco::ByteBuffer const*) + 40
4 com.filemaker.support.framework 0x000000010d8c93e8 Draco::Image2D::Image2D(Draco::ByteBuffer const*, bool, Draco::QuadChar const&, Draco::UniversalPath const*, Draco::PhotoOrientation, Draco::PListMetadata*) + 304
5 com.filemaker.support.framework 0x000000010d8c974e Draco::Image2D::LoadImage(Draco::ByteBuffer const&, Draco::QuadChar const&, Draco::UniversalPath const*, Draco::PhotoOrientation, Draco::PListMetadata*, bool) + 108
6 com.filemaker.dbengine.framework 0x000000010c758757 Draco::DBBinaryData::GetExactThumbnailOnHost(Draco::CoordSize*, bool, Draco::CacheItem**, bool, unsigned int&) const + 1935
7 com.filemaker.dbengine.framework 0x000000010c907713 Draco::RCDownloadThumbnail::Perform() + 609
8 com.filemaker.dbengine.framework 0x000000010c90ffde Draco::RCNetworkStack::DispatchTransaction(Draco::RCConnection*, unsigned int, unsigned char*, unsigned int) + 1220
9 com.filemaker.dbengine.framework 0x000000010c90fa1f RPO_i::Perform(unsigned int, char const*, unsigned int, OctetSeq&, OctetSeq_out) + 173
10 com.filemaker.dbengine.framework 0x000000010c90bd19 _0RL_lcfn_07aabdf5dad7309d_70000000(omniCallDescriptor*, omniServant*) + 180
11 com.filemaker.omniorb4.framework 0x000000010d1bcd4f omniCallHandle::upcall(omniServant*, omniCallDescriptor&) + 1099
12 com.filemaker.dbengine.framework 0x000000010c90cf01 _impl_RCFMP12::_dispatch(omniCallHandle&) + 1697
13 com.filemaker.dbengine.framework 0x000000010c90d278 non-virtual thunk to _impl_RCFMP12::_dispatch(omniCallHandle&) + 16
14 com.filemaker.omniorb4.framework 0x000000010d2ae5cf omni::omniOrbPOA::dispatch(omniCallHandle&, omniLocalIdentity*) + 715
15 com.filemaker.omniorb4.framework 0x000000010d2867c2 omniLocalIdentity::dispatch(omniCallHandle&) + 94
16 com.filemaker.omniorb4.framework 0x000000010d216c06 omni::GIOP_S::handleRequest() + 330
17 com.filemaker.omniorb4.framework 0x000000010d216865 omni::GIOP_S::dispatcher() + 323
18 com.filemaker.omniorb4.framework 0x000000010d233ae6 omni::giopWorker::real_execute() + 1378
19 com.filemaker.omniorb4.framework 0x000000010d233cd6 omni::giopWorker::execute() + 46
20 com.filemaker.omniorb4.framework 0x000000010d2391e7 omniAsyncWorker::real_run() + 657
21 com.filemaker.dbengine.framework 0x000000010c61b056 Draco::threadCreateInterceptor(omni::omniInterceptors::createThread_T::info_T&) + 22
22 com.filemaker.omniorb4.framework 0x000000010d239492 omniAsyncWorker::run(void*) + 46
23 com.filemaker.omniorb4.framework 0x000000010d2bde59 omni_thread_wrapper + 148
24 libsystem_pthread.dylib 0x00007fff90061899 _pthread_body + 138
25 libsystem_pthread.dylib 0x00007fff9006172a _pthread_start + 137
26 libsystem_pthread.dylib 0x00007fff90065fc9 thread_start + 13
Thread 59 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x00001485bb01d000 rcx: 0x00007fff8baf3901 rdx: 0x0000000000001000
rdi: 0x0000000000000103 rsi: 0x000000010d5bd000 rbp: 0x0000000118ae5ae0 rsp: 0x0000000118ae5a60
r8: 0x0000000000000010 r9: 0x00000000fffffff0 r10: 0x00007fff8af5dca6 r11: 0x0000000000000206
r12: 0x00007fa9a01df600 r13: 0x00002134b6ea3000 r14: 0x0000000000000000 r15: 0x000035bba9fc9000
rip: 0x00007fff8e9657e1 rfl: 0x0000000000010206 cr2: 0x0000000000000000
Logical CPU: 0
Error Code: 0x00000006
Trap Number: 14
The log files like that don't mean much to me. I would reinstall the OS from recovery mode and reinstall the Filemaker server. Hopefully you have good backups to just make sure. But I would not consider this machine stable anymore.