Product and version: FileMaker Server 16.0.3, 16.0.4
OS and version: macOS Sierra 10.12.6
Hardware: Mac Mini with SSD boot drive (2011, 2012)
This has happened on 2 different servers at 2 different sites with 2 different sets of databases so it is not a one-off so I think its worthwhile to describe the situation because both occurrences resulted in significant loss of data despite multiple levels of backup. Here's is the sequence of events that occurred at both sites.
FileMaker Server 16 (in one case 16.0.3, in another 16.0.4) running on a Mac Mini (one from 2011, one from 2012) both with SSD main drives and 8-16GB memory. The servers had been running for more than a month. Both servers had Time Machine enabled (with FileMaker Server folder and FileMaker backups excluded), had multiple FileMaker backup schedules to both internal and external drives. On one of them they swapped the external drives for off-premise backup on the other it was set up to back up to CrashPlan from a separate backup saved to the boot drive. MacOS Server app was *not* installed on either machine.
With no prior warning emails or messages the machines stopped responding. When rebooted the databases were damaged. Checking the FileMaker backups found that no backups had run for a week in one case and 45 days in the other. In the most severe example the machine failed on 5/1, the live database after recovery had no updates since 4/6 and no backups had run since 3/13. In addition the Time Machine backups had also stopped running on the same day the FileMaker backups stopped. There were no system.log entries after 3/13 or entries in the FileMaker logs after 3/13. The only indication that there were any problems prior to the crash was the *lack* of notification emails for successful backups (which nobody noticed). No I/O errors were logged in the system.log and there were no obvious log records to indicate the cause of the problem.
On the one machine data loss was 1 week, on the other it was 3 weeks.
My working theory is that somehow the machine or OS became unable to write to any disks. FileMaker Server continued to run until some point where either the machine or FileMaker could no longer continue. Since the 3/13 backup never completed no more backups ever started (or failed).
How to replicate: Unknown
Workaround (if any)
No workaround that I can think of other than paying more attention to the *lack* of schedule notifiation emails which had they been noticed would have resulted in less data loss.