So Thursday morning while at DevCon, I got a wake up call from my users on the east coast that no one could get into our database, that they were getting a message about script missing when they opened the file and then couldn't go anywhere. I opened it up and sure enough, all of the scripts in the file were gone, folders, separators, everything. I have no idea how this happened. I looked through the server logs and there are no warnings or errors that would indicate a problem occurred, no users accessing the file who shouldn't be. I'm one of 2 people with Admin access to the file, the other person was also at DevCon.
I restored the file from an hourly backup, but then later that day a record in one of the tables got corrupted.
I spent this afternoon/evening running different recoveries on copies of the file, all come back with no problems found, and it passes consistency checks. When I did a recovery on the file from Thursday (which I moved off that morning to an external drive to work with later), all the scripts came back (or at least I think all, they were completely out of order and most were not in the folder they belonged in). When I did recovery on the file with the 1 corrupted record, no errors, but the record was removed.
I decided to build a new server this afternoon right away, and migrated my solutions to it, to rule out possible hardware/OS problems.
Original Server: 2013 Mac Pro, macOS 10.12.6, FileMaker Server 16.0.1, files stored on local drive, backups to external drives.
Replacement Server: 2010 Mac Pro, macOS 10.12.6, FileMaker Server 16.0.2, files stored on boot drive, backups now to internal drives.
Has anyone else ever experience 1 or both of these types of corruption?
If I have any further problems, I'm going to create a clone of my solution and export/import the records into it, but that'll take a considerable amount of time with the amount of tables I have.