How do you make your back up copies?
When you restarted the server, did you first close the files using the Server Admin application?
I restarted the server by closing down FMS with the Server Admin application, and then restarted the computer.
FM Server is set to keep daily backups; those, and the original databases, are all backed up to an external disk by Time Machine, so I end up with a lot of redundant copies. I also periodically ftp to an off-site server. All of them are corrupted.
I checked the filesystem and permissions, and there was no corruption there.
Using Time Machine to back up your open files is not recommended. At best the backups of the open files will trigger a consistancy check when opened. At worst, they will be corrupted. That does not explain why backups of the closed files backed up by Time Machine would be damaged, however.
I have read warnings here in the forum from a FileMaker tech that backing up an open file can damage the original. I don't quite see how that could be possible and haven't gotten an explanation on that claim from a FileMaker tech....
Agreed - when I open any of these backup files, corrupt or not, FM does a consistency check because it had not been closed properly. But I thought this was Filemaker Server's normal behavior - it normally backs up open files, and then Time Machine just copies those to an external disk. Is this not normal? And is there another way of backing up the database without shutting it down every day? It's in use most of the time; but it would be great if FMS had an easy mechanism to tell when nobody was using it, close the databases, back up, and start up again. That's another thread, though. I just want to know how FMS can continue for three weeks to use, close and open a file that's so corrupted that none of the backups work. That doesn't seem to be related to the problem of backing up open files.
But I thought this was Filemaker Server's normal behavior
Opening back up copies should not trigger a consistency check unless the file was open at the time the back up copy was made. It doesn't sound like your back up software is backing up the copies created by Server's scheduled back up but is instead making copies of the open or "active" file. I could be wrong as weird things happen with damaged files, but I'd take a very careful look at how your back up software is set up. Using 3rd party software to copy the backup copies already produced by a server schedule should not result in files that need consistency checks when they open.
You may have two different issues involved here, the back up system and latent corruption in your active copy. Latent corruption that presists undiscovered for long periods of time, while unusual, is not unheard of nor unique to FileMaker databases. (I've had to roll back over a month of back ups with MS Access to get a clean copy of the DB...)
Here's a purely theoretical scenario to consider.
1) An open file is backed up by 3rd party software and the planets are in improper aligment at the instant the back up is made and so the file is seriously corrupted. Maybe a user was committing structural changes at the moment the file was copied by the back up software.
2) A problem with the active file is discovered and the file is eather replaced by a copy of the back up, data from the back up is imported into the active copy, or structural elements of the backup are copied or imported into the active copy. Hidden damage from the bad backup then piggybacks on that event to introduce hidden damage to your active copy of the file.
3) The damaged file is backed up on a regular basis--producing damaged backups and then an unknown further event trips over the hidden damage and your file is sufficiently damaged that now you can't open the file, and you find that your backups are damaged also.
I find the above to be a very unlikely series of events, but can't say it's impossible....
One small clarification:
I've been discussing backups and file corruption privately with someone that works for FileMaker Inc. They've checked and can confirm that backing up an open database file could produce a damaged copy, but will not damage the original. Either someone gave me wrong info or I misunderstood on that point...
Thanks -- you're absolutely right, that the backups made by Filemaker Server (and stored in the Data/Backups folder) were closed properly and don't trigger a consistency check, while the backups made by Apple's Time Machine of the open, active files do trigger a consistency check -- although then they always open properly, they're not corrupted, until Nov. 30 when the original was corrupted. The system I'm using backs up both the active files and the FMS backups, so lots of redundancy. But the basic problem is that since Nov. 30, both the original file and all the backups have been corrupted, and yet the server still continued to work, people still added data, all of which is now lost -- latent corruption, as you say. I don't know why it finally decided to fail, or what corrupted it in the first place - although that day, I added a new database to the server, which was linked to the corrupted file; and maybe that screwed something up. Perhaps coincidentally, all the databases on the server have the same user names and passwords except that last one which was added the day of the corruption: is it possible that linking a file with different username and passwords could corrupt the access of the original?
For the future I'll set up FMS to send an email when anything unusual happens - this wasn't turned on. And I'm writing scripts for each database that will export all the tables into tab-delimited text files every day. Would those text files be corrupted, too, or is it more likely to be something else? The message I'm getting only mentions access privileges, not data or index corruption. I'm rebuilding the database from the last good copy, too.
I'm also going to buy Stellar Phoenix, which is able to open the damaged file, and will let y'all know how that works.
although that day, I added a new database to the server, which was linked to the corrupted file
Did this require any change to the design of the corrupted file?
No, it didn't -- I linked from the new file into the corrupted one.
Is there a better way of managing user names and passwords through the server, than the one in Filemaker? I would prefer to have the databases be non-password protected, and have the server handle all that authentication.
Actually, I take that back -- I did link from the damaged file to the newly added one, which has a different password and user name set. Could that have caused the damage? Bad design if so.
I doubt that the passwords have anything much to do with this, but could be wrong. This specific error message is kinda of like the "blue screen of death" on older windows systems. In my personal experience, it tells you that something major is wrong with the file and that you are unlikely to be able to repair it with recover.
I'm only speculating, but if you made a structural change to the DB while it was live and you had a network glitch at the moment you were committing the change, it could have corrupted your file--so this is a possible explanation as to how it happened. The fact the damage was latent (hidden) is even more exceptionally bad luck, but what I am describing would appear to fit the known facts...
Well, I've got to figure out a better way of backing up and archiving, so I don't end up with corrupted archives. I can live with random glitches, but only if I'm notified immediately. I've turned on email notifications and hope that helps; and will export all tables as Unicode text every day and save those files; at least then I'm not losing the essential data if it all gets hosed. Many thanks for your help!
Bought Stellar Phoenix Filemaker Recovery yesterday, and it didn't work - it couldn't even open and recover a good filemaker file, much less this corrupted one. I have a tech support call in to them, but I'm not impressed so far.