10 Replies Latest reply on Jul 22, 2014 9:43 AM by wimdecorte

    Needing an error reference for Server 12 that actually tells me what's going wrong

    Michael Frankel

      Perhaps I missed the memo somewhere, but I have not been able to find a reference or technical document that not only tells me the basic, elemental description of error codes from FileMaker Server 12 (which I can easily find on-line) to a more in-depth, technical description of what causes (or might cause) some or all of these errors.

       

      Does such a document or reference exist? I have searched, repeatedly, on the web and on the FileMaker website with no success.

       

      Any assistance on this would be much appreciated.

       

      Thanks,

      --

      Michael Frankel

      Wizard Consulting Group, Inc.

        • 1. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
          wimdecorte

          I think your expectation is off.

           

          Anyone one of the reported error codes can be caused by a multitude of underlying factors.  The error code gives you a clue, but not the solution.  Sometimes the clue is obvious sometimes it is not.

          • 2. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
            wimdecorte

            To expand on that: there can never be a definitive document that outlines all of the possible scenarios that can lead up to an FMS error code.  That document would perpetually be out of date.

             

            What particular error are you struggling with?

            • 3. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
              Michael Frankel

              Hi Wim -

               

              I appreciate your response. While I do understand that the reasons for errors can be quite voluminous, some of the error messages could be much more instructive. The two that I'm dealing with are 685 (can't delete a previous backup) and 8 (can't replace a file).

               

              I suspect that 685 is caused because FileMaker Server creates hard-links (sym-links on Mac OS X) that point to the original file and are used to reduce the backup of the same file in a redundant fashion. In the solution I'm working on, there are thousands of remote containers and I suspect that previous backups can't be deleted because the old backup folder contains the original file that is referenced in a more recent backup. It seems to me, however, that FileMaker Server could handle this more elegantly - perhaps I'm expecting too much.

               

              The other error, error 8, is a mystery. I do not know why it's happening and have not found any indication of what's going on. The message in the log is Unable to back up database <database_name>; can't replace item with that name on destination <backup_file_path>. I've checked permissions on that folder and volume, and they should allow this action to take place.

               

              Any ideas on what might be causing that?

               

              Your input is much appreciated.

               

              --

              Michael Frankel

              Wizard Consulting Group, Inc.

              • 4. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
                wimdecorte

                sym links are very different from hard links so be careful with what you do with them.

                Hard links point to the underlying inode on the physical hard disk, sym link point to the file name, which is an abstraction of the inode.

                Any assumptions you make here can contribute to the errors.

                 

                 

                Do you get these errors on progressive backups or regular backups?

                Are any of those locations shared through the OS?

                 

                Anything special about the backup destination?  Are those shared folders, removable media?

                • 5. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
                  Michael Frankel

                  Hi Wim -

                   

                  Forgive any  errors in my terminology. I did some reading and now understand that a hard-link points to the directory tree (i-node) and a symlink points to the file name (at least that's how I read it). I'm simply pointing out that FileMaker Server does something with the files so that the same file does not get backed up twice. It seems that, based on how it behaves, that it is using symbolic links because a hard-link would not break if the source file was deleted but a symlink would. If you know differently, please tell me.

                   

                  As far as my backups, they are all regular backups. I have multiple backups running - one backup, done daily, goes to the boot drive; another backup, done weekly, goes to an external USB drive; a third backup is a clone backup and also goes to an external USB drive. There is a shared folder on the external drive, however NONE of the backup folders are shared. The server machine is a MacMini and my client says they cannot afford the downtime to replace the boot drive with a larger one.

                   

                  I'm also very wary of doing this with the large number of remote containers. Personally, I would prefer to move the entire server over to another computer (they have a spare Mac Pro) but again, I'm very wary of what would happen to all of the remote containers. I estimate I have over 70,000 of them. I've heard horror stories of others who have tried and I'm not anxious to share in their experience.

                   

                  Anyway, I appreciate your input in the event you might know what Error 8 is.

                   

                  Thanks,

                  --

                  Michael Frankel

                  Wizard Consulting Group, Inc.

                  • 6. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
                    wimdecorte

                    wizardconsultinggroup wrote:

                     

                    It seems that, based on how it behaves, that it is using symbolic links because a hard-link would not break if the source file was deleted but a symlink would. If you know differently, please tell me.

                     

                    The FMS backup mechanism does not use symbolic links, it uses hard links.  It is easy enough to verify, I showed how in my 2012 devcon presentation, I can dig up those slides if you want them.

                     

                    A backup file does not have a link to the original file, the hard links are strictly between backed-up files of the same backup schedule.

                    So if you delete the original source file, no backup is ever going to be affected (that's kinda the whole point of a backup).

                     

                    Of your 3 backup schedules, which one is giving you the errors? 

                    • 7. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
                      Michael Frankel

                      Hi Wim -

                       

                      I realized I actually have a 4th backup schedule, one which uses the OS X command line to do a separate backup (to the external drive) specifically for Time Machine. I noticed that doing that backup using FileMaker Server in the Java GUI (this is still FMS12), the Time Machine backup kept ballooning up in size, eventually filling up the entire backup (this Time Machine backup is done over the network at night). Using a command-line version and pointing Time Machine to only that single backup, it does not do this. I'm not sure why this happens, but perhaps Time Machine resolves the hard-links and copies the original file anyway.

                       

                      In any case, the two backups that are generating errors are the daily backup, which runs at 11pm every night (onto the boot drive) and generates an error 685 (can't delete previous backup folder) and the backup for Time Machine, which runs every 6 hours and generates an error 8 (can’t replace item with that name on destination).

                       

                      If the backups are using hard-links, then there would not be a problem with removing older files (error 685) so I don't know what would cause this. Since a hard-link is simply a pointer to a directory location on the hard-disk, deleting the "original" or older backup folder should not cause a problem since the original file is still intact and should not affect the live database files. Do you concur?

                       

                      As far as the error 8, unable to replace an item, I need to verify that it happens on every run of this script. Perhaps the script is bumping into the actual Time Machine backup process.

                       

                      Your feedback and input is invaluable, Wim, and is much appreciated.

                       

                      Thanks,

                      --

                      Michael Frankel

                      Wizard Consulting Group, Inc.

                      • 8. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
                        wimdecorte

                        wizardconsultinggroup wrote:

                         

                         

                        If the backups are using hard-links, then there would not be a problem with removing older files (error 685) so I don't know what would cause this. Since a hard-link is simply a pointer to a directory location on the hard-disk, deleting the "original" or older backup folder should not cause a problem since the original file is still intact and should not affect the live database files. Do you concur?

                         

                        I don't.  I think you are over-thinking the hard linking and looking in the wrong place for what is going on.

                        EVERY single file in OSX is a hard link; that's how deep the mechanism is embedded into the OS.  So trust me when I say that FMS backups are using hard links where it can.

                        When the OS deletes a file it does not care whether it is hard link file or not (because they all are).

                         

                        Something is clearly preventing FMS from cleaning up after itself and deleting the previous backup folder.  If you look into that old backup folder, how many of the files remain?  All of them, just a few?

                        Of those remaining, what do their permissions look like (from Termina, not from Finder's Get Info).

                         

                        You mentioned 70,000+ files.  How are those structured?  All in one folder?  A subfolder per file?  Both OSX and Windows have trouble once you put a lot of files in one folder.

                        • 9. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
                          Michael Frankel

                          Hi Wim -

                           

                          Agreed - overthinking it.

                           

                          To answer your question, the container data is definitely structured. For the table with the most items, there is one folder per record and one sub-folder for each of the three types of images that can be stored. Typically there are no more than 10 images per record (folder).  There was a time when it was not so structured, however I changed that 3-4 months ago. The one thing I noticed is that, for the folders that would not delete, the new structure was not present. I'm not sure why that's the case.

                           

                          And, I checked this morning, and the most recent backup has over 120,000 remote container files, most of them being JPEG images.

                           

                          I  checked the permissions on the oldest folder that FileMaker Server said it could not delete. I checked the main backup folder; it had permissions of fmserver (Read & Write), fmsadmin (Read & Write), everyone (Read only). I also traced down the permissions through each level of the sub-folder structure (Databases > Folder Name from FMS > RC_Data_FMS > Database_Name > Files > Database_Name...) and every one had the same permissions. Once I got to the actual files themselves, they had the same permissions except that the permissions for everyone were Read & Write. This would seem to indicate that permissions are not the issue.

                           

                          Since these are backup files and the originals would not be affected, would there be any harm in manually deleting the older backup folders?

                           

                          Thanks,

                          --

                          Michael Frankel

                          Wizard Consulting Group, Inc.

                          • 10. Re: Needing an error reference for Server 12 that actually tells me what's going wrong
                            wimdecorte

                            wizardconsultinggroup wrote:

                             

                             

                             

                            Since these are backup files and the originals would not be affected, would there be any harm in manually deleting the older backup folders?

                             

                             

                            No harm whatsoever.