5 Replies Latest reply on Aug 21, 2012 11:54 AM by KeithProctor

    External /Remote Container Data Backups

    Tim_Ballering

      So I was excited about the new External Container Data as I thought this would allow us to backup only the files added since the last back up. I am guesing this is possible, but fear I have my set up incorrect.

       

      Here is what we are doing:

       

      We are making a separate backup once per day. The problem is the image files within the FMS created back up have the creation and modification date of the time of the backup rather than the time of the original file creation. Therefore our incremental backup is backing up every image file rather just those that were created after last night's back up.

       

      The actual process I fell upon is using Applescript to parse out the path to the RC_Data_FMS folder of the daily backup and then rsync via a do shell script to our backup volume.

       

      The only other thing that comes to mind is to rsync the actual RC_Data_FMS in the live database folder. But that scares me as I do not like to allow anyone or anything to touch the database folder.

       

       

      Please show me the errors of my ways.

       

      Thanks

       

      Tim Ballering

        • 1. Re: External /Remote Container Data Backups
          wimdecorte

          Hi Tim,

           

          Rule # is to let nothing whatsoever touch the live files, including the remote container data.  I would change the whole process to push a FMS backup out to a network share where your other backup software can work with it.  You can then do the rsync against that folder.

          • 2. Re: External /Remote Container Data Backups
            Tim_Ballering

            Wim

             

            First let me thank you again for your DevCon 2012 presentation on back ups.  I imagine we would have gotten into serious problems some day in the  future with our lack of understanding of hardlinks and how FMS utilizes them.  It was one of the most important aspects of FMS 12 I learned while there.

             

            As for my back ups, I am currently doing as you describe, having FMS make a back up - then using rsync to put the back up on remote volumes. 

             

            The problem is the FMS back up changes the created mod timestamp to the time the back up was created.  Then rsync replaces all 680,000 image files with "new" ones based on the mod time. 

             

            I do not believe there is an option for FMS backups to retain the original creation & mod timestamp.  I do not know rsync well, so maybe there is another setting that would only back up the actual new files. 

             

            Other than taking a looooong time to copy all the files to offsite file servers, I fear there is a lot of unnecessary wear on the server and back up hard drives as well.  While disks are cheap, HDD failure is not.

             

            Tim

            • 3. Re: External /Remote Container Data Backups
              KeithProctor

              Tim,

               

              Thanks for the information that you provided.  I have a couple of servers using backup and iBackup and I saw that there timestamps were original timestamps from the first backup and these timestamps go back many months. So the issue is not a continual one but simply a first time backup issue.  As is we don't have enough information to understand your full issue as I don't know what your rsync command is actually doing.  I do believe this comes down to your rsync command so here is a couple of examples.  I would expect your rsync command to compare against the very last backup of a specific name and your final destination folder for the sync.  I would never expect  an rsync to compare against the hosted files and the last backup to work as a good chunk of the files have changed since those to locations in time. 

               

              Other notes:

               

              Also you should never compare against the live hosted files.  You can compare against the folders that are being backed up to but this can be dangerous if the schedule for a particular backup is running while your external process is running.  The only way to do this is to guarantee that these two processes NEVER compete for resources (the actual backup files).  I'll give an example:

               

              Lets say you have 10g worth of data and your backup is guaranteed to complete a full backup in 1/2 an hour.  This is important that we consider the time for a FULL backup rather then FileMaker Servers optimized hard link style backup which could take seconds. You can think about your growth rate for the database and pick a safe time such as 2 hours later to perform your offsite backup.  In my case I create a schedule that only has 1 copy in the backup folder using the keep 0 copies option in the schedule.  This creates 1 copy but the 0 means to not place a timestamp after the schedule name.  I know that this backup will take till say 5 AM in the morning to complete.  I then tell our IS&T department to backup this folder many hours later in the day.  There portion of the backup takes about 45 minutes.  In my case our IS&T department grabs the backup around 9 PM or 11 PM.  So we are totally safe on this folder as FileMaker server finishes with the backup at 5 AM and there are at least 16 hours between the IS&T backup and IS&T is completed at least 5 hours before FileMaker Server wants access to the folder.

               

              I have reported an issue, on your behalf, regarding the first backup changing the dates but I don't believe that this issue should get in the way of rsync working.  Given that you point rsync to the correct set of folders.  I'm sort of guessing at some of this so, if I didn't get something write, please let me know if I didn't guess correctly.

              • 4. Re: External /Remote Container Data Backups
                Tim_Ballering

                Okay this makes sense. 

                 

                You state FMS initial back up  contents are timestamped with the creation date of the first back up.  Looking over our end of month back up,  that appears correct.

                 

                Our server returns a 685 error and does not completely remove the last back up, creating a huge number of remnant back ups if allowed to.  There is a lot of talk here about this issue. https://fmdev.filemaker.com/message/92039#92039

                 

                Our work around is to remove the RC data folder after we rsync to remote storage via AppleScript. This way we get one back up that we can target for moving.  This explains why the timestamps are causing an issue in my case as every back up for this schedule is in essence a new one.

                 

                So I guess once the 685 rproblem is solved the back up problem will no longer exixt.

                 

                Tim

                • 5. Re: External /Remote Container Data Backups
                  KeithProctor

                  The 685 issue is a separate issue and is most like caused by another application working on the file.  When this issue occurs (IOW within seconds of it happening)  perform the following commands

                   

                  cd into the folder where the file in question  then do

                  sudo lsof | grep filename

                   

                  This command will report the application or applications that is using the file.  This is kind of time sensitive so you should keep the server having the issue up and running.  You can then use that information to take the right action If you don't know what application it is you can connect with FileMaker Technical Support and they should be able to help you figure out what application it is.

                   

                  Keith