1 2 3 4 Previous Next 56 Replies Latest reply on Oct 18, 2015 1:15 PM by jfletch

    FileMaker Offsite Backups and large Remote Container storage

    taylorsharpe

      I have worked up a backup system I'm testing on a development server for off site backups.  It would be nice if StandBy server would work this way, but I understand it is really only designed for another computer on the same subnet. 

       

      I used to use 360Works' SafetyNet, which was pretty slick.  It basically just watches a file or folder and backs it up to the Amazon cloud.  Pretty great idea except that it copies the entire database each time.  This is not bad if you have a medium sized database.  But if you have a large database, especially with a lot of remote container storage, then copying the whole file every time gets unwieldy if not impossible depending on your internet speeds.  It sure would be nice if it did incremental syncing. 

       

      What is needed are incremental backups, especially for the remote container storage where most of the data does not change.  One thing we know from FileMaker is that you do not ever want to scan, copy or otherwise touch a live FileMaker database.  But I'm pretty sure that only applies to the database and not the remote container storage (confirmation????). 

       

      I'm am using a Mac and like Carbon Copy Clone (CCC) which is a program that can copy a whole disk or, in my case, watch a folder and its subfolders and copy only the changes to a remote storage location (similar to SafetyNet).  What I have done is created a CCC task that watches the live remote container storage for this one file and does a backup each night only syncing the changed files.  This goes MUCH faster than copying the entire remote container storage. 

       

      Separately I wrote a schedule script using the MBS plugin on the server that looks for the latest backup of this file in the backups folder and copies that file to another folder on the server.  CCC watches that folder and backups up that FileMaker file each night (*.fmp12 file, not any remote container storage). 

       

      This seems to address the problem of a FM file with a lot of remote container storage and only syncing (copying/deleting) the changes from day to day and not the entire remote container storage.  Since it has a lot less to transfer, this process is working well across the internet for remote container storage.  I now get both the FM file and the remote container storage backed up off site each night. 

       

      Obviously the easiest thing is to just copy all of your backups, but as your remote container storage grows, the copying can take really long.  This is a work around to minimize how much has to be copied each time making remote backups work when it would take many hours to sync the entire remote container storage each night. 

       

      Probably the trickiest part was making the script to watch the backups folder and find the latest backup and copy that file to another folder.  And while I use MBS, you can use any server plugin that allows you to manipulate files and list folder contents.  Just remember the file you are copying, and the folder it is going to, must be accessible (read/write access) by fmserver, especially if you are on a Mac.  If someone is interested, I can post that script. 

       

      Note that CCC is good for connecting to other servers for syncing (e.g., SMB, AFP and maybe FTP), but it does not support Amazon storage, which has its own protocol. 

       

      I'm open to any better suggestions on off-site backups through the web where there is a limited bandwidth. 

        • 1. Re: FileMaker Offsite Backups and large Remote Container storage
          wimdecorte

          Taylor Sharpe wrote:

           

           

          What is needed are incremental backups, especially for the remote container storage where most of the data does not change.  One thing we know from FileMaker is that you do not ever want to scan, copy or otherwise touch a live FileMaker database.  But I'm pretty sure that only applies to the database and not the remote container storage (confirmation????).

           

          Wrong.  It is vital that you stay away from the RC storage.  It is FileMaker's to manage and when you touch it directly you will very likely at some interfere with what FMS needs to do.  Treat the RC storage just like other live FM files.

          • 2. Re: FileMaker Offsite Backups and large Remote Container storage
            wimdecorte

            I would set up a single dedicated backup schedule that keeps 0 old backup sets.  That takes the pain away from having to find the most recent, there will only be one and there is not going to be a date/time stamp on the backup folder.

             

            FMS itself will backup efficiently as per its hard-linking.  So then the only thing you need to do is run your off-site backup mechanism some time after FMS finishes with that particular backup schedule.  Easier than setting up folder monitoring.

            • 3. Re: FileMaker Offsite Backups and large Remote Container storage
              taylorsharpe

              Wim:  Wrong.  It is vital that you stay away from the RC storage.  It is FileMaker's to manage and when you touch it directly you will very likely at some interfere with what FMS needs to do.  Treat the RC storage just like other live FM files.

               

              I've experienced a lot of client issues I've had to fix with things like virus scans that have killed FM databases, but never anything with the remote container storage.  It makes sense why scans of files open by FileMaker such as the *.fmp12 file can get corrupted if two services look at it.  But the remote container files are only read and saved, but not really opened by FileMaker in an editable state.  You can see a preview of the document in a container field, but you can't edit it in FileMaker. 

               

              I know you are giving a conservative read on things, but I'm trying to figure out technically how FMS would have conflict with a backup service reading the same set of remote container files that aren't actually opened.  The worst case scenario is that both service read the same file, but neither are editing it, and that is commonly done on a computer. 

               

              It may be that my assumption that filemaker is only reading/writing and moving files is incorrect and that files viewed in a container field are treated as an open editable file.  That is what I would like to know, but probably only someone at FileMaker technical really knows. 

              • 4. Re: FileMaker Offsite Backups and large Remote Container storage
                taylorsharpe

                wimdecorte wrote:

                 

                I would set up a single dedicated backup schedule that keeps 0 old backup sets.  That takes the pain away from having to find the most recent, there will only be one and there is not going to be a date/time stamp on the backup folder.

                 

                FMS itself will backup efficiently as per its hard-linking.  So then the only thing you need to do is run your off-site backup mechanism some time after FMS finishes with that particular backup schedule.  Easier than setting up folder monitoring.

                 

                The hard linking is great if you're on the same file system.  What I am looking to do is sync the data at a remote service across the internet that does not share the same file system and hard linking will not work.  And I'm trying to do it without having to copy files over the internet that have not changed.  I only want changed files copied and deleted ones removed.  This makes the network much more efficient. 

                 

                The dedicated backup schedule that keeps 0 old backup sets would work too, but it creates another backup.  The MBS script just copies the latest daily backup and puts it in another folder.  But as I am writing this, I realize it is doing the same thing.  So Wim's suggestion is the better one except for one thing.  The backup includes all of the remote container files and I only want to copy the *.fmp12 file.  The remote container storage I want as a separate backup that syncs on changes in it.  However, every day I need to copy the entire *.fmp12, but don't want to also copy the remote container files when I do that.  So that is why I have it set up the way I do. 

                 

                Thanks for the thoughts and ideas, Wim.  It is appreciated. 

                • 5. Re: FileMaker Offsite Backups and large Remote Container storage
                  wimdecorte

                  Not sure what to say more about it.  You can either believe it or not. The absence of evidence Is not evidence of absence.

                  In other words: the fact that you have not been bitten by meddling with the RC data files and folders does not mean it is not a bad idea.

                   

                  FMI engineers have said so in a few under-the-hood sessions at Devcons and it makes a lot of sense.  RC data is not a purely document management system.  It is to allow FMS to keep its data in an efficient way and do efficient backups.  But it still is pure FM data.  Just like its normal records.

                  • 6. Re: FileMaker Offsite Backups and large Remote Container storage
                    wimdecorte

                    Taylor Sharpe wrote:

                     

                    wimdecorte wrote:

                     

                    I would set up a single dedicated backup schedule that keeps 0 old backup sets.  That takes the pain away from having to find the most recent, there will only be one and there is not going to be a date/time stamp on the backup folder.

                     

                    FMS itself will backup efficiently as per its hard-linking.  So then the only thing you need to do is run your off-site backup mechanism some time after FMS finishes with that particular backup schedule.  Easier than setting up folder monitoring.

                     

                    The hard linking is great if you're on the same file system.  What I am looking to do is sync the data at a remote service across the internet

                     

                    You're missing the point.  FMS will back up efficiently with hard linking to the backup location that IS on the same file system.  Since that runs on a schedule you can then grab those files for an incremental backup once FMS is done with its backup.  You indicated that part of the issue was to be able to identify the latest FMS Backup.  When you have a schedule that stores 0 old sets, you never have to worry about that.  It's is always going to be the exact same folder.

                    • 7. Re: FileMaker Offsite Backups and large Remote Container storage
                      CICT

                      Where FileMaker have got this wrong is that they only offer the option to backup the container fields RC folders globally in the Folder tab within FMS admin. It needs to be within the Schedules tab that would allow the option to run and retain more frequent database backups and less frequent static data backups stored in the RC folders.

                       

                      Any software intrusion into the RC folder during operation will cause all sorts of problems, often unexpected quitting of FileMaker Pro.

                       

                      However, as the amount of data stored on our cloud servers does affect our costs, we took the decision (reluctantly) to exclude the Gbs of data in the RC folders from the FMS backup and use 3rd party software to backup the RC folders remotely during the quiet hours (small window due to use from Asia to the US). I appreciate we could use 0 old backups and build our own incremental backups with 3rd party software, but we're still keeping more copies that we wish to.

                       

                      Don't get me wrong, we absolutely love the external storage of the new(ish) container fields. None of the above is ideal, but until FileMaker allow schedules to opt in or out of the RC folder backups, we'll stay with what has worked for us since v12 was released.

                       

                      Regards

                       

                      Andy

                      • 8. Re: FileMaker Offsite Backups and large Remote Container storage
                        wimdecorte

                        CICT wrote:

                         

                        Where FileMaker have got this wrong is that they only offer the option to backup the container fields RC folders globally in the Folder tab within FMS admin. It needs to be within the Schedules tab that would allow the option to run and retain more frequent database backups and less frequent static data backups stored in the RC folders.

                         

                        Eh?

                         

                        It does not sound line you grasp the nature of the FMS backup mechanism.  The hard-linking feature of the FMS backups will give exactly what you are after.

                        • 9. Re: FileMaker Offsite Backups and large Remote Container storage
                          CICT

                          Hi Wim

                           

                          No I understand the hard link concept, various backup software uses differential or incremental backups to achieve something similar. Our issue is that have to make all backups available for download to our clients as part of their business continuity plan and getting the latest versions of a full complement of RC files to each location brings us back to Taylor's original post.

                           

                          Therefore we have implemented the simplest solution. If FileMaker allowed us more control over the backup within the Schedules rather than a global setting would give us more flexibility.

                          • 10. Re: FileMaker Offsite Backups and large Remote Container storage
                            BruceRobertson

                            You have implemented the "simplest" solution or the guaranteed to fail solution?

                            • 11. Re: FileMaker Offsite Backups and large Remote Container storage
                              CICT

                              Wow Bruce, that is a provocative question.

                               

                              Guaranteed to fail? It is too late to continue here now, but I look forward to reading your expansion of this in the morning. And  welcome your suggested solution to getting complete backups with Gbs of supporting files to clients across 5 continents on a daily basis.

                               

                              We live to learn!

                               

                              Kind regards

                               

                              Andy

                              • 13. Re: FileMaker Offsite Backups and large Remote Container storage
                                wimdecorte

                                CICT wrote:

                                 

                                Hi Wim

                                 

                                No I understand the hard link concept, various backup software uses differential or incremental backups to achieve something similar. Our issue is that have to make all backups available for download to our clients as part of their business continuity plan and getting the latest versions of a full complement of RC files to each location brings us back to Taylor's original post.

                                 

                                Therefore we have implemented the simplest solution. If FileMaker allowed us more control over the backup within the Schedules rather than a global setting would give us more flexibility.

                                 

                                 

                                I'm not entirely sure I understand what the problem is in your setup...  What is "the simplest solution"?

                                 

                                FMS backs up in a very efficient fashion, minimizing the required disk space and the time required to complete the backup - the more RC data you have, the more efficient that backup is.

                                 

                                Making all of the FMS backups available to your clients is a different issue.   You seem to be hinting at a problem of incomplete backups but I don't see that.  Each FMS backup is a complete set, including all RC data.

                                (unless you use the new FMS14 option to NOT include RC data in each backup - which quite frankly I do no get why anybody would go that way)

                                • 14. Re: FileMaker Offsite Backups and large Remote Container storage
                                  electon

                                  If the file has not changed only a hard link will be created.

                                  I understand from your white paper that the hard link is to the last backup set.

                                  http://www.soliantconsulting.com/sites/default/files/blog-files/FMS12_Backups_0.pdf

                                   

                                  After initial backup: If the file changes, filemaker server will create a new copy of the entire file, right?

                                  Will the container data be dealt with per record / container separately?

                                  Meaning: if only one container content has changed the rest will be hard linked even if the file has changed?.

                                   

                                  If that's the case then the maximum extra container backup storage is equal to all the container data in RC folder.

                                   

                                  I think what you're trying to say is don't sync the live RC Data but the one in the Backup Folder after a scheduled backup is complete.

                                  Both folders are basically the same since it's a backup and you will avoid the issue of touching the live system.

                                  1 2 3 4 Previous Next