1 2 Previous Next 19 Replies Latest reply on Sep 30, 2014 4:43 AM by wimdecorte

    FMS 12 External Storage, backup of large file sets

    CarstenLevin

      I really want to hear your views.

       

      The setup

      A customer with a very interesting training authoring system. Lots of materials, each with a lot of modules, each module with a lot of lessons and each lesson with a lot of training and testing elements. And each element has a lot of files (sounds, pictures and ...... video's)

       

      The solution is up and running. And the customer is happy.

       

      The data in the FMS Server External Storage foldes is +1 TB as of today. And it is growing. Will probably hit 3-5 TB when they are finished creating everything.

       

      We did a lot of testing with realistic loads etc. FileMaker can handle it - no problem here (we are using MySQL & Apache as the front UI for the many students, FileMaker is feeding the SQL database and the files is moved to the webserver on a nightly basis).

       

      The issue - backup

      We need a FileMaker backup every hour. No problem.

       

      Question

      • Is it possible to set up backup backup schedules taking only DB's and not files in external storage associated to the fmp db file?
      • Will it be OK to take backup of the files in the running FileMaker Server 12 external storage folder every night (incremental etc.). This will take less than 20 minutes every night. We will of course NOT backup the database files from the active db folders, just the External Storage files.

       

      I am looking forward to hearing you views and advices.

       

      Best regards


      Carsten

        • 1. Re: FMS 12 External Storage, backup of large file sets
          wimdecorte

          Carsten Levin wrote:

           

           

          • Is it possible to set up backup backup schedules taking only DB's and not files in external storage associated to the fmp db file?

           

          No, but it should not be too big of a worry, FMS will hardlink any remote container data file that has not changed since the last backup, so:

          - it does not consume more space

          - it is very fast

           

           

          • Will it be OK to take backup of the files in the running FileMaker Server 12 external storage folder every night (incremental etc.). This will take less than 20 minutes every night. We will of course NOT backup the database files from the active db folders, just the External Storage files.

           

           

          Not sure what the perceived value of this backup is?  Without the FM files it would not be a restorable unit.  And restoring the FM files and the external files separately is something that would need to be thoroughly tested.

           

          If you want to go down this road, I would pause the FM files first to make sure there is no interference

          1 of 1 people found this helpful
          • 2. Re: FMS 12 External Storage, backup of large file sets
            Mike Duncan

            You might also want to review the backup options in your server's OS, which will vary depending on what it is. You won't want to have OS level backup being perfomed on your live database files, leave that up to FM Server to backup for you. Not really your question, I know, but there has been plenty written on backup scripting... Wim has written an excellent guide (just search for it).

             

            Also, it's good to keep in mind what kind of scripting you do that will be backing up other files like the backups that are made or files living in the file system you might want to move off for redundancy. If the script is performing copy/compression/other processing of files, that can eat into processor time.

             

            I know when I've done something like this, I have used a separate machine that connects to the server when needed, usually nightly, copies the backup files needed, disconnect and performs a zip on them there so the FM server is not doing this... then copy them to another location for offsite backup. This is very particular to the servers involved and takes a bit of testing to get right. Hope that gives you some ideas.

             

            Mike

            • 3. Re: FMS 12 External Storage, backup of large file sets
              Extensitech

              FMS will hardlink any remote container data file that has not changed since the last backup, so:

              - it does not consume more space

              We're having a similar issue as Carsten. We have one client with about a half a TB of files, and we'd like to use external storage with them (they're already using SuperContainer) and now another project where their attachments are roughly a TB.

               

              Are we missing a setting on our backup schedule? We have, for instance, our own internal solution, with about 650 MB of externally stored files. Each and every one of our backups also ends up with about 650 MB of externally stored files. For our internal solution (with fewer files) this isn't an issue, but if we could make our backups work the way you've described, external storage would be viable for our clients with way more files.

               

              Chris Cain

              ccain@extensitech.com

              • 4. Re: FMS 12 External Storage, backup of large file sets
                CarstenLevin

                Hi again,

                 

                After moving to FileMaker 13 we used the new servers ability to store the external files on another disk. So now we are using a muche easier approach.

                 

                We are backing up FileMaker (shedules) and then moving them to tape + offsite disks.

                 

                We are doing incremental backup of the files on the external disk to tape + offsite disks.

                 

                Thanks FileMaker 13!

                 

                We have had some issues with external storage when moving FileMaker solutions from one server to another or temporarely down to a user computer and back. We need to figure out exactly how to move files and paths around related to the FileMaker bases ..... have to dig deeper into this!

                • 5. Re: FMS 12 External Storage, backup of large file sets
                  wimdecorte

                  Extensitech wrote:

                   

                  Are we missing a setting on our backup schedule? We have, for instance, our own internal solution, with about 650 MB of externally stored files. Each and every one of our backups also ends up with about 650 MB of externally stored files.

                   

                  You're checking that in Finder/Windows Explorer, right?  Those correctly report the size of a full backup set.  That does NOT mean that each backup set consumes 650MB of actual hard disk space.  There is some low-level OS functionality at work here.  I'll see if I can find my 2012 Devcon presentation on the subject.

                   

                  But here's a very simple test.  Run a backup.  Record the free disk space on the hard disk.  Don't change anything in the files and run another backup.  The free disk space on the hard disk will NOT go down because nothing changed.  But in Finder / Windows Explorer, each backup set will be reported as 650MB.  And you can copy each backup set without having to worry about the set being incomplete.

                  What has happened in the background is the "hard linking"

                  • 6. Re: FMS 12 External Storage, backup of large file sets
                    wimdecorte

                    Download the PDF referenced in this blog post.

                     

                    I included a lot of screenshots that describe how those hard links do not consume physical hard disk space but are still reported to the OS as complete backup sets.

                    http://www.soliantconsulting.com/blog/2012/09/everything-you-ever-wanted-know-about-filemaker-server-backups

                     

                    Let me know if it is not clear and I can do a follow-up to tackle specific questions

                    • 7. Re: FMS 12 External Storage, backup of large file sets
                      MacEasy

                      Hi Carsten,

                       

                      Here's what I do with large solutions.

                      I use dropbox for offsite storage because it recognizes symbolic links just like filemaker server does.

                      So it in essence backs up the incremental new documents you create.

                       

                      If you don't understand symbolic linking here's my attempt at a simple explaination.  Symbolic links allow the OS to store the same document in more than one folder without storing separate copies of the file(s).   This is why filemaker says to duplicate your backup, because your Monday backup has most of the same documents as your Sunday backup and if you modify your monday backup many of the file(s) are the same ones at your Sunday backup the same file just shows in two places.

                       

                      Now I don't know if you use Mac or Windows Server but on mac server I use a program called symbolic linker available here.

                      http://www.macupdate.com/app/mac/10433/symboliclinker

                       

                      Step 1 - Create a backup folder specifically for you offsite backup

                      Step 2 - Create a backup routine that runs at a non-user hour (1am for example or maybe 6pm if your office closes at 5pm).

                      Step 3 - Symbolically link your backup folder in step 1 to a directory in your dropbox folder.

                       

                      That's it.

                       

                      Now here's what happens on the backend.

                      Let's take a simple example

                       

                      Database.fmp12

                      RC_Data_FMS

                       

                      Now let's say you have 100,000 documents stored externally in your database and today you added 50 new documents.

                      Now let's say your database is 100mb.

                      Then all 100mb gets moved offsite since you created 50 new records.

                      But your documents folder has 1TB, but the documents are each stored externally.

                      Now the first 100,000 are already offsite and because it's symbolically linked only the 50 new documents need to get transferred.  Ain't that cool!!

                       

                      Hope this helps.

                       

                      A couple more comments. 

                      1) Throttle your dropbox connection so it doesn't bleed your sytem resources

                      2) By setting up the backup in the off hours this stops dropbox from running during the daytime hours assuming it finishes from the previous night.

                      3) Setup your first offsite backup to start Friday night, assuming you don't work weekends.  The very first backup will require it to move 1 TB offsite, that's going to take awhile.  So give it all weekend.

                      • 8. Re: FMS 12 External Storage, backup of large file sets
                        CarstenLevin

                        Hi Peter,

                         

                        Thanks for your suggestion ... we do not have any problems with symbolic links etc. etc. But we would probably never use Dropbox for anything serious.

                        And our external stored files are at the moment around 6-7 TB and rising.

                         

                        FileMaker 13 and its ablity to have the external storage at another disk than the FileMaker files solved all our problems.

                         

                        Best regards


                        Carsten

                        • 9. Re: FMS 12 External Storage, backup of large file sets
                          wimdecorte

                          Peter Beehler wrote:

                           

                          If you don't understand symbolic linking here's my attempt at a simple explaination.  Symbolic links allow the OS to store the same document in more than one folder without storing separate copies of the file(s).   This is why filemaker says to duplicate your backup, because your Monday backup has most of the same documents as your Sunday backup and if you modify your monday backup many of the file(s) are the same ones at your Sunday backup the same file just shows in two places.

                           

                          Let's not muddy the waters too much, there are a few differences between sym links and hard links:

                          http://en.wikipedia.org/wiki/Symbolic_link

                           

                          The FMS backups do not use sym links, but hard links.  For the most part the effect is the same, but the differences can be crucial depending on your expecations.

                          • 10. Re: FMS 12 External Storage, backup of large file sets
                            jormond

                            And one strong word of caution, if Dropbox ( or Google Drive, etc, etc... ) is looking at the file and preparing ot sync, the files may be hosed if you copy them off or move them or try to open them.

                             

                            This happens especially with the actual FileMaker backups and files. If you move a backup copy out of the sync program folder before it's done syncing it, the file will likely not work, convert, or recover. It has happened to several of my files, and I have long since move away from using Dropbox ( or Google Drive ) as a reliable source of immediate backups for backups. I do use if for archive stuff that I won't need to restore from. But not my recent backups that I may need to restore from in an emergency. If the hard drive or server crashes, the files are often useless.

                            • 11. Re: FMS 12 External Storage, backup of large file sets
                              Extensitech

                              Wim, great test. Thanks for the information. Our only documented experience up to now involved Windows reporting that the drive had several TB more data on it than it's actual capacity. In that instance, our client had to get some pretty intensive work done to keep the OS from freaking out about available space. We've been skeptical since but this demonstration lets move ahead with confidence.

                               

                              Thanks again,

                               

                              Extensitech

                              • 12. Re: FMS 12 External Storage, backup of large file sets
                                MacEasy

                                You're right on symbolic vs. hardlinks,  sorry wasn't trying to muddy the waters too much, but good you pointed it out for others.

                                • 13. Re: FMS 12 External Storage, backup of large file sets
                                  grzesgbc

                                  Hi Carsten,

                                   

                                  So you are using an additional database folder to store the fmp12 files. All externally stored files are moved to an container data folder (on a different drive). While setting up the additional folders you chose not to backup the external files in container data folder. This way you can schedule a FileMaker Server backup thas includes only database fmp12 files.

                                   

                                  My question is about the incremental backup of externally stored files. Is it done by FileMaker Server (progressive backup) or by a third party tool? I was not able to find any information on the additional container data folder. Is it also forbidden to touch the files stored externally in a container data folder, just like it is not allowed to do anything with live fmp12 database files in the default folder?

                                   

                                  Grzegorz

                                  • 14. Re: FMS 12 External Storage, backup of large file sets
                                    jormond

                                    Carsten is referring to the feature of FM called "External Storage". It relates to container fields and how/where FileMaker stores them. With FM 13, you store them not only outside of the database ( but inside the same folder structure in FM Server ), you can also store them on another disk.

                                     

                                    Not fmp12 files and databases, files stored in container fields.

                                    1 2 Previous Next