5 Replies Latest reply on Sep 3, 2016 5:31 PM by keywords

    Container field with too many files - best practicies?

    samgiertz

      We have a FM solution with invoices stored as PDF files in an container field. Luckily we are pretty good at generating invoices and we have about 10000 generted. For each PDF file a JPG and PNG file is also created by FM so we now have about 30k files. The folder is about 3.3 GB. My database file is about 20 GB.

       

      We use a backup setup including weekly backups to dropbox and this is where I run into problems.

       

      We started using the secure function but quickly realied that it creates massive amounts of subfolders. Dropbox quickly started to suffer. With about 100 weekly historical backups, my dropbox has stopped functioning. Apparently it is a well known issue with dropbox. And once you have loaded too many files and folders to your dropbox, there is no going back, you just get error messages.

       

      Anyhow I started thinking, maybe I should not store all these invoices going back five years in this way. What do you guys say, should I:

       

      1. archive some data/files at this point to another database? I am not too sure how this would work and I dont have any experiance from setting this up.

       

      2. do away with the container field and bring the PDFs into the database? My main issue is not 25GB of database but all the files...

       

      3. just move PDFs to a backup folder when they are over certain age? I am not sure how filemaker behaves if it cant find the file. I dont have an issue with digging in a separate folder if I ever need a two year old invoice. Plus the data needed to generate the PDF is in the database, I mostly store it to see what the PDF we sent out really looked like.

       

      4. Something else?

        • 1. Re: Container field with too many files - best practicies?
          coherentkris

          your solution should lock invoice records when they are "finalized" and produce a pdf on demand when someone wants to see the invoice in pdf.

          The actual official record is the data in the database that was used to create the PDF/JPG in the first place so that is what you must endeavor to protect and preserve.

          • 2. Re: Container field with too many files - best practicies?
            FabriceNordmann

            Hi,

            a good practice is to prepare your backup for a recovery scenario.

            If you once need to restore a backup, you'll have to be quick. Which means sevral things:

            - it should be accessed easily and fast, which means you should zip it (or rar, or whatever archive system) because a single compacted file is transferred more efficiently across the network.

            - it should be in a form that can be uploaded to he server as is. And that is NOT the case with external containers. (Which is a shame, I have to say). Unlike what you get from the download database menu of the admin console, backups cannot be restored as is because the container data and database files have incorrect relative paths.

            What we do is run a system script that reorganizes backup files, compacts them and move them to the recovery data center.

            1 of 1 people found this helpful
            • 3. Re: Container field with too many files - best practicies?
              keywords

              I largely concur with coherentkris's comments, but I like the idea of having a copy of important documents such as invoices outside of the database also:

              1.     If the only copy of an invoice is inside the database and you lose the database, you have no record at all.

              2.     For this reason I don't rely on the PDF being stored inside the database either; rather, I store it elsewhere on my computer system, although with some things I do both (see next point).

              3.     I store a copy in a container field if it is something I want ready access to on my desktop—e.g. details of a quote letter exactly as supplied to a client.

              4.     A pdf version is a frozen in time copy of exactly what was sent to a client, in the format in which they see it, whereas the database may contain additional admin details—underlying calculations and so forth.

              • 4. Re: Container field with too many files - best practicies?
                coherentkris

                "you lose the database, you have no record at all."

                 

                Thats where a thoughtful, robust backup strategy comes in

                • 5. Re: Container field with too many files - best practicies?
                  keywords

                  True enough, and I don't disagree at all. The chance of a total loss is pretty minimal but I still like to keep important documents generated by FM in a different format. I could have added point 5—PDF is very universal and can be viewed with many programs, whereas FM is just accessible by FM. PDF is the nearest there is to an electronic substitute for paper files, which is how I see them.