6 Replies Latest reply on Apr 28, 2014 10:16 AM by Michael Frankel

    FMS 12 Container Storage and Space Management

    Michael Frankel

      Hi Everyone -

       

      I'm going to say up front that this may get long because I really want to understand what's going on.

       

      I have a client with FMS 12 (with plans to upgrade to FMS 13 in the next 3-6 months). They have ALOT of remote containers, probably in the neighborhood of 30,000 - 50,000 images & documents of various sorts. Their backups are done every day (kept for 7 days) and every week (kept for 4 weeks). When I look at their complete database (Database plus remote containers) it comes out to about 81gb of data.

       

      For reference, this server is running on a Mac, Mac OS 10.7.5 and FMS 12.0v5.

       

      I recently made some changes to how the containers were stored which, as you might imagine, took a long time since they have so many images. The reason for doing this was that I was noticing that old backups were not getting deleted and the Server was throwing a 685 error. My research into this error provided me no useful information except a vague reference to a limit of 100 items per folder(?). I don't know if this is accurate. My hope was that doing this would, upon the next backup, cause the server to clean-up the old backup folders. This did not work as expected.

       

      I also noticed that a stand-alone backup, which was done via the Command Line (and stored in an Automator script), was more than double the size of the actual files.

       

      I also noticed that the folders that I expected to be removed after re-working the remote containers were NOT removed. This is the what I don't understand - why were those folders NOT cleaned-up in the backup process?

       

      Next, when I looked at the normal scheduled backups, I noticed that old backups are NOT being deleted there either. Perhaps I missed something and this is a known bug in FMS 12, however I need to know how to clean it up so that the disk space on the drive where the containers are stored. If this 100 item limit is for real, then that will be a problem because my customer is storing multiple images on close-to 10,000 items with more being added every day. And I come back to the same question - why were those folders NOT cleaned-up in the backup process?

       

      So, I'm looking for some input and hopefully some answers on what's going on and how to resolve it. I've already been successful at reducing the used space on the drive by changing the backup scripts and manually moving the backups to an external drive. However I need to understand what's going on and how to clean-up those backups.

       

      I understand that the Backup process in FMS 12 uses hard-links to eliminate redundant data. However, what am I trading for that space reduction feature? does it really work in a high-volume situation like this one?

       

      Thank you in advance for any input you may have on this.

      --

      Michael Frankel

      Wizard Consulting Group, Inc.

      Certified FileMaker Developer

      Office: 818-706-8877

      Office: 818-483-4686

      Skype: michaelfrankel

      Mobile: 310-291-3419

      Website: www.wizardconsultinggroup.com

        • 1. Re: FMS 12 Container Storage and Space Management
          taylorsharpe

          This is just a guess, but something to look at.  Are you using Remote Container Storage?  Are you trying to manage the folder locations yourself?  If so, you have to also manage how many files are in a folder and that is a real pain.  If this is the case, do the easy thing and follow the standard FileMaker Policy of letting FileMaker manage the folders and encrypt the remote container storage.  That will solve your problem because FileMaker will create subfolders on its own as needed and you don't have to manage this folder issue.  And while it may cause backup problems like you are seeing, it more importantly will cause performance problems, particulary if your server is a Mac OS X.  File Management on a Mac does not do well when there are hundreds or thousands of files in a folder and accessing a file in a large folder is much slower than accessing a sub folder with the same file. 

           

          Let us know how you are managing remote container storage and this might be an answer if you are managing the file storage yourself.  If not, it might be another problem.  But this is something to look at. 

          1 of 1 people found this helpful
          • 2. Re: FMS 12 Container Storage and Space Management
            Michael Frankel

            Hi Taylor -

             

            Thanks for your reply.

             

            Yes, I am using Remote Container Storage and yes, I am managing the folder locations myself. My concern is, considering the amount of data and the client's requirements, that losing access to the data would be catastrophic for the client. In the current set-up, I still have a reasonable chance to preserve or find a attached file or document if something goes wrong. If I'm depending on FileMaker Server to manage it, finding a file if needed would be virtually impossible since the client does NOT use unique file names in most cases.

             

            I do agree that file management on a Mac can be slow at times, however I have seen the same issues under current versions of Windows, so I don't know if the platform makes that much of a difference with this number of files.

             

            I am most concerned about the Server software not doing what is advertised, that is removing old backup folders when new ones are created.

            • 3. Re: FMS 12 Container Storage and Space Management
              taylorsharpe

              I have to admit, I thought like you did and tried to manage the files myself and unecrypted for the same reasons you suggest.  It resulted in a lot of problems.  I have had to recover several clients who did this too.  After having this issue several times, I finally figured out that you have to be really on top of things if you don't want to follow the default setting by FileMaker.  I think you are in the process of learning the lesson I learned a while back in 12. 

               

              You say you are worried about the server not doing what it is advertised to do.  It is doing what you have told it and within the limitation of the OS too.  Again, I reiterate, following the default FileMaker setup for the server and remote container storage will result in more stable and less problematic solution. 

               

              You can keep fighting the system you have implemented manually and it will involve managing folders, subfolders, and file counts per folder.  It can be done, but it is not for the faint of heart.  Or you can do the easy way that FileMaker recommends.  I had to learn my lesson and now do it the FileMaker way after beating my head on the wall trying to manually do it. 

              • 4. Re: FMS 12 Container Storage and Space Management
                Michael Frankel

                I'm pretty good at following directions and using the recommended settings. I just don't ever recall reading anything in the documentation that said it would only manage backups properly if you let FileMaker manage container storage using secure storage.

                 

                If that is the case, I would like someone from FileMaker, Inc. to tell me so.

                 

                I have no problem switching over and having FileMaker manage the containers, even if it will take multiple hours to migrate all of the images & documents. However, if it does not eliminate the problem with my backups, and if I can't get someone who has been in the same situation to tell me it did for them, I can't justify taking the time to do it, at least not until I switch over to FMS 13.

                 

                If you've been in this exact same situation and it worked for you and cleaned-up your backups, please let me know.

                • 5. Re: FMS 12 Container Storage and Space Management
                  wimdecorte

                  wizardconsultinggroup wrote:

                   

                  I'm pretty good at following directions and using the recommended settings. I just don't ever recall reading anything in the documentation that said it would only manage backups properly if you let FileMaker manage container storage using secure storage.

                   

                  You should not read that into Taylor's statement.  Using secure storage in your scenario MAY be more robust because it reduces the likelilhood of storing hundreds or thousands of items in one folder.

                   

                  A couple of things I would suggest:

                  - set up a test bed with FMS13 and try the backups, to see if anything was fixed in 13 that covers your scenario

                  - open a support ticket with FMI to get to the bottom of why the error 685 happens and why backups do not clean up after themselves.

                  - set up your RC folder structure so that the chances of many files in one folder does not happen

                  1 of 1 people found this helpful
                  • 6. Re: FMS 12 Container Storage and Space Management
                    Michael Frankel

                    Hi Wim -

                     

                    Thanks for jumping in on this one...

                     

                    The changes I made with the remote container storage locations would have reduced the number of files per folder to less than 20.

                     

                    Do you believe that the issue is with the number of folders inside a folder or the number of files inside a folder?

                     

                    Please advise.