1 Reply Latest reply on Jun 15, 2016 2:07 PM by amallo

    A problem with Secure Storage

    amallo

      My client has 200GB of documents linked to tasks in FM via external secure storage.

      We moved from SuperContainer to this option and it served us well.

      We turned on secure because we didn’t want others to look at the files.

      Then we decided for WebDirect speed we needed to move it to a 2 server deployment in a data centre.

      We wanted to archive most of the data off, to save disk space, and only take the last 12 months data to the data centre.

      Here is the problem. You can’t.

      You have to convert it to open folders based on folders that relate to year or month containers etc. eg YYYYMM

      So we started down this path.

      It started ok until we kept running out of disk space. We couldn’t work out why.

      What seems to happen is FM takes the secure files and writes them into the fmp12 file and then writes them out. Nothing is deleted in this process until the new 200GB open folder data is confirmed to be complete.

      Meanwhile the FM file keeps growing in size. It looks like it is wanting to grow to 200GB.

      I had the server crash 4 times. After each time we expanded the disk. Looking back on the files I can see the FM file went from 5mb to 20GB then 80GB and then 125GB.

      (Note: make sure if you do this process that you turn off any server backups on this file.)

      The process never finished. I think the client now has access to about 50% of their files. This is a real problem.

      When it crashes the file never gets the required command to flush the temp files inside the fmp12 file.

      This is now a big issue for future backups.

       

      A file recover on the 125GB file crashed the machine again.

       

      I’m going to restore from a pre-bloated file and try again with a TB drive to see how it goes ok without crashing. Then I'll clone the current bloated file and reimport the data and move the hopefully open storage created files to the appropriate places.

      Instead of cloning and importing, I have asked FMI if there is a temporary table or field that is holding this bloated data and can it be deleted with the help of an SQL plugin.

       

      So the take home is, if you ever need to archive off older secure data then it can’t be done unless you have the space and time to convert to a date based Open Storage. Often the reason to archive is lack of disk space which could lead to the above outcome. So be careful turning it on.

      A date (YYYY or YYYYMM) or calculation folder naming option above the secure folder option looks like it is needed.

      I will likely be only using Open Storage in the future.

        • 1. Re: A problem with Secure Storage
          amallo

          Just a followup.

          In one of my attempts I got a warning that said I needed more then twice the amount of space then what was stored securely.

          Screen Shot 2016-06-11 at 6.24.24 PM.png

           

          I transferred a backup of the whole solution to a 2TB USB drive. I ran the change to open storage. It took 22 hours. It basically worked and the FileMaker file ended up the same size. But by this stage most of the recovered secure files don't link with the current FileMaker file.

           

          FM support said the file bloating talked about above was probably from a bug or file corruption. They said they didn't expect this behaviour.