8 Replies Latest reply on Jul 13, 2015 2:28 PM by BarbaraCooney

    Best practice for version updates and external data storage

    psijmons

      The option to store container data externally from FMP has now been around for a couple of years.

       

      Especially with secure storage, one has to be careful when moving files around from production server to developer machines and vv. as some of us may have found out the hard way.

      In our solution with data separation, when occasionally we need to update the data file, we download the files from FMS to our servers, run import scripts to migrate all data to the new version, then upload back to production servers.

      As we are using SuperContainer, the data files remain relatively light. However, for a new solution we contemplate external data storage in the field option for container fields and would like to run all import scripts to fresh clones directly on the production server.

       

      I am curious to see best practices for installing updates, migrating all table data from old version to new version and keeping all links to external data storage intact.

       

      Peter

        • 1. Re: Best practice for version updates and external data storage
          Stephen Huston

          As long as you use the Server Admin tool to do your file uploads, the external storage should remain intact. However, If you try to move the files to the server manually, expect the external container data to fail.

           

          Another solution discussed on a parallel thread today is to place container data internally, but in a separate FMP file with a 1-to-1 relationship to the main table/file. This way you have the benefit of offloading the data ( especially if containers change infrequently) and being able to manage everything as FMP files both during updates and for server uploads.

           

          That's much simpler adn faster than the alternate option of switching container data to internal  before moving it and switching it back to external after moving it.

          • 2. Re: Best practice for version updates and external data storage
            psijmons

            thanks Stephen,

             

            Unfortunately, in FMS13 you can no longer upload using FMSAT but you need to use FMPA, but that is just a minor inconvenience that I will get used to. Anyway, I tested this scenario and it does break the links (at least in my tests).

            This was the tested procedure, using a test table with a few records and different storage options

            • create clone of the latest version files
            • download data file
            • migrate all data to latest version on locally opened files
            • remove old version from server
            • upload new data and UI files to server

             

            I contemplated a dedicated file with 1 table for container data some years ago but since the uploaded files can be really big (video, large DNA sequence files etc) and this trend is getting worse, I then decided to go for SuperContainer. In older FMP versions (<8)  I had bad experiences with container data that would occasionally corrupt files. I am not sure if this can still happen but I am not willing to test this.

            So I am really very much in favour of external data storage.

            • 3. Re: Best practice for version updates and external data storage
              Stephen Huston

              Let me see if I've got this correct about your sequence:

              1. create clone (from what copy of the file?)
              2. download data
              3. migrate data to loca file (the clone?)
              4. remove old file from server
              5. upload new file (which began as a clone)

              If that's correct, I would expect problems between the new file and the external storage on the server for a couple of  reasons:

              1. a recently cloned file will have different internal record file IDs (used to match data pieces on a disk to a specific record), so they are not likely to match the links to the external container data, and
              2. the new clone was created without links to the external data because it was not downloaded as part of the data used to populate the clone with records.

              In other words, using a clone and populating it with new internal data is not going to populate it with the necessary hardlinks to the external container data still on the server.

               

              I think you will need to change the storage to internal while the file is on the server, then move the file, import into a clone, then upload to the server, and if you want to, then reset it to external storage. (That would be more reliable than changing it to external storage locally and then uploading everything via the application.)

              • 4. Re: Best practice for version updates and external data storage
                psijmons

                :

                1. create clone (from what copy of the file?)  from a more recent version then the one on the server
                2. download data
                3. migrate data to loca file (the clone?)
                4. remove old file from server
                5. upload new file (which began as a clone)  yes

                 

                Yes, I agree with your conclusion as to why this would break the link, but I was hoping if there was a workaorund which would not involve migration of all external data to internal prior to the update.

                This can be scripted using containers with external and internal storage options but it would blow up that file size considerably.

                 

                I will reconsider the dedicated file option you suggested. As I am using data separation, just one extra datafile with one table dedicated to (external) file storage might be a very stable situation as that file would very rarely require a change in ERD or logic. So when doing update cycles this one file would just stay in place. Probably a separate folder even on the server to prevent accidental removal.

                The file could be prepared with duplicate containers to allow switching between internal and extrnal files.

                 

                You've helped me think this through, thanks

                • 5. Re: Best practice for version updates and external data storage
                  Stephen Huston

                  Sometimes it's the rethinking the problem that gives the new solution.

                  When I first hit on the separate file for offloading high-volume date which would be 1-to-1 with the main table, it was an AHA moment of sorts. No changing structure in that file, no need for it to have data refs to other files, scripts, etc. Kind of the ultimate extra bit for a Separation Model.

                  • 7. Re: Best practice for version updates and external data storage
                    psijmons

                    The latest release FMP13v2 may have solved this issue, copied from the release notes:

                     

                    • Added a “Preserve external container storage” option to skip re-importing external container storage and, instead, reuse existing external contents during records import between FileMaker databases.

                     

                    I will check this as soon as I have the time but if I read this correctly it makes updatig a solution easier.

                    • 8. Re: Best practice for version updates and external data storage
                      BarbaraCooney

                      "Another solution discussed on a parallel thread today is to place container data internally, but in a separate FMP file with a 1-to-1 relationship to the main table/file."

                       

                      Can you link to that discussion? I cannot find it.