10 Replies Latest reply on Aug 29, 2014 1:35 PM by Mike_Mitchell

    Compress a copy of a file that uses secure storage?

    laura@solutionmakers.com

      Hello:

       

      I've been trying to find a way to save a compressed copy of a file from our server that has secure storage in use. I've tried stopping it, downloading it using the FMadmin tool, opening, saving a compressed copy and then reloading through the console - all of the image links are now broken.

       

      I then tried removing the file, opening it in the removed folder, saving a compressed copy and the reloading it. All images are broken. I then tried a opening a backup and reloading through console- same thing.

       

      How can I compress a copy and save the securely stored container objects?

       

      And is there any way to recover the images once the links are broken? I know the images are still on the server.

       

      Thanks in advance!

      Laura Bowyer

        • 1. Re: Compress a copy of a file that uses secure storage?
          Mike_Mitchell

          Laura -

           

          I believe you'll need to convert the database to internal storage, download it, then compress it, then upload it, then convert back to external storage. I've not found another method for doing this that doesn't break the links.

           

          Perhaps someone who's done this a lot (Wim DeCorte or Stephen Huston) can provide a better answer. (I hope so!)  

           

          Mike

          • 2. Re: Compress a copy of a file that uses secure storage?
            taylorsharpe

            When you upload the file, is the Remote Container file (RC_Data_FMS) on the same folder level as the file you are uploading?  Within that folder has to be a folder of the images with matching folder name to the FileMaker database name. 

             

            Speaking of which, why do you need to compress you file?  Also, note that it only compresses the FileMaker file and NOT any of the remote container field data.  I recommend you only compress the file if there are issues with the database like maybe corruption issues.  And if you are concerned about file size, it is most likely that the size of your remote container files are much larger than youf database.  So management of remote container files is more important than shrinking your database a little bit.

             

            I will say that if you used the admin console to download and upload, FileMaker should be taking care of the Remote Container location in the FileMaker server live database folder.  Most of the time people have problems when they try to manually move things in and out of the live FileMaker database folder, which is a bad practice.  So you are following the recommended practice for downloading and uploading. 

             

            Another possibility may be that when you saved the compressed version, you changed the file name, which means it is not matching up with the folder name in the Remote Container folder.  The File name and the Remote Container subfolder file name must match exactly.  That is the only other thing I could think of. 

            • 3. Re: Compress a copy of a file that uses secure storage?
              BobGossom

              Laura,

               

              Mike is correct (as usual). The links to the external storage items are maintained internally by FMP and not part of the data layer. It is known that saving a clone and importing into the clone breaks the links. Thanks to you, we now know that saving a compressed copy also breaks them. If you aren't using secure storage, and you know the file path to each of the external items, I do know of a developer that came up with a modestly complex way to loop through and relink the images. If you are interested, post back and I'll put you in touch with her.

               

              This is an issue that isn't likely to be changed soon, so the best bet now is to learn to live with it. I think the best practice for external storage is to move all externally stored items to a separate very flat file and reference them through a one-to-one link. This allows you to update the rest of the solution, but leave this one file alone, or make any needed changes to it directly.

               

              While it's a pain to make a change like this, I'd seriously consder biting the bullet and doing it. Any file using external storage has those external items permanently and inextricably linked to that specific file. You'll be hamstrung on updating the solution until you do it. 

               

              Hope this helps.

              Bob Gossom

              • 4. Re: Compress a copy of a file that uses secure storage?
                taylorsharpe

                I had previously done like Bob had suggested storing files with external references.  This is similar to what was done with like Super Container.  When FM first made remote container storage files available, I decided to manage my own files in the RC folders.  What I found out is that FileMaker had done a lot more work than I had and optimized many things including making sure not too many files are in a given folder since this will slow down OS's, particularly the Mac OS.  Also, FM will handle encrypting files and make my security stronger. 

                 

                While I first ventured away from the FM way of managing container files, I've come full circle and now highly recommend you let FileMaker manage your remote container files.  It will be more secure and FM will do things to optimize performance for you. 

                • 5. Re: Compress a copy of a file that uses secure storage?
                  BobGossom

                  Taylor,

                   

                  My reply was about using external storage, but making a separate file exculsively dedicated to this task. Other files in the solution that need the images would get them through a relationship. This lets you clone/update/replace the rest of the solution without having to move all of your external data as Mike detailed everytime you want to update the solution. You would leave this simple flat file on the server and never replace it.

                   

                  Bob G.

                  • 6. Re: Compress a copy of a file that uses secure storage?
                    taylorsharpe

                    Ahhh... yes... I now understand, Bob.  I have done that too, particularly before remote container storage was available and it reduced corruption issues I had with a database accessed by a lot of uses.  I think it is less of an issue now that we have remote container storage, but separating things in separate files can minimize binary storage files causing problems with the rest of a database records/fields.

                    • 7. Re: Compress a copy of a file that uses secure storage?
                      laura@solutionmakers.com

                      Mike, Bob, Taylor – thank you all for your responses on this!

                       

                       

                       

                      An excellent idea Bob on using a separate file solely for managing externally stored containers. I may move in that direction. Since this is a very large file used for multitudes of users and doing some very heavy lifting for media management and revision tracking, we simply cannot afford to lose one image. I would love to hear how the developer came up with a way to locate the broken links to the images!

                       

                       

                       

                      Changing the containers to internal and the compressing the file just isn’t in the cards. There is almost 2 TB worth of images to manage. Since we are importing data from a SQL source as well as using local FM tables and deleting a lot of previous records, the file is beginning to show some inconsistencies and I felt that a compressed copy might clear up some cobwebs. The more I think of it, the more I love the idea of the separate file.

                       

                       

                       

                      Thanks again folks!

                       

                       

                       

                      Best regards,

                       

                      Laura Bowyer

                      • 8. Re: Compress a copy of a file that uses secure storage?
                        taylorsharpe

                        I understand wanting to clear out the cob webs.  The compressed versions basically going through to clean out wasted space in things like indexing (e.g., when you delete a record, I think the index space still remains or something like that).  That may clean up things like indexing, but if you want to clean out cob webs, I personally would do a recover instead.  Actually, what I did on the database I cleaned up last week was save a clone, run recover on the clone, and then reimport all of the data table by table. 

                         

                        There are a few others here that really are experts on recovery.  Maybe one of them will chime in. 

                         

                        I mostly make sure to have lots of backups!

                        • 9. Re: Compress a copy of a file that uses secure storage?
                          laura@solutionmakers.com

                          Hello again - revisiting this idea.

                           

                          If I am to break off the images into their own file, how do I move the images that are stored securely/externally without breaking the links? If I have  a new name for the image file, won't that break my links? Or do I keep the same file name for the images and move all of the other tables to another file?

                           

                          Thanks! This is going to take days to do with all of the images we have linked and I certainly do not want to break the links!!

                           

                          Laura

                          • 10. Re: Compress a copy of a file that uses secure storage?
                            Mike_Mitchell

                            One method is to create the new, separate file, then just import the images from the old file to the new one. This allows FileMaker to build new links as it goes. When you’re done, just delete the image table(s) from the original file. Time-consuming, but it works.

                             

                            Another method is to save the file as a self-contained copy (converting the storage to internal). You can then duplicate that file, delete all the non-container stuff from one copy, all the container stuff from the other, repoint the table occurrences, and then convert back to external storage.