10 Replies Latest reply on Oct 1, 2012 2:02 PM by darrenburgess

    External secure storage - deleted database restore vs. asset restore

    Mike_Mitchell

      Good day. We just had a somewhat odd incident. We accidentally deleted a database from a server. We restored it, so that wasn't a problem. However, the database has some container fields that use external secured storage, and when the hosting provider restored the database, the external containers created a new directory structure, parallel to the old one. The assets are, apparently, still there, but the database is reporting them "missing" in the containers.

       

      The provider tried copying the folders over into the new directory tree, to no effect. The database still thinks the assets are missing. What can be done at this point to recover the encrypted assets?

       

      Mike

        • 1. Re: External secure storage - deleted database restore vs. asset restore
          Stephen Huston

          Hi Mike,

           

          Was the data file "restored" by copying from a backup with the hard-linked container data, or from something else?

           

          My suspicion is that with the new hard-linked backups, the full backup folder may need to be used for restoring, not a specific file. Haven't had to do this myself yet (knock on wood), but the new hard-link backup system seems to impose new requirements on restoring from a backup.

          • 2. Re: External secure storage - deleted database restore vs. asset restore
            Mike_Mitchell

            Thanks, Stephen. It appears you're right. It also appears that just deleting a database out of the directory (through the admin tool) breaks the links between the container fields and their associated files, and they don't repair themselves.

             

            So, word to the wise: Before deleting a file from a server, be careful!  

             

            Mike

            • 3. Re: External secure storage - deleted database restore vs. asset restore
              CarstenLevin

              Yes: And Secure Storage is OK, but only when/if you can explain why.

              Do not use secure storage unless it is requested by business rules or other good reasons. Having the files stored openly on the FileMaker server will also be very secure if the server is well protected, and you will not end up in a dead end by some system-administrator changing something without knowing the consequenses.

              • 4. Re: External secure storage - deleted database restore vs. asset restore
                wimdecorte

                If all possible the best practice should probably be to use the admin console's Remove feature so that you can see what else is being removed.  You can then delete from the "removed by fms" folder.

                • 5. Re: External secure storage - deleted database restore vs. asset restore
                  Mike_Mitchell

                  Thanks, Carsten and Wim.

                   

                  In this case, the customer is dealing with personal information (it's a non-profit servicing veterans, and they have a number of documents they're storing with PII-type information). So secured storage is probably best.

                   

                  And, if I had access to the Admin Console, I would definitely use the Remove feature. Unfortunately, I'm using the hosting provider's web-based tool, and honestly, I don't know how it works. According to their tech support, they're still working on rolling the version 12 external storage into their system. Hopefully, it will mimic the Remove feature soon.

                   

                  Thanks again.

                   

                  Mike

                  • 6. Re: External secure storage - deleted database restore vs. asset restore
                    wimdecorte

                    Carsten Levin wrote:

                     

                    Yes: And Secure Storage is OK, but only when/if you can explain why.

                     

                    I would disagree.  Secure store is the default for a good reason: externally stored container data should be considered just as container data that is stored inside the FM file.  The benefit of external storage is reduced backup times and reduced disk requirements, NOT the ability to see or manipulate the stored container data outside of FM.

                    • 7. Re: External secure storage - deleted database restore vs. asset restore
                      JonThatcher

                      Two questions:

                       

                      1. How did you move the restored file into place?  If you were emailed a copy of the database, could it have been opened in Pro, checked a little and closed, then later uploaded to Server?  It's possible that opening it in Pro reset the remote container base directory paths, which caused Server to not reconnect the base directory paths to the folders it already had on disk.

                       

                      2. Sorry, I have to ask: Are you certain the restored file was fairly recent?  If the file you got back didn't have all of the latest changes you had made to the container fields, especially the base directory paths, or changes from open storage to secure storage, that would explain the failure too.

                       

                      Puzzled...

                      Jon

                      • 8. Re: External secure storage - deleted database restore vs. asset restore
                        Mike_Mitchell

                        Hi, Jon.

                         

                        Unfortunately, I can't give you precise answers to your questions because the actual operation was carried out by the hosting provider's tech support. I can tell you what I did, and what they told me.

                         

                        Using the web-based tool, I accidentally deleted the file. I then downloaded what I thought was the most recent backup and uploaded it to the directory. (That may have been what killed it right there.) They do their backups every morning at 3:00 AM, so we ended up losing that day's work.

                         

                        Contacting support, I asked if anything could be done. They reported that the "deleted" file wasn't actually deleted, but moved to another directory. (This sounds remarkably like the Removed feature of Server, yes?) They then moved the "deleted" file back into place.

                         

                        At this point, I had completely forgotten about the external containers. They're used for archiving, so the customer didn't notice it either, until about 2 weeks later. Lots of the files came up "missing" at that point. Contacted the provider again.

                         

                        The tech support rep (who was very patient and persistent, BTW), said the directory structure for the external storage not been deleted; rather, a parallel structure had been created. In an attempt to restore the files, he copied the contents of the one structure into the other. This had no effect. He then copied the other way, making them mirror images of each other. That didn't work either.

                         

                        I tried turning external storage off (pulling the assets into the database) and then back on again, which didn't do anything productive (not that I really expected it to); it just reported that X records were skipped. So, we basically just went back and found the copies of the files that were stored locally and put them back on the server (I just finished doing that tonight).

                         

                        Any insights would be great. Thanks!

                         

                        Mike

                        • 9. Re: External secure storage - deleted database restore vs. asset restore
                          Stephen Huston

                          Hi wimdecorte,

                           

                          I absolutely agree with you on this: Secure storage always makes sense in a served-file situation. There is no time when any user should be sticking their fingers into the container data on the server via the OS interface.

                           

                          The FMI engineers indicated that they had not even intended to offer Open storage as an option, but decided that some users would appreciate that access for local files. Unfortunately, those file settings are  available whether or not the file is served, so many users think everything they can get at is fair game. It was not the FM engineers' original plan that users would go in and mess with the container data at all.

                          • 10. Re: External secure storage - deleted database restore vs. asset restore
                            darrenburgess

                            I did some testing with this issue.  I discovered the following (windows OS hosting FMS server):

                             

                            • A backup of a file with secure remote containers will break the containers if opened outside of server
                            • Uploading the files to the server with the admin console will break the containers
                            • Copying the files to the server data directory, then opening in Admin Console will restore the container data
                            • Downloading the file via the FM 12 Admin Console will preserve container field data

                             

                            By files, I mean the backup file of the FM database and the RC_Data_FMS directory containing encrypted container data.

                             

                            According to the FMS 12 course on www.vtc.com (Thanks Wim!), hardlinked backups must copied and not moved.  If they are copied the OS does the working of patching things together.  If moved, the OS does not do this work.

                             

                            Darren Burgess

                            www.mightydata.com

                            darren_burgess@mightydata.com