1 2 3 Previous Next 31 Replies Latest reply on Jan 9, 2017 10:42 AM by dale_allyn

    Storing Container Content Externally: Best Practice?

    dale_allyn

      Description: I’m developing a decent-sized solution (single file), currently with 30+ tables, 550+ fields, 150+ scripts, etc. and growing. Development will continue with both UI and Schema, as well as functionality, especially for the first year of service. It will be served to 5-10 users to start, later growing to 25 users and more, via FMS 15. At a later time, clients (customers of the business) will have access to some of their account info and history, some documents (as PDF) as well as current job status, etc. via the Web. I believe I can limit customer access to container data to only PDFs of their reports.

       

      There are two main “areas”: Customer Service side, and Laboratory Analysis and Reporting side. The two areas are strictly partitioned for two-way anonymity, but the data ultimately crosses over.

       

      The customer service side has few containers: An account ID barcode, a submission barcode (for each item submitted, could be hundreds or thousands over time), and photos of contacts associated with each account. Later, billing will have invoice barcodes in containers.

       

      However, the lab side has many containers, and in some cases, each submitted specimen will have numerous photos of the specimen, photomicrographs, advanced analytical instrument output, etc. These images are associated with different areas of description/reporting or scientific tests for analysis. Also, there will be QR codes generated and stored for completed reports.

       

      Concerns: “Proper” storage method for container data which allows decent performance and reliability while also allowing access to some content by outside (of FM) services.

       

      Plus, relatively painless uploads of new solution versions which will be frequent for several months.

       

      Question: What are the “best practice” storage choices for container data to allow for easy backups, accessibility by FMP and FM Go, as well as some web services, etc? (Not asking about backup strategy; there will be mirrors and frequent incremental backups, redundant local and off-site, etc.)

       

      For demo purposes and portability in the earliest stage of development, my test data was held locally in containers. This was to allow passing the solution from my workstation to my laptop easily, as well as re-using the solution on another project.

       

      For production the container data will be stored externally. I have read that choosing encrypted storage has a performance advantage in FM, but I’m concerned about future-proofing the data, as well as the possibility of accessing it via other, non-FM services. Therefore, I’m planning to use the open storage option, at least for some container fields. Am I overlooking pitfalls with this decision? Encryption is of value, but with the data being “mission critical” I’m concerned about the propriety nature of the FM default “Secure Storage” option.

       

      How would you suggest I store such container data externally? Open Storage and encrypt via FMS’s EAR option? Accept the FM default Secure Storage option, and figure out a way to save originals elsewhere??

       

      Sorry for the verbosity – hoping to provide enough info. Thanks in advance for sharing any thoughts and insight.

        • 1. Re: Storing Container Content Externally: Best Practice?
          Malcolm

          Question: What are the “best practice” storage choices for container data to allow for easy backups, accessibility by FMP and FM Go, as well as some web services, etc?

          The answer is simple. Use external containers.

           

          The decision to use encrypted or unencrypted storage for the container data is not an engineering decision. Choosing to use external storage means that the data is accessible from the file system, where it is not protected by the user accounts security provided by FileMaker. Is it acceptable for the container data to be accessible in that way? If not, it must be encrypted. This ensures that only a user who has logged into your FMP database and has the proper credentials can view the data.

          • 2. Re: Storing Container Content Externally: Best Practice?
            dale_allyn

            Thank you for your reply, Malcolm.

             

            I'm aware of the decision process for choosing to encrypt data or not. Most of the content will be encrypted. My question was specifically concerning options other than FileMakers Encrypted Storage option compared with a solution using "Open storage" externally and encrypting as needed. The point being that FM's Encrypted Storage option uses a propriety algorithm accessible only via FileMaker. I'm wondering if developers are widely accepting this option, or choosing to store external container data in a form that can be retrieved (and decrypted) independently from FM.

            • 3. Re: Storing Container Content Externally: Best Practice?
              Malcolm

              Are you imagining a situation in which the stored data has to be accessible to other applications at the same time as it is stored in FMP? Or are you imagining a situation in which all the data has been stored in FMP but now, for some reason, FMP is not available to decode the data?

               

              Malcolm

              • 4. Re: Storing Container Content Externally: Best Practice?
                dale_allyn

                Or are you imagining a situation in which all the data has been stored in FMP but now, for some reason, FMP is not available to decode the data?

                ^ This was a first concern. Wanting to future-proof the data if the solution migrates to a non-FMP environment.

                 

                Are you imagining a situation in which the stored data has to be accessible to other applications at the same time as it is stored in FMP?

                ^ This is secondary, as there are ways to interact with FMP, so I was looking for input/ideas of others (like you) to find out if there are favored approaches besides the sort of "default".

                 

                Thanks again.

                • 5. Re: Storing Container Content Externally: Best Practice?
                  Malcolm

                  Or are you imagining a situation in which all the data has been stored in FMP but now, for some reason, FMP is not available to decode the data?

                  ^ This was a first concern. Wanting to future-proof the data if the solution migrates to a non-FMP environment.

                   

                  At the time of migration, change the container field storage to internal.

                   

                  Are you imagining a situation in which the stored data has to be accessible to other applications at the same time as it is stored in FMP?

                  ^ This is secondary, as there are ways to interact with FMP, so I was looking for input/ideas of others (like you) to find out if there are favored approaches besides the sort of "default".

                   

                  The user authenticates via FMP. Having authenticated, FMP will deliver any/all data that they are authorised to access.

                   

                  Malcolm

                  1 of 1 people found this helpful
                  • 6. Re: Storing Container Content Externally: Best Practice?
                    dale_allyn

                    Malcolm:

                     

                    Thanks for the tip/reminder on changing storage to internal at time of migration. That escaped me.

                     

                    Sure, the user authenticates via FMP to gain controlled access.

                     

                    => So it sounds like you're in favor of using only FM's Encrypted Storage option (assuming encryption is needed), and relying on then changing to internal storage at time of (possible) migration, even if there may be many hundreds of thousands of data files (or more) to migrate and dozens of fields affected. (?) That was part of my consideration which I hadn't worded well in my post. In the "SQL environment" this is all generic and I'm just wanting to prepare as much as possible to avoid potential traps in the event that FMP is no longer available or suitable.

                    • 7. Re: Storing Container Content Externally: Best Practice?
                      taylorsharpe

                      One big plus of having FileMaker handle remote container storage and encryption for you instead of just using open storage is that FileMaker will manage all of the subfolders.  That doesn't sound like a big deal until you realize that there are significant performance issues at the OS level when you put thousands of files in a single folder.  You will see when FileMaker manages it, it only keeps one file per folder.  I only tested this on a Mac, but once I went from remote open storage all in one folder to FileMaker managing encrypted remote container storage, the performance of accessing those container documents went a lot faster. 

                      1 of 1 people found this helpful
                      • 8. Re: Storing Container Content Externally: Best Practice?
                        Malcolm

                        So it sounds like you're in favor of using only FM's Encrypted Storage option (assuming encryption is needed), and relying on then changing to internal storage at time of (possible) migration, even if there may be many hundreds of thousands of data files (or more) to migrate and dozens of fields affected.

                        If your FMP solution is going be superseded by another system, you'll have more than enough notice.  Planning will begin months in advance. No one is going to say "we're moving the entire system tomorrow and we want all your data now"

                        • 9. Re: Storing Container Content Externally: Best Practice?
                          dale_allyn

                          Thank you, Taylor. I had seen that letting FM handle the encryption had performance advantages (in the docs). I'm glad to "hear" that you saw real-world evidence of this. I had set the containers in this database to store data externally and encrypted, but grew uneasy as I considered a possible exit or recovery strategy.

                          • 10. Re: Storing Container Content Externally: Best Practice?
                            dale_allyn

                            Malcolm, no of course not. You're correct that any change would occur with reasonable lead time. My reason for posting the questing was to confirm that what I have currently set up is considered "best practice" and to be sure that I wasn't neglecting to provide a fairly safe path to transition if needed, especially if someone else was faced with that task.

                             

                            Thanks again for your input.

                            • 11. Re: Storing Container Content Externally: Best Practice?
                              taylorsharpe

                              Dale, to be honest with you, I didn't trust FM and thought I knew better with the open storage, etc.  I guess that is my stubborn side of things... but the University of Hard Knocks teaches some good lessons too!

                              • 12. Re: Storing Container Content Externally: Best Practice?
                                dale_allyn

                                Taylor, that's why I'm here... attending class.

                                 

                                I read as much as I can (here and elsewhere), explore and experiment, but I really value the combined experience of the members of this community.

                                 

                                Thank you for sharing your experience!

                                • 13. Re: Storing Container Content Externally: Best Practice?
                                  Malcolm

                                  The main consideration for the change will be to perform the job on a fast drive with sufficient disk space. I don't know exactly how FMP manages this task but it will involve a lot of reading/writing and will require (a) existing space in use (b) space for decryption (c) space for the decrypted copy in internal storage. That probably means space required for the job is double (a) plus overhead.

                                  • 14. Re: Storing Container Content Externally: Best Practice?
                                    dale_allyn

                                    Malcolm, yes, that makes perfect sense, and I never do critical work on crowded volumes. Regardless of using FMP or a traditional DB stack, such operations should be done on volumes with LOTS of space. A good reminder, certainly.

                                    1 2 3 Previous Next