1 2 Previous Next 25 Replies Latest reply on May 13, 2015 6:22 AM by mikewson

    "Preserve external container storage" makes imports incredibly slow

    jnouwen

      Summary

      "Preserve external container storage" makes imports incredibly slow

      Product

      FileMaker Pro

      Version

      13.0.2

      Operating system version

      Mac OS X 10.9.2

      Description of the issue

      Importing records from a table with externally referenced container data is significantly slower when using the new "Preserve external container storage" import option.

      (original thread here: http://forums.filemaker.com/posts/de42b22b99)

      Steps to reproduce the problem

      Run an import from a table with externally referenced container data both with and without the "Preserve external container storage" option enabled.

      Expected result

      When running with the option enabled, the import should be significantly faster than running without since it doesn't need to copy any of the container data.

      Actual result

      Running with the option enabled is 8x slower to move a fraction of the data.

      Exact text of any error message(s) that appear

      n/a

      Configuration information

      Here's my setup for the tests I ran.

      Source

      SSD drive, empty except for...
      ... non-served FileMaker file and it's "sidecar" folder of container documents
      Total documents size: 229GB
      Number of records: 47,652

      Destination

      Separate SSD drive, empty except for...
      ... non-served copy of the Source's FileMaker file created with a Save as Clone (a new clone was created for each test run)
      A script containing an Import Records step bookended by Get( CurrentTimestamp )
      The Import Action was set to Add new records
      Auto-calcs were disabled in the import options
      "Preserve external container storage" was toggled as necessary depending on which test I was running


      Running the script with "Preserve external container storage" disabled: 1 hour 6 minutes to import 229GB of document data

      Running the script with "Preserve external container storage" enabled: 8 hours 24 minutes to import 45.1MB of record data

      Workaround

      Don't use the "Preserve external container storage" option.

        • 1. Re: "Preserve external container storage" makes imports incredibly slow
          TSGal

               Kepi:

               Thank you for your detailed post.

               I do not have a data set that large, nor do I have the number of documents to test.  I have sent your entire post to our Development and Testing departments for review.  When I receive any type of feedback, I will let you know.

               TSGal
               FileMaker, Inc.

          • 2. Re: "Preserve external container storage" makes imports incredibly slow
            TSGal

                 Kepi:

                 Initial results from Testing show they could not reproduce the issue using a 6GB file.  Tester asked:

                 1. Is the customer using open storage or secure storage?

                 2. Are the two SSD drives within one computer?

                 Once the tester has this information, he will proceed with the test and larger files.

                 TSGal
                 FileMaker, Inc.

            • 3. Re: "Preserve external container storage" makes imports incredibly slow
              jnouwen

                   1. Open storage

                   2. Yes, they're in the same machine

                   I forgot to mention in my initial report that it started out relatively quickly but got slower and slower as the import progressed, so I'm not surprised the issue didn't present with a small dataset.

              • 4. Re: "Preserve external container storage" makes imports incredibly slow
                TSGal

                     Kepi:

                     Thank you for the quick reply.  I have sent your information back to the tester.

                     TSGal
                     FileMaker, Inc.

                • 5. Re: "Preserve external container storage" makes imports incredibly slow
                  TSGal

                       Kepi:

                       Our Testing department has some mixed results.  They asked for a clone of your file so they can test this out with your file.  Please check your Inbox at the top of this page for instructions where to send the file.

                       TSGal
                       FileMaker, Inc.

                  • 6. Re: "Preserve external container storage" makes imports incredibly slow
                    TSGal

                         Kepi:

                         Just a quick confirmation that I received your file.  This has been sent to our Testing department for review.  I'll keep you posted.

                         TSGal
                         FileMaker, Inc.

                    • 7. Re: "Preserve external container storage" makes imports incredibly slow
                      TSGal

                           Kepi:

                           I apologize for the late reply.

                           Development and Testing have been unable to reproduce the issue.  Using your file, the results have been a 2.5 minute import without the option on, and a 25 second import if you turn on "Preserve external container".

                           Even with that said, the issue is still being investigated.

                           TSGal
                           FileMaker, Inc.

                      • 8. Re: "Preserve external container storage" makes imports incredibly slow
                        jnouwen

                             Hi TSGal,

                             Well, that's unfortunate. What was the test data size and record count? We're up to 355GB of container data across 56,500 records now, but I haven't re-run the test on my end in a while. Considering we have more data than before, I expect it would be worse.

                             I'm really curious as to what's going on, because I've run the import many, many times and every time it's horrendously slow when using the "Preserve external container" option (and expected data speeds when actually moving the container contents).

                             Two of my co-workers are coming to DevCon and will be bringing a copy of our solution to run it at the support labs to demonstrate the slowness. I mean, the tests here were run on an Xserve between two separate SSD drives, so it should have been fast!

                             I really think this is an issue of scale. Can you have Development and Testing load up the file with several hundred gigs of data over tens of thousands of records? A 2.5 minute import indicates to me that very little data was being used.

                        • 9. Re: "Preserve external container storage" makes imports incredibly slow
                          TSGal

                               Kepi:

                               Another tester took a 5 MB JPG file, duplicated and reduplicated until there were 16,384 total.  Creating a file from scratch and cloning it (so there were two identical files), imports were then performed.  Without the option on the first file, it took 18 minutes.  With the option on the second file, it took almost 60 minutes.  Therefore, it does appear to slow down with larger data sets.

                               TSGal
                               FileMaker, Inc.

                          • 10. Re: "Preserve external container storage" makes imports incredibly slow
                            jnouwen

                                 I just want to be sure that my understanding of what the "Preserve external container storage" option is supposed to do is correct. Is it intended for the import process to run without needing to copy the actual container file data and instead just transfer the necessary metadata to the destination record?

                                 If so, shouldn't using that option be almost as fast as not importing the container field at all?

                                 It was 8.5 hours to copy 45MB of data from a 229GB source dataset; now that we're over 350GB, it's only going to be worse.

                            • 11. Re: "Preserve external container storage" makes imports incredibly slow
                              TSGal

                                   Kepi:

                                   Your understanding of the "Preserve external container storage" import option is correct.  FileMaker Pro reuses the existing external contents during import between FileMaker database files.  Why this is taking longer under large data sets is why it was sent to Development for review.  When I have more information, I will let you know.

                                   TSGal
                                   FileMaker, Inc.

                              • 12. Re: "Preserve external container storage" makes imports incredibly slow
                                dbail22@comcast.net

                                     I can confirm these horribly slow times. I had a file with 70,000 image scans of 100k each.  It took 12 hours.  An even bigger problem..... This file was hosted by a filemaker hosting company.  After importing into a clone file ALL of the scans were gone including the directory itself.  Is there some information somewhere in the scant help available indicating you should not.

                                     1. Close a file on the server.

                                     2. Download the file.

                                     3. Create a clone of the file.

                                     4. Open the file on the server.

                                     5. Open the clone on your local computer.

                                     6. Select to import the data and preserve external container storage.

                                     How can you recover a file like this if this will destroy the data?

                                • 13. Re: "Preserve external container storage" makes imports incredibly slow
                                  jnouwen

                                       Interesting, that seems to indicate the slowdowns are based on record count alone, not dataset size.

                                       TSGal: 16,384 records, ~80GB, 60 minutes

                                       Me: 47,652 records, 229GB, 8.5 hours

                                       dbail: 70,000 records, ~6.7GB, 12 hours

                                        

                                       I recall monitoring the 8.5 run and the "Records remaining" counter did seem to go slower and slower the longer the import went on.

                                  • 14. Re: "Preserve external container storage" makes imports incredibly slow
                                    TSGal

                                         dball:

                                         Thank you for your post.

                                         The images should NOT disappear.  Can you check the server to make sure the data is stored?  What type of images are being saved?

                                         Regardless, I have forwarded your times to our testing department for review.  I will keep you posted as information becomes available to me.

                                         TSGal
                                         FileMaker, Inc.

                                    1 2 Previous Next