1 2 Previous Next 21 Replies Latest reply on Apr 18, 2014 3:14 PM by LSNOVER

    Problems inserting PDFs in Container field

    LSNOVER

      Hi Folks:

       

      I'm bringing in a LARGE number of PDFs into a Filemaker database using interactive container fields.

       

      I have approximately 400K documents.

       

      I have a script that works. The problem is I get through maybe 2K records and Filemaker just goes off to never never land. I get the hour glass, and it looks like it might still be working, but it isn't. I then have to kill the process and start it up again. This results in the file being checked for consistency, and then I repeat the process. Not ideal.

       

      I've tried making the window very small to minimize redraw issues and the like. It's a very simple table, just some descriptors, the file name and the container.

       

      I'm running on Windows 7 with Filemaker Pro 13 v2. I have 512MB set for the cache.

       

      Anyone else have any suggestions on how to make this process more reliable? I'm guessing there's some kind of memory leak, and after X number of inserts, it's just walking off the deep end.

       

      I'm running this all single user FYI, I tried it with the DB on the Server and it was even worse. I'm commiting after each insert. and I'm refreshing every 1000 records.

       

      Thanks,

      Lee

        • 1. Re: Problems inserting PDFs in Container field
          Stephen Huston

          Several thoughts, no certain solution yet...

          You say your script works, but can you tell us what it's scripted to do?

          If this is a loop, they are known to have potential maximum reps due to memory limits.

           

          Could this be done via an import process (or is that what's scripted)?

           

          If it's an import process, you could perform it while on a layout which doesn't even include the container field from that table, avoiding most screen refresh issues.

           

          Another thought that comes to mind would be to perform the script in an intermediary FM file, containing just the container field and primary key, along with any data being captured from the imported PDFs, then use that populated file as the source for importing into your more complex FM file.

           

          I recall when switching from FMv10 to 11, and again with 12, that importing large groups of PDFs became a very long and slower process, and one had trouble grabbing only thumbnails, which had worked fine in 10. Files started getting bigger faster, and I saw cache problems when working on server files, but not so much when working with a local unhosted file.

          • 2. Re: Problems inserting PDFs in Container field
            LSNOVER

            Steve:

             

            The script is inserting a pdf into a container with a script set path variable.  One record at a time.   The loop is only going maybe 2000 to 2500 records before slowing and eventually becoming unresponsive.  

             

            An import process is not viable due to the number of files in the directories being accessed.

             

            Thanks,

            Lee

            • 3. Re: Problems inserting PDFs in Container field
              danielfarnan

              Lee,

               

              What happens if you set the script to stop after looping 1000 times, and start it again? As painful as it may be, triggering the script manually 400 times in a row might end up being quicker than dealing with the fallout from the current situation.

               

              Then again, maybe having one script loop ~400 times with a nested call to the insertion script (that runs for 1000 records and then ends) is workable.

              • 4. Re: Problems inserting PDFs in Container field
                monkeybreadsoftware

                Did you check memory usage of FileMaker?

                With 400K PDF files even a small memory leak in FileMaker would add up to a huge waste of memory.

                And 32bit FM Pro will crash on Windows if around 1.9 GB of memory is used...

                • 5. Re: Problems inserting PDFs in Container field
                  LSNOVER

                  Daniel:

                   

                  That's basically waht I'm doing.  Once I hit about 2000 Filemaker becomes completely unresponsive.  The problem is it won't even quit on it's own.  It has to be force quit.  I can close the file first.  But this will take me days to do at this rate.

                  • 6. Re: Problems inserting PDFs in Container field
                    LSNOVER

                    Christian:

                     

                    I'm sure that's the case.  But it doesn't really solve the problem. ;-)  I'm going to try doing this on a Mac.  But the files are on a Windows volume and it's not easy or efficient to move them around.  Fun, fun.

                    • 7. Re: Problems inserting PDFs in Container field
                      MartinCrosman

                      While I don't have a fix I have been doing this with larger sets of PDFs with no issue. One import was over 20000 records. My script sounds like yours. Looping through records and inserting PDFs.

                      • 8. Re: Problems inserting PDFs in Container field
                        LSNOVER

                        Martin:

                         

                        On Mac or Windows by chance?

                        • 9. Re: Problems inserting PDFs in Container field
                          monkeybreadsoftware

                          On Mac FileMaker will run out of memory at about 3 GB being used.

                          So does in Activity Monitor FileMaker show memory usage increases?

                          • 10. Re: Problems inserting PDFs in Container field
                            LSNOVER

                            Well, it seems to be related to doing individual inserts, or the fact that I'm doing an Interactive field.  If I turn off the Interactive field properties, I can't "Insert as PDF" which is what I need.

                             

                            If I do a bulk Folder Import, it seems to work fine, and then I can turn on the Interactive properties.   This has been a real mess.

                             

                            So first step, is pulling the files into folders to get the # of files down to a reasonable level.  Second bulk Folder import, then I'm going to have to move the rest of the data/index information I need into this file.  Total PITA, but in the end it should work.

                            • 11. Re: Problems inserting PDFs in Container field
                              user19752

                              A work around may be

                              use another layout that use 'image' instead of 'interactive'

                              use 'insert file' instead of 'insert PDF'

                              after insert, 'set field [the object; "image:" & the object ]'

                              I tested this with 5000 records (using one PDF to all records). not 'reference only', not 'external storage'

                              on a PC that crashed inserting PDF to 'interactive' about 150 records.

                              • 12. Re: Problems inserting PDFs in Container field
                                MartinCrosman

                                Hi Lee

                                I wish I had a solution for you but I am doing just what you are suggesting. Importing via script with insert PDF to interactive containers. Are you storing these on remote drives? It does not sound like but I thought I'd ask. How about your directory structure? Is it more that 2 levels deep? (this was something FileMaker support said to look at.)

                                 

                                What are you using as your pdf viewer and browser? I have had problems with Actobat 11 reader and IE 11 in one install where things started to slow down on viewing pdf which was fixed but rolling back to Acrobat Reader 9 and IE 10.That bieng said, this never impact on the import or export of the PDFs but maybe it is having some impact there?

                                 

                                Are you trying to import 400K pdfs into one directory?

                                 

                                Hope this helps

                                Martin

                                • 13. Re: Problems inserting PDFs in Container field
                                  Stephen Huston

                                  Your post reminded me of a major problem with files which use open external storage of container data:

                                   

                                  If you don't structure your storage to keep the file count per directory under 1000 files, the OS-level directory system become unresponsive. This is why external storage of containers should use the FM-managed storage rather than open storage. The FM-managed file structure keeps the directory contents under any limits that will slow the system.

                                   

                                  I recall testing a large PDF import when 12 first came out, and Open storage failed after a few thousand records without an elaborate system of storage folders and means of assigning files to them as one processed the PDFs.

                                   

                                  Try copying 40K files into one directory/folder on any disk via the OS and you will see the whole OS grind to a halt....

                                  1 of 1 people found this helpful
                                  • 14. Re: Problems inserting PDFs in Container field
                                    MartinCrosman

                                    Hi Stephen

                                     

                                    I have been using container that are stored Externally using the Open Storage option. I then have further directories defined by the database. I have gone well over 1000 files in these directories using this method with no issues.

                                     

                                    I don't understand your statement.

                                     

                                    "This is why external storage of containers should use the FM-managed storage rather than open storage."

                                    1 2 Previous Next