Several thoughts, no certain solution yet...
You say your script works, but can you tell us what it's scripted to do?
If this is a loop, they are known to have potential maximum reps due to memory limits.
Could this be done via an import process (or is that what's scripted)?
If it's an import process, you could perform it while on a layout which doesn't even include the container field from that table, avoiding most screen refresh issues.
Another thought that comes to mind would be to perform the script in an intermediary FM file, containing just the container field and primary key, along with any data being captured from the imported PDFs, then use that populated file as the source for importing into your more complex FM file.
I recall when switching from FMv10 to 11, and again with 12, that importing large groups of PDFs became a very long and slower process, and one had trouble grabbing only thumbnails, which had worked fine in 10. Files started getting bigger faster, and I saw cache problems when working on server files, but not so much when working with a local unhosted file.
The script is inserting a pdf into a container with a script set path variable. One record at a time. The loop is only going maybe 2000 to 2500 records before slowing and eventually becoming unresponsive.
An import process is not viable due to the number of files in the directories being accessed.
What happens if you set the script to stop after looping 1000 times, and start it again? As painful as it may be, triggering the script manually 400 times in a row might end up being quicker than dealing with the fallout from the current situation.
Then again, maybe having one script loop ~400 times with a nested call to the insertion script (that runs for 1000 records and then ends) is workable.
Did you check memory usage of FileMaker?
With 400K PDF files even a small memory leak in FileMaker would add up to a huge waste of memory.
And 32bit FM Pro will crash on Windows if around 1.9 GB of memory is used...
That's basically waht I'm doing. Once I hit about 2000 Filemaker becomes completely unresponsive. The problem is it won't even quit on it's own. It has to be force quit. I can close the file first. But this will take me days to do at this rate.
I'm sure that's the case. But it doesn't really solve the problem. ;-) I'm going to try doing this on a Mac. But the files are on a Windows volume and it's not easy or efficient to move them around. Fun, fun.
While I don't have a fix I have been doing this with larger sets of PDFs with no issue. One import was over 20000 records. My script sounds like yours. Looping through records and inserting PDFs.
On Mac or Windows by chance?
On Mac FileMaker will run out of memory at about 3 GB being used.
So does in Activity Monitor FileMaker show memory usage increases?
Well, it seems to be related to doing individual inserts, or the fact that I'm doing an Interactive field. If I turn off the Interactive field properties, I can't "Insert as PDF" which is what I need.
If I do a bulk Folder Import, it seems to work fine, and then I can turn on the Interactive properties. This has been a real mess.
So first step, is pulling the files into folders to get the # of files down to a reasonable level. Second bulk Folder import, then I'm going to have to move the rest of the data/index information I need into this file. Total PITA, but in the end it should work.
A work around may be
use another layout that use 'image' instead of 'interactive'
use 'insert file' instead of 'insert PDF'
after insert, 'set field [the object; "image:" & the object ]'
I tested this with 5000 records (using one PDF to all records). not 'reference only', not 'external storage'
on a PC that crashed inserting PDF to 'interactive' about 150 records.
I wish I had a solution for you but I am doing just what you are suggesting. Importing via script with insert PDF to interactive containers. Are you storing these on remote drives? It does not sound like but I thought I'd ask. How about your directory structure? Is it more that 2 levels deep? (this was something FileMaker support said to look at.)
What are you using as your pdf viewer and browser? I have had problems with Actobat 11 reader and IE 11 in one install where things started to slow down on viewing pdf which was fixed but rolling back to Acrobat Reader 9 and IE 10.That bieng said, this never impact on the import or export of the PDFs but maybe it is having some impact there?
Are you trying to import 400K pdfs into one directory?
Hope this helps
1 of 1 people found this helpful
Your post reminded me of a major problem with files which use open external storage of container data:
If you don't structure your storage to keep the file count per directory under 1000 files, the OS-level directory system become unresponsive. This is why external storage of containers should use the FM-managed storage rather than open storage. The FM-managed file structure keeps the directory contents under any limits that will slow the system.
I recall testing a large PDF import when 12 first came out, and Open storage failed after a few thousand records without an elaborate system of storage folders and means of assigning files to them as one processed the PDFs.
Try copying 40K files into one directory/folder on any disk via the OS and you will see the whole OS grind to a halt....
I have been using container that are stored Externally using the Open Storage option. I then have further directories defined by the database. I have gone well over 1000 files in these directories using this method with no issues.
I don't understand your statement.
"This is why external storage of containers should use the FM-managed storage rather than open storage."