AnsweredAssumed Answered

Import speed

Question asked by bthez on Oct 27, 2009
Latest reply on Oct 27, 2009 by mrvodka


Import speed


I have been using Filemaker for nearly 10 years. Currently, I run version FM Pro 8.5 on an 2Ghz Intel Core 2 Duo Imac under OS 10.5.8.


Every month I create from scratch a database of approximately 55 million records, which increases in size by about 250,000 records per month. It is not currently possible to just get the new records separately, to append to a previous database. Moreover, there occasionally are records that were previously in the database, which get modified.


I import from an ASCII file of records, each containing 18 tab delimited fields, 13 of which are numeric, and the rest are text. There are no calculations being performed on this fields during input, but indices are being created for all of them. (In fact I import from a series of ASCII files, because the whole file of source data is much larger than 2^32 bytes, and I've had to break it into chunks smaller than 2^32 to be able to process properly)


The import process takes well over a week, 24/7. Admittedly, I use the machine for other things while the import proceeds in he background.


Is there any way that the import time can be improved? Is there anything like a compiled version of a database, which won't be interpreting as it works?


I do have FM Server 7.0, and could import portions of the database on separate workstations, but I've brought the Server to its knees on other projects before, when the record count gets really large.