AnsweredAssumed Answered

What is best way to process large data files (many GB).

Question asked by suesaunders on Mar 22, 2017
Latest reply on Mar 28, 2017 by srzuch

I'm processing large financial data files.  We have about 600GB of text files (tick data) that are being imported into around 32 FileMaker files.  The next step is to merge the FileMaker files into several composite files (around 8) which are then each processed using different trading strategies.

 

We have a high end machine operating Windows 10 but note that FileMaker only uses one of the 16 cores available which means that it is taking forever to process the data particularly the imports.  Is it possible, in some way, to have multiple copies of FM client application open, each processing data and each using one of the available cores.  The operation system would allocate one operation to each core. It is frustrating that we have plenty of processing capacity but do not know how to make use of it.  Each FM client application is well able to handle the large files but in order to process all the data in a realistic time frame, I need to come up with a better solution.

 

I have a licence for 5 instances of FM and have FM Server on another machine.  However, the data is on the Windows machine and it is by far the faster beast-y that we have.

 

Sue Saunders

Outcomes