I am looking for best practice tips for my unique (I think) use of FMP. I receive raw data from customers and I must transform each lines and possibly create many lines from one. the original files typically contain from 10k to 100K lines. I script variables to hold each field and transform it's value before switching to the Output table. on the output table I create a new record and insert the values from the variables.
That process is necessary in my case to always preserve the original data. This method works fine, but is getting painfully slow when the original file gets above 20k and may take hours when getting close to the 100k.
The transformations manly consist in simple string aggregation.
everything is done on a local database on my computer.
Is there an other method I can use?