The way this is set up creates a lot of redundant and useless work.
If you could have the electronic machine stop writing to the same file from time to time and create another, new file, everything would be much easier.
We do have a similar situation at a client - there's a weather machine outputting a csv data file with 5 - 100 lines every 30 minutes to a ftp location. The filemaker database connects to the ftp site (we're using the 360Works FTPeek plugin for this), gets the directory, downloads every file and then deletes each downloaded file from the ftp directory. Every file is read into a text field of a new record in an import table, then the text field is processed (and split into single records, into another table). The import record is marked as "processed". Everything works flawlessly, no checking needed.
Maybe you can recreate such a scenario ?
Thank you for the recommendation. Yes, it is much less complicated to update/add to my local FM file when the source file writing is suspended periodically. I can make this happen; so that on import, only newly created records are added.
I am thinking that my goal will be to have a solution which scripts passing the records from the source via a socket connection to my FM table. Not sure which scripting language would suffice.