by "parse" do you mean you're reading out of the raw CSV file?
Are you sticking the CSV file into a text field?
What if you used Insert from URL to stick the file in a container field, then exported the CSV file to a temporary path, and immediately imported that file?
I'd imagine you need to import instead of "parse" for speed.
Also, plenty of error checking required with the above approach, just in case the files are not there.
When we do the parsing of the CSV, we're taking the individual fields in the row and putting them into separate fields.
The CSV is getting placed into a field via the Insert from URL script step. That works filne.
When I export the text back to a CSV file and try to import it, the carriage rerturns aren't recognized and the entire thing wants to come in as a single row.
The remote service that generates the CSV is rock solid. We error check for the file but its never been an issue.
We want to get away from the parsing method as it takes way too long. Importing will be pretty quick, we just can't get the file into a format that we can use. We're stuck with the remotely generated CSV fomat - that won't change.
Parsing a 45,000 record loop is going to be slow. I'm sure you've already exhausted some optimization steps like "freeze window". An export and import I'd imagine would be much faster. It should be easy to script after what you've already done:
Set Variable $path = "file:" & get(temporarypath) & nameOfFile.csv
Export Field Contents table::csvStorageField to $path
Import File $path
You will just need to specify the import order.
I would think the real issue is why would a CSV file change format just by exporting?
That's the $64,000 question. Tried multipe encoding formats with zero luck so far.
The import/export stuff is easy - that works fine. Its just the format of the exported file that isn't working. We've done the Insert From URL as well as loading the CSV into a web viewer and grabbing that. If we could just do an import from an http source that'd be great.
David Zakary may be parting with a substantial amount of money ($64,000),
Use Export field Content, not Export Records.
When exporting 'records' the return is the record delimiter so a return inside a field is not allowed in the exported content.
Thanks Jimmy. That solved the missing carriage return issue. The remaining issue is that we'd like this to be a server-based, scheduled script. The Export Field Contents isn't server friendly.
I guess we'll set up a robot machine or something similar to do the import.
Try putting your csv file in a field in a table that only has 1 record in it. Then you can do a regular export records on the FM server. A possible alternative would be to put the csv file in a global field and then export.
Not sure if this will work in your situation, but think that it will.