Hopefully you'll get some feedback from others, but in the mean time, my biased opinion :
This is exactly what we built RefreshFM for. Your issue is going to be the huge amount of data you have, and whether it's feasible to move that much data via imports. Some things that might help :
Building indexes is usually the most time consuming part of imports. Reduce the number of indexes if they're not required, or turn them off temporarily in the field definitions if you can. Don't index huge things like the content of emails, as it's just too big to be sensible for large record counts.
Turn the cache up as high as it will go in the FileMaker application preferences, this helps immensely. If you hit the cache limits, then disk speed will help, use SSDs.
Split the files into separate data files where it makes sense. Especially separate files for container data if you can, so that you can avoid importing those when you don't have to. Try to put the largest data sets by themselves if you can, so that there's the fewest tables in that file.
For importing records with external data, you need to use at least the latest version of RefreshFM v3. There is functionality in there that uses a new feature in 13v2 where it doesn't duplicate container data, it just re-references the original, so you're not also duplicating all 1.5TB of external files.
Also note that you don't need to import the whole solution. If you do have separate data files, you can just restrict RefreshFM to one set of tables from one file. I've used this before to split a massive update into 8 different PCs so it could run in parallel - and it still took 8 hours.
And using RefreshFM for these large imports is also exactly what we've designed it for. I often do client imports that start at 8pm, and I just leave it run. I know that when I check in again at 7am if there's anything that's gone wrong with the import, the original files are still in the same location, and I just re-open them in the server admin, and nothing has changed.
The sort of thing RefreshFM does is certainly something you could build yourself. The advantage of RefreshFM is that it builds it for you, so whenever you change your solution, you just click the setup button again with the new files and it's all done for you. Plus RefreshFM manages all the aspects of it, so you can instruct your IT dept in how to do all of it.
Feel free to ask any questions here, or off list if you prefer.
Thanks Nick! Always nice for the feedback, biased or not
I’ll give this a deeper test and see if we can get where we need to be for IT with this option.
Have a great day!
Laura Bowyer |Sr. Software Developer | Wizards of the Coast, LLC | www.wizards.com<http://www.wizards.com/>