2 Replies Latest reply on Sep 3, 2015 8:03 AM by laura.bowyer

    Feedback on RefreshFM for migrating data (FM12)?


      Hi there!


      We're looking for ways to move large sets of data from a dev server, to QA, then staging, then production. We have about 1.5 TB of assets stored in externally secure containers and about 500GB of data in a few files, using the separation model.


      Our initial idea is the save the assets to the Staging and production servers first, then import any deltas once we move to that environment. Anything to avoid moving 1.5TBs of assets....that's for the assets only. But for the databases.....


      We need to automate the push of data from Dev to QA (we can copy the backup files for this step). But then when we go to Staging and Production, we need a way to wipe out the data, import production records back in and verify it all works, with the push of a button so that someone in IT can do this as needed without our intervention. There are currently 34 tables in the main data file, with the largest table having about 400,000 records. When we do a manual import of data, it takes 8+ hours to complete.


      I've reviewed Jonn Howell's DevCon 2015 session on this subject (thanks Jonn!). That looks good for smaller apps and less data. Pulling all of the container data back into the FM files each time will take more than 20 hours - unacceptable for this.


      I'm checking into RefreshFM from the great folks at Goya, but am having issues getting the demo to run properly.


      Is anyone using RefreshFM for their migration? Any feedback? Options or suggestions?


      Thanks in advance for your feedback!



        • 1. Re: Feedback on RefreshFM for migrating data (FM12)?



          Hopefully you'll get some feedback from others, but in the mean time, my biased opinion :


          This is exactly what we built RefreshFM for.  Your issue is going to be the huge amount of data you have, and whether it's feasible to move that much data via imports.  Some things that might help :


          Building indexes is usually the most time consuming part of imports.  Reduce the number of indexes if they're not required, or turn them off temporarily in the field definitions if you can.  Don't index huge things like the content of emails, as it's just too big to be sensible for large record counts.


          Turn the cache up as high as it will go in the FileMaker application preferences, this helps immensely.  If you hit the cache limits, then disk speed will help, use SSDs.


          Split the files into separate data files where it makes sense.  Especially separate files for container data if you can, so that you can avoid importing those when you don't have to.  Try to put the largest data sets by themselves if you can, so that there's the fewest tables in that file.


          For importing records with external data, you need to use at least the latest version of RefreshFM v3.  There is functionality in there that uses a new feature in 13v2 where it doesn't duplicate container data, it just re-references the original, so you're not also duplicating all 1.5TB of external files.


          Also note that you don't need to import the whole solution.  If you do have separate data files, you can just restrict RefreshFM to one set of tables from one file.  I've used this before to split a massive update into 8 different PCs so it could run in parallel - and it still took 8 hours.


          And using RefreshFM for these large imports is also exactly what we've designed it for.  I often do client imports that start at 8pm, and I just leave it run.  I know that when I check in again at 7am if there's anything that's gone wrong with the import, the original files are still in the same location, and I just re-open them in the server admin, and nothing has changed.


          The sort of thing RefreshFM does is certainly something you could build yourself.  The advantage of RefreshFM is that it builds it for you, so whenever you change your solution, you just click the setup button again with the new files and it's all done for you.  Plus RefreshFM manages all the aspects of it, so you can instruct your IT dept in how to do all of it.


          Feel free to ask any questions here, or off list if you prefer.




          • 2. Re: Feedback on RefreshFM for migrating data (FM12)?

            Thanks Nick! Always nice for the feedback, biased or not


            I’ll give this a deeper test and see if we can get where we need to be for IT with this option.


            Have a great day!




            Laura Bowyer  |Sr. Software Developer | Wizards of the Coast, LLC | www.wizards.com<http://www.wizards.com/>

            laura.bowyer@wizards.com<mailto:laura.bowyer@wizards.com> |Location: 371F  | Office: 425-204-7351 | Cell: 206-226-4903