Built-in Background Sync

Idea created by Chris Irvine on Oct 27, 2015

    Most of the ideas submitted thus far have been incremental improvements, so I'm just going to put this game changer out there. It's a biggie I know, and it might even plunge us into another file format change. I clearly don't have everything figured out, but I have been thinking about how this might work ever since Go started gaining momentum.


    FileMaker excels at making hard things simple. The product is very accessable and a simple step up from organizing data with spreadsheets. The technology behind syncing is pretty hard, and it is out of reach for many novice developers. Further more, fluid syncing is now ubiquitous in our mobile world. Customers expect to read mail, listen to podcasts, or catch up on headlines without thinking about their connectivity or where to find the sync button.


    There has been some great work on 3rd party syncing solutions for FileMaker Go, and many of us have rolled our own, but they all get in the way of our desired user experience. Reserving chunks of development, or worse, training time for sync is a distraction. In order for sync to disappear from the end user's radar, it needs to be built-in.


    Important Capabilities

    • Asynchronous – Syncing should always in a separate thread without interfering with UX or scripts. The only indication of an active sync should be spinning arrows in menu bar.
    • Easy Initial Adoption – A novice developer should be able to host a data file and then attach a slave solution, at least for download only, without mastering any complex topics or using a command line interface.
    • Near Silent Failure – Intermittent connectivity or an unavailable server should only notify the end user if the developer allows it.
    • Multiple directions – down, up, and bi-directional
    • Good Server Scalability – Hunderends of users should be able to sync against a server, either via intermittent connections or lightweight protocols similar to IMAP IDLE.
    • OnSyncFinish Trigger – Allow developers to react after a partial or complete sync.
    • Scripted Bootstrap – Developers should be able to script a process to initiate a rebuild or initial download.
    • Table Level Configuration – A solution might have a mix of remote (online) data sources, local data sources, and synced tales.
    • Support FileMaker Go – online/offline support for FileMaker Go opens up a huge range of solution possibilities
    • Support FileMaker Pro – local data w/ deferred database updates opens up a new range of WAN based solutions that perform like local solutions and with high availability
    • External Container Storage, Authentication, Encryption – yup
    • Not a Server to Server sync – Sufficient third party solutions exist for that.



    Potential Directions

    • Promote use of UUIDs for primary keys. Maybe the sync direction is limited to download for any tables that include auto enter serial numbers.
    • Get(RecordID) might have to be retired for Get(RecordUUID)
    • A "specify" dialog might be attached to table definition where you configure sync options:
      • Master server: {file reference} table: {table name}
      • Sync every {n seconds or manual or after: {table name in this file}}
      • OnSyncFinish: {script reference}
      • Direction: master>slave; slave>master; bi-directional
      • Conflicts: latest edit wins, early edit wins, on conflict run: {script reference} (I could live with only the latest edit wins.)
      • A sync activity window could be available for power users that want to observe sync status, pause syncing, or rebuild.
    • Syncing should probably not work like or count against normal server connections. Maybe there is a sperate license for concurrent sync connections, where the first one is free.
    • We probably need a way to throttle sync bandwidth on the server, so that a one or more new users don't flood a limited or metered connection.
    • Somehow, editing fields definitions is prohibited on the slave and schema changes automatically propagate from the master to the slave.
    • We're going to need another log.
    • HTML5 Offline Mode for Web Direct isn't expected until version 17



    There are a ton of related features we could ask for, but it feels like basic capabilities above would cover 80%. Did I omit a core component? Advanced developers could still work up solutions that require more complexity such as record merge or sequence generation. The main thing is offloading the data transfer to a background thread.