Complete overhaul of FMI's major release pre-shipment testing & tools

Idea created by Vincent_L on Aug 19, 2017



    Since Filemaker 12, all but one (13) major release, suffered (and suffers) from major bugs, even data threatening ones.

    In fact, if those issues were knows at launching time, most people wouldn't have deployed those releases, because they're not at all risk free for the solution and data health.

    That's even more a problem if you pay your licence yearly (as FMI pushes for), because you're paying for the latest and greatest, that you can't use safely for months if ever.

    And this will be even worse for the cloud version, because FMI will probably try to push the latest version as soon as possible.


    Let's recap some of those issue :


    - FMP 12 : Extreme slowness that was moderately fixed in v4, at least 6 months after release. I ended up skipping that version

    - FMP 13 : no problem i can recall with that one

    - FMP 14 : Even after waited for first .2 release, I battled for 6 months with FMS random crashes, those were fixed in v4

    - FMP 15 : FMS  corrupt cache issue I documented. Still no fix

    - FMP 16 : Still corrupt cache issue, plus unstorred calc that don't refresh, and script that vanishes

    And that's just the important stuff, there's also lots of GUI issue (terrible filed picker, non resizable script maker script column for a year on the mac,

    terrible new value list function performance in 16v1)


    In retrospect, most of us who value data coherence would advise not to deploy something else than FMP 14, which is 2 releases yet.

    But in the other hand, since then a lot of bugs wee fixed in 15 and then 16, so if you were hit by those you can be tempted to go forward.

    However those fixed bug are much less an issue than the 15 & 16 introduced ones.


    So having a platform, that's only reliable two release before the current one, is a serious underachievement for a software company to say the least.

    And especially if it pushes for yearly licensing scheme, and looks in the cloud with alway up to date software (and with built-in auto-update mechanism).


    So, at the same time of the whole FMI strategy is focused to yearly releases, the release quality, as far as the core function of a databases are concerned, is worse and worse.


    Therefore a major complete overhaul of the pre-release process is more than ever needed.


    But, I've been told that the pre-ealse process has already been made a while ago, in the wrong way.

    In the past I was blessed to be part of the ETS process, but no more, as, I've been told, they decided to narrow down the tester head count.

    You can understand they want more quality feedback, and multiple feedback can be distracting, but the obvious risk is too mis a lot of bugs.

    And obviousness always strike back, and it did in 15 / 16.


    Here's for the current status, whoch itself should provoke some changes.

    But how to improve it ?


    Just changing the tester headcount upward and downward won't help alone, because the problem is that the only real test is, with the platform in production !

    Till FMI doesn't do testing in production, major releases won't be safe.


    Of course, I will be drenched by "you're not testing in production", "there're legal issues fo a company such big as FMI"…

    I'll answer that  every community users of the v.0,v.1,v.2 released version for the past 4 released versions are actually using buggy ß software in production already.


    Of course, early ß should still be kept out of production, and within a limited tester head count, but we have the preview version.

    One or 2 months before shipment, FMI releases a kind of golden version to FDS members.

    But the problem is that it comes with all sorts of warnings dissuading people to use it in production, even though the final version is very very close, if not the same that of the preview.

    The second problem is that FMI doesn't expect, nor does want any feedback on those previews. There's no private forums (last time I checked), and there's no real way to make feedback about it (it's not allowed to talk about it publicly, which I can understand, but hence you can't post in product issue). I was in FDS in FM12, and I immediately saw the terrible speed of it, I documented it at length but had a lot of trouble to send any feedback about it.

    Worse, they act as if they didn't want this feedback, and they won't correct anything, for them it seems that preview is the final. Which also means that actually think it's ok for production, but won't say so.


    But what if they would offer an optional "contract" allowing you to deploy the preview in production, will all disclaimer possible, for those who want to, with a proper way to report bugs. Ideally those user may be getting it a tad earlier than FDS, and maybe for free (if they're in subscription / maintenance licensing model), as an incentive and reward for their testing time, and dedicated forum space to discuss it.


    So the release cycle would be


    internal alpha  - ß current ETS - Production Preview GM - FDS GM

    rather then today's (as far as I know)

    internal alpha  - ß current ETS - FDS GM


    Of course that would also mean that FMi will need to allow some more time in their schedule to fix glaring issues, so v0 would ship without it.

    If current schedule slips for 1 month it wouldn't be the end of the world, if important bug are killed before shipment, we would get much stronger v1 releases, and this will restore confidence in the released version, and hence fits their yearly release cycle strategy much better.


    But those tester need to have some confidence that their testing is safe, that could be ensured by special tools that FMI would provide them. Those tools would be command line utilities (so no gui to create, and this will discourage the average joe) :


    - A tool to extract data from a the new version solution, and putting them in a clone file of the previous version. A la refresh fm, but much faster.

    That would allow Tester to go back to previous version as needed and quickly should a problem arise.

    - A tool to extract the DDR quickly and compare it to another reference DDR to see if something changes, accounting for / ignoring the new introduced feature.

    - An fmdiff like tool that would only compare the structure part of fm solutions

    - A tool to compare the data and indexes between 2 solutions, and highlight the differences, of course there will be some changes, but the user will see those and decides if they're legit or not.

    - And of course a tool to check for corruption


    All those could launched by the server at night, on a backup of the solution, so there wouldn't be any downtime, and the FMS process would offload this to the command line utilities, so not competing with FMS processes. Those could be even offloaded to another computer.

    Of course they'll work solution wide (that means multi-files).


    Their output would be xml, and / or optionally filemaker files (yes command line can create fmp12 files, as filemaker CLI migration tool showed)


    Moreover those tools could be great for everybody and offer stronger reliability. Heck, they can even address most of data separation needs.


    FMI would release the tools, a pre-made command line samples, and maybe even a small filmmaker solution storing the results, with mail notifications.


    With those tools, the testing will be highly secure, and will offer a safe go back route. We would be much more safe, because we'll be able to track any unwanted changes, and react soon.

    For instance the deleted scrip issue of 16, would be tracked easily by the ddr  / structure ones. Even if it occurred in a much less used database file (I've 120 files in my solution, but some files aren't used that often, I wouldn't notice a script deletion issue for weeks if it would occur in those files).


    Those will be even better whenever filemaker format changes, because nowadays, if you migrate you can't go back practically. Therefore tester, there's much less real testing done..


    Of course, those tools, would be super useful even if that pre/parrallel fds testing exists or not.

    But they will provide safe production testing, and this will lead to much sooner problem discoveries.


    Finally, let's address confidentiality. Since FMI, and righfully so, now offer Roadmap previews, there won't be much to leak, and to get the tools and preview you'll have to sign NDAs.


    I realize that non in-house professional, probably wouldn't use that,. But for the in-house people this will be huge, and it will be huge for the platform.