8 Replies Latest reply on Jul 8, 2016 1:43 AM by Vincent_L

    FMI needs to look at and improve it's testing process. Discuss.

    mrwatson-gbs

      The errors which have become evident in FileMaker 15 in the last weeks are unacceptable.

       

      1. Databases not opening because of (internal) temporary file problems/clashes.
      2. External file References not resolving properly.
      3. Copying + Pasting a Popover Button with <<MergeField>> name trashes the object AND the layout it is in.
      4. Fm15 client on an fm 15 server on mac does not work.

       

      Such basic errors surely must be found during testing, before the product is released? What is going wrong here?

       

      When errors exist that damage  databases (e.g. #3), FMI should have a warning system, to inform developers and avoid unnecessary damage to their work, shouldn't they?

       

      Discuss

        • 1. Re: FMI needs to look at and improve it's testing process. Discuss.
          Fred(CH)

          Not sure it is the correct place to discuss that. Nevertheless, yes i agree with you.

           

          Just a "plus" note, you have surely noticed the big effort of FMI to solve a lot of existing acknowleged legacy bugs reported in previous versions.

           

          Also, maybe the lack of test quality is the result of the new "fashion" to release a "major version" each year. That let probably few time for the testing process ?

           

          Fashion which is probably generated by the business / marketing depts... I think FileMaker is actually extremely focused on its direct revenues (runtime deprecation, new licensing model, best marketing efforts, priced update each year) and less on the development of its product (in term of new features).

           

          My wish : more resource in Testing and Dev dept, and a bit less on Marketing dept. In other words : The package is important but where is now the genius inside the package ???

           

          Fred

          • 2. Re: FMI needs to look at and improve it's testing process. Discuss.
            mrwatson-gbs

            Fred(CH) wrote:

             

            Not sure it is the correct place to discuss that.

            It seems to be the only place available to me. Certainly better here than in the main Discussions area.

             

            Nevertheless, yes i agree with you.

            It's a tough topic, ... but I feel it needs to be openly discussed, ... and that is what we are doing.

             

            Fred(CH) wrote:

             

            Also, maybe the lack of test quality is the result of the new "fashion" to release a "major version" each year. That let probably few time for the testing process ?

            My argument is that that is unacceptable, thus I wish, too, that FMI should look at and improve it's testing process.

             

            …But I'm willing to consider other standpoints.

            • 3. Re: FMI needs to look at and improve it's testing process. Discuss.
              richardsrussell

              When 15 came out, I started a discussion thread here called "Whatcha all think of 15?", which I began by acknowledging that I'm a slow adopter, since I normally wait for y'all to do my gamma testing for me and for the bug-fix X.01 release to come out.

               

              In the absence of any compelling benefit to upgrading to 15 (something akin to, say, the Top Navigation part that 14 gifted us with), I still haven't made the switch. Is it safe to go in the water yet?

              • 4. Re: FMI needs to look at and improve it's testing process. Discuss.
                hschlossberg

                All software gets released with bugs, usually minor, some more obscure than others.  It's just impossible to catch all potential issues and uses and configurations that FM users might come up with. 

                 

                I think the disappointing piece of this is that it takes sooo long to get things fixed.  Minor bugs only seem to get fixed on major new releases, and major bugs (other than security issues) only seem to get fixed in quarterly v-revs, if at all (i.e. the Adobe temp fonts issue).

                 

                Now that FM15 has a sort of self-updating mechanism built in, I'd expected updates to be coming every few weeks, as groups of pertinent issues are resolved, or as usability tweaks are available (as an example, the "check for script errors" function was included in 15 as an unfinished feature; I'd have hoped that the two months since release would have given them time to finish the feature and deploy it as an update). 

                 

                Hopefully FMI is on a track to address issues on a more timely basis, but it doesn't seem that way so far in 15.

                 

                Howard

                • 5. Re: FMI needs to look at and improve it's testing process. Discuss.
                  Benjamin Fehr

                  As I said before, FMI is running out of favours and I don't know for how long I'm willing to spend any contributions to this forum or waste my time with bug reports.

                   

                  Just a few Bugs reported more than a Year ago:

                  Anchors / dynamic layout fail with iPhone 6

                  Script with Script Task "Speak" freezes after several loops

                  WYS ≠ WYG - FMPro vs. FMGo

                   

                  Feature requests that are imperative:

                  Bug Report inventory AND Bug fix protocol by FMI

                  FMI Whitepaper "Under the Hood" for every new release

                   

                  … and a answer to this is overdue anyway:

                  Still terrible miscalculation in Swiss Filemaker Store!

                  • 6. Re: FMI needs to look at and improve it's testing process. Discuss.
                    Benjamin Fehr

                    Fashion which is probably generated by the business / marketing depts... I think FileMaker is actually extremely focused on its direct revenues (runtime deprecation, new licensing model, best marketing efforts, priced update each year) and less on the development of its product (in term of new features).

                    A poor try to compensate weak revenues caused by the self inflicted FM12 disaster.

                    FMI is doomed if it is driven by greedy marketeers with no sense of sustainability.

                    • 7. Re: FMI needs to look at and improve it's testing process. Discuss.
                      mrwatson-gbs

                      It is not my aim in starting this discussion to stimulate ranting about bugs that have not been fixed - we all have our pet bugs we would like to have fixed - rather to look positively at FileMakers testing processes and ask if there is a better way to find and eliminate more bugs, before the product goes to market.

                       

                      As far as I am aware, FMI have cut back considerably the number of testers, and I wonder if within this new test-schema there are too many holes.

                      • 8. Re: FMI needs to look at and improve it's testing process. Discuss.
                        Vincent_L

                        The real problem is that FMI behaves like they to don't care about bug reports, and like they think those bugs are acceptable, otherwise we would have much more frequent bug-fixs updates.

                        I was part of FDS, to get an early preview, there's just no way to report bugs during that time. And even if you do so, they're not fixed. So that experience is frustrating.

                        They seem to have an agenda ans stick to it regardless of feedback.

                        FM 12v0 was unacceptably slow till v4, they knew at leat 1 month before release, and it became usable by v4 so many many months after release. FM 12 was trashed a lot about it's slowness, but they didn't care till v4

                        FM7 introduced a very bad data destroying bug : matching filed change on imports if new field appeared, it was solved in FM 12 (albeit half fixed, you had to re-edit every existing imports to get it really fixed)

                         

                        I said behaves like, becauseI don't want to think that's it's like that because it's been decided to be like that, and that I hope the new update mechanism, and the new cleaned 64bit code base (that must have thrown out some hard to support legacy code in it), will help them and is part of a plan to address things fasted.

                         

                        But 2 months after 15 release there's no bug fix yet at all, and yes they've cut back number of testers, so I guess we're wrongly headed