8 Replies Latest reply on Oct 15, 2014 3:40 AM by NickLightbody

    Your invitation to join the FileMaker Performance Data Project

    NickLightbody

      After attending Ray Cologon's stimulating FM13 Masterclass in London this February Alan Stirling and I had a discussion about the fastest means in Filemaker 13 of undertaking various operations. The result of which was we decided to build a FM Performance Data Server to which our various solutions and Apps could automatically upload the performance data we collected for comparison.

       

      Our logic was that if we could see how long on average - for example - specific record creation and deletion techniques took to complete - we then had good baseline data from which to judge and improve our designs.

       

      This has enabled us to be pretty confident about how fast some things can be expected to be in specific environments and in particular comparing FMP over WAN with FMGo and FM WebDirect and comparing server side scripting with the traditional client side stuff.

       

      So now - in order to benefit our whole community with this tool - on the basis that the more data we collect the more we will all learn - and everyone should have a clear idea of what is normal performance and what is poor performance - we invite every Filemaker Developer to join this open project - FOC.

       

      Attached you will find a small open example file which demonstates how to collect data, how to upload it to one of our Performance Servers and how to access the data on that server.

       

      So you can now - if this interests you - download the example file and get started!

       

      Please let me know on this list and/or email me on nick.lightbody@deskspace.com whether you have any difficulty in incorporating the collection method - or other suggestions?

       

      Here are some tips:

       

      First - open the example and try it out:

       

      • The first button creates a Data Record which action also creates an Event record
      • The second button uploads all the Event Record(s) not previously uploaded - once the LogCentral file has been opened - the upload is fairly quick as record creation runs server-side.
      • The third button opens LogCentral so you can find sort and analyse the data

       

      • We normally call the Upload script in a solution as part of the logout / close procedure.
      • It is essential to name the scripts on which you wish to collect data so that it is obvious what they do
      • Use the xOnSvr function to record whether or not your script is running serverside
      • Use the $$Version global in the Capture Environment script to pick up the current version name / no of your solution

       

      • The example file includes the LogCentral server as an external data source - this obviously costs on performance when you startup your solution as it has to look for the EDS. Once you have the right bits of code in place and collection and uplaod working as intended we recommend that you remove the external datasource - disable the upload script and go collect data.
      • Only add the path back into the EDS when you are ready to upload your data.

       

      Second - incorporate the parts of the example file that collect and upload data into your own solution:

       

      1. Import the custom functions from the example file into your solution
      2. Add the External Data Source - LogCentral
      3. Copy the Events Table into your solution
      4. Copy the za4cts field into the TO used by your normal Ui layout
      5. Create a relationship from your normal Ui TO to the Events Table permitting records to be created through the relationship
      6. NB: Event record creation is through picking up a current TimeStamp, setting za4cts to that value then setting the timestamp field in the events table to that value
      7. Copy the Capture Environment script into your solution and call it from your startup script
      8. Copy the New Event script into your solution
      9. Set the variable $$startUTC to get(currenttimeUTCMilliseconds) at the beginning of any process on which you wish to collect data - at the end call the New Event script and then clear the $$startUTC value
      10. Copy the Upload Performance Data script, Open LogCentral script and Access LogCentral scripts into your solution

       

      No doubt we can improve on what data we collect and on how it can be analysed - so your suggestions for improvement will be welcome - and in doing so you may be volunteering to under take such improvements yourself for thebenefit of all the users.

       

      Here is what we currently collect and record:

       

      Scriptname

      Period that operation took to complete

      Host Name

      IP Address

      Current Host TimeStamp

      Platform

      Operating System Version

      FMServer Version

      Whether communication with the server is encrypted

      Whether the file is encypted

      The type of Device

      The current device display scale

      ConnectionType - a string describing the type of client and server - such as MacOnPC, iiPhoneOnMac or WDOnPC

      FMClient version

      User Account Name

      User PrivilegeSetName

      Device Persistent ID

      Current user TimeStamp

      Your solution version

      Whether or not server side scripted

      DBEngineVersion

       

      Cheers, Nick Lightbody & Alan Stirling

       

      The example file was replaced with v3

        • 1. Re: Your invitation to join the FileMaker Performance Data Project
          jbante

          There are 2 things I always do with performance testing that might be worth considering adding:

           

          1. I always include a "Pause Until Start Of Next Millisecond" script before setting the start time to take some of the randomness out of timing for very fast operations:

          Set Variable [$start; Value:Get ( CurrentTimeUTCMilliseconds )]

          Loop

          Exit Loop If [Get ( CurrentTimeUTCMilliseconds ) > $start]

          End Loop

          Exit Script []

           

          2. For any comparative performance data, i.e. comparing several ways to achieve the same effect, I set up a "null" case whenever possible to time the overhead of running the test independent of the operation being tested. When the test overhead is a large fraction of the total time, it can mask much larger real differences in the performances of different methods.Screen Shot 2014-09-23 at 9.40.51.png

          • 2. Re: Your invitation to join the FileMaker Performance Data Project
            gdurniak

            Just be careful

             

            As shown here,  performance data can be misleading

             

            https://fmdev.filemaker.com/message/155693#155693

             

            greg

             

            > Please let me know on this list and/or email me ... or other suggestions ?

            • 3. Re: Your invitation to join the FileMaker Performance Data Project
              NickLightbody

              Hi Greg

               

              Thanks - I appreciate your note of caution.

               

              We started this project in February - and provided significant performance data to FMI earlier in the year - we would hope that as we build a larger set of records from a variety of sources we can develop some metrics which have some substance - for example across a variety of operations the difference between:

               

              • different versions of WebDirect
              • different hosts with different resource specs
              • different versions of FMGo
              • etc

               

              The current system which we have opened to public access here has just a few thousand records in it but this should build quite quickly as people start to use it.

               

              So from one perspective where folk are trying to improve their system performance if they can see that their current method takes 7 x longer than a common yardstick for a similar operation in a similar environment it should assist them in understanding that the issue is not with FileMaker it is with their design and hence help them to improve their design.

               

              From another - as we build a large set of data - interaction with that data itself could form a useful example of performance in its own right.

               

              We do need someone who is good with statistics to mastermind improving the output side - any volunteers?

               

              Cheers, Nick

              • 4. Re: Your invitation to join the FileMaker Performance Data Project
                NickLightbody

                Hi Jeremy

                 

                Thanks - two very useful ideas - we will put both on the list for the next version.

                 

                Cheers, Nick

                • 5. Re: Your invitation to join the FileMaker Performance Data Project
                  bigtom

                  How do you know what script steps are in the script that are run? I am trying to see how you compared the scripts from various solutions.

                   

                  Are you providing back a report on average of the data collected or that  vs the data collected from each individual solution?

                   

                  Is that the statistical analisys you need help with? I should be able to help. Msg me with what you need or want. I have a very strong math backgrond.

                   

                  Speed is huge thing for me and I take it very seriously as it is a paramount part of the user experience.

                  • 6. Re: Your invitation to join the FileMaker Performance Data Project
                    NickLightbody

                    Hi Tom - yes let's have a chat - I agree with you that speed is everything

                    Cheers Nick

                    • 7. Re: Your invitation to join the FileMaker Performance Data Project
                      NickLightbody

                      Hi Tom

                      Thank you for volunteering to take a look at how we extract performance data from the collected data on the server.

                      Looking forward to seeing what you come up with.

                      Cheers, Nick

                      • 8. Re: Your invitation to join the FileMaker Performance Data Project
                        NickLightbody

                        Thanks to everyone who has taken a look at this.

                         

                        We now need to create a subscriber table and enable folk to register, mark their Event records as theirs and filter / search for the records they wish to compare.

                         

                        We also need to sort out an appropriate method of doing the stats and displaying the results which BigTom is working on in Japan - thank you BigTom.

                         

                        We will post here when we have something more to announce.

                         

                        Cheers, Nick