1 2 Previous Next 15 Replies Latest reply on Aug 28, 2015 2:28 AM by IanJempson

    FileMaker Server - Performance Measuring

    CarstenLevin

      I hope that some of us who are interested in this issue can share our experiences here. I took the time to write this text because I would like to hear if we are the only ones who had to tweak larger solutions and had to be able to follow the results during real time use.

       

      Purpose

       

      • To make sure that large systems used by many users (+50) are performing well.

       

      Issues causing performance problems

      • Less than optimal FileMaker development (using unindexed fields, procedures that work well in test environments but becomes slow in production or when data amounts rise etc. etc.)
      • Server architecture/FileMaker Server 11 architecture. Virtual servers that should perform well, 64 bit environments not performing well with FileMaker, I/O bottlenecks, Network problems etc. etc.
      • Other?

       

      Our experience

      For a year we have used our own very very simple performance test at some of our larger customer sites.When a solution is performing slow for the users, one way of knowing is when they tell you!But we needed objective answers (and warnings).

       

      FMS statistics

      How do we measure the performance of a FileMaker Server and isolate the bottlenecks. We did of course use the FMS statistics a lot. But still sometimes when the statistic results was not terribly bad we could see that the UI was slow for the users and at other times when the statistic result implicated bad performance the users actually got pretty acceptable response times.

       

      Our first solution

      Since the FileMaker Server Statistic Page did not show exactly when the users faced delays we needed another test - another reason was that we could not easily program the server to send an alarm sms/email when the performance decreased.

      We designed an independent robot function testing a set of routines on the server. We used a copy of the userinterface of the solution and made a script that simulated user-interaction. Then we created a user interface with reporting and graphs and a robot function sending alarms when the solution had performance problems.

       

      Our second solution

      We wondered: Do you need to do the test-simulation with script based on data from the actual solution and the UI of that solution?

      We created a separate set of performance files and tests and then ran them beside the first performance-test. Our test showed that there was no difference when testing on a special performance file on the server or on data from the real solution. The testresults and slow downs where the same whether we used the generic solution or the files used by the users.

      Conclusion: We could create a generic solution for performance measurement of different servers.

      The only tiny item we are using from the running solution is a field in the data file get(currentusercount) that we use to display the number or users in our test graphs - since we could not find a way to ask the server for the actual usercount independent of a specific file.

       

      The graph from one of the solutions we are monitoring

      For each solution we are setting a tresshold and if this is crossed an email or sms is sent to us or the sysadm.

      codeoperformance2.jpg

      In this case the peak at 3-5 in the morning is caused by a lot of housholding scripts by server or robot, integration with other solutions and setting of stored values for reports.

      This particular graph is from a relatively large solution. Between 10 am and 4 pm apx 70 users will use it heavely.

      We started testing it because the solution sometimes slowed down during the daytime, waisting productive time for the users.

      After implementing the performance test we began ironing out small and large performance hogs and after a while the graph showed that there are not no delays during the daytime.

      Reading the graph: On this server test times between 2 and 6 seconds is showing that the server is answering each call from the user at once.

      A large report taking 30 seconds to generate will still take the time it need and a short procedure finishing in apx 1/10 of a second wil finish this fast. But if the test time increases to 15 seconds the 30 seconds will turn into apx. 38 seconds (not very problematic), but the 1/10 second procedure will have a duration of apx 8,1 seconds (unbearable for the user).

      Most users start working between 8 and 10 - you can see them checkin inn on the graph and that the solution keep performing well.

       

      What are we using it for?

      We are using it for:

      • Measuring the fundamental performance of a specific server/client/network setup.
      • Checking that the system om the server does not have problematic parts causing owerload.
      • Checking that improving/tweaking of the solution actually helps (comparing results in the log before/after)
      • Alarming us if a solution begins to underperform

       

      After trimming and modifying this little testsolution ower a year it is now very unobtrusive. It has to be "heavy" enough to check that the server has enough reserves to deliver to the users without delay - while not adding so much to the burden that it becomes part of a potential problem.

       

      Important: This test can really be used to test how a soluton will work with users without the users. But when we install this test solution on a server the measurement show whether the server is still responding at once while the customers FileMaker solution is in full use. And if it does not we can use it to keep measuring while improving the solution or while tweaking and adding ressourses to the server.

       

      Questions

      • Can anybody give us clues to use the FMS statistic better - where are the best articles on how to interpret and use the statistics?
      • Can the FMS statistic be used to evaluate actual user performance and send alarms when the server is to slow?
      • Can you share you view with us: Is it also your experience that you can actually measure the server-performance and the performance of your customer solution with a generic set of files that can also be used on different servers?
      • Has anyone made a general server performance test solution for FileMaker?

       

       

      Best regards

       

      Carsten

       

      ps. As it is obvious from the text, my native language is not English. I would like to remove the worst grammar/spelling errors. Please email me if you want to help me improving this little discussion-teaser.

        • 1. Re: FileMaker Server - Performance Measuring
          CarstenLevin

          And I feel that I have to add this very performance-measurement-relevant quote:

           

          Not everything that can be counted counts, and not everything that counts can be counted

           

          I would have liked to use this as a little Christmas Quizz, but since Wiki and Google is just around the corner the answer is here: It is of course Albert Einstein ... which lead me to one very important finding by Niels Bohr: When you watch and analyse something within an installation you will learn what will happen, when you watch. But not what would have happened if you did not watch. This is indeed the truth about measuring server performance

          • 2. Re: FileMaker Server - Performance Measuring
            BeatriceBeaubien

            Hi Carsten,

             

            Thank you for your post. I've embarked on establishing a stress test for our new FMSA 11.0v3 production server (Windows Server 2008 R2) (the first of four) and am running out of ideas. Any insights on how to monitor a production environment would be welcome. Test files are another issue I am grappling with.

             

            Mark Richman of Skeleton Key presented an insightful session at DevCon 2011 for monitoring FMS WAN/LAN performance, but I am also interested in your experience.

             

            Best wishes,

             

            Beatrice Beaubien, PhD

            i2eye, Toronto, Canada

             

            FileMaker Business Alliance

            FileMaker 11 Certified Developer

            • 3. Re: FileMaker Server - Performance Measuring
              DavidZakary

              A FileMaker Inc. produced tool for stress testing a Server installation would be a handy thing to have.

               

              What would we need?..

              1. Database of substantial size
              2. Scripted test routines that we could run on a number of client machines
              3. A set of metrics - based on Server logs for different numbers of connected users and network types that we could compare against

               

              Every database would obvously be different, but we've probably got enough "real world" examples that we could put something together. Item #3 would be the key item. We can look at logs and the stats in the Admin console and know what the numbers should be in an ideal environment, but how often do we have that?

              • 4. Re: FileMaker Server - Performance Measuring
                usbc

                "A FileMaker Inc. produced tool for stress testing a Server installation would be a handy thing to have. "

                 

                Perhaps a replacement for the Sample file we all get now...

                It could have a number of test routines which are recorded and presented in a graph(s).

                The install download could be accompanied by a number of CSV files containing various record counts from small to gigantic, to be imported or not.

                The tests could be weighted for a representative selection of hardware / client count / etc.

                 

                Knowing this crowd, even an imperfect first attempt would soon become a indispensable tool for we practitioners and maybe the engineers at FMI.

                • 5. Re: FileMaker Server - Performance Measuring
                  BeatriceBeaubien

                  +1

                   

                  Thanks for this, David.

                   

                   

                  Best wishes,

                   

                  Beatrice Beaubien, PhD

                  i2eye, Toronto, Canada

                   

                  FileMaker Business Alliance

                  FileMaker 11 Certified Developer

                  • 6. Re: FileMaker Server - Performance Measuring
                    BeatriceBeaubien

                    Hi Chuck,

                     

                    Great thread. I like the idea of test routines and differently sized CSV files.

                     

                    And it is possible that this would be useful for FMI engineers to get back standardized parameters of server functions. Jon Thatcher might find this idea interesting.

                     

                    Bringing a number of FMS/A production servers on-line in a short period of time would definitely benefit if a tool like this was available.

                     

                    Best wishes,

                     

                    Beatrice Beaubien, PhD

                    i2eye, Toronto, Canada

                     

                    FileMaker Business Alliance

                    FileMaker 11 Certified Developer

                    • 7. Re: FileMaker Server - Performance Measuring
                      CarstenLevin

                      Hi David,

                       

                      Yes, a common reference could be good. But mayby a solution like this should consist of more than one module.

                       

                      Our present solution is just the test & measuring part. The load on the server is created by the actual users.

                       

                      A generic test/benchmark solution should maybe have an application to run on a number of clients to simulate users. Running a test of just one computer is maybe not enough to find out how a server system is performing.

                       

                      We also have to establish very clear destinctions:

                      • When are we testing the strength of a server setup?
                      • When are we testing the solution looking for bottlenecks?

                       

                      I am really listening here. Anybody having input on really using FileMaker statistics?

                      Anybody having benchmark solutions for FileMaker?

                       

                      Best regards

                       

                      Carsten

                      • 8. Re: FileMaker Server - Performance Measuring
                        lavendt

                        Hi Carsten and other

                         

                        The FMS logs is stored in (Mac path) /Library/FileMaker/Logs/

                        These are basically tab seperated files, which could be imported into a FM file.

                        The stat log also shows user counts.

                        So maybe we could get some usefull information from that....

                        Furthermore there are logs about other actions, such as schedules, start/stop, Username for users, who login etc.

                        And, if you have server plugins, there are also logs for them....

                        • 9. Re: FileMaker Server - Performance Measuring
                          CarstenLevin

                          Hi Claus,

                           

                          Yes, thats right.

                          Does anyone here know if someone has already made a solution reading the logs in and analysing them?

                          • 10. Re: FileMaker Server - Performance Measuring
                            lavendt

                            no, but it should be straight forward to do so....

                            I would consider using a FMPA on the server itself and use Troi File to import the log. When you have most lines of the log, then import the newest lines from the file and insert them to your log table.

                            With simple relation, you could choose interval etc. and show a graph.

                            In the log, you have great info.

                            However, since you have digged into the "engine" you probably know a lot more about, what is interesting data. You could say, that getting the server logs, gives you info about the solutions on that server, and the server itself, without having to interfere with the solutions.

                            • 11. Re: FileMaker Server - Performance Measuring
                              nmurphy24

                              Deep thought,  Are they  heading towars the Heisenberg uncertainty principle on a macro level?

                              • 12. Re: FileMaker Server - Performance Measuring
                                NickLightbody

                                Hi Carsten

                                 

                                Just read your piece from a little while ago - very interesting - perhaps you would be interested in helping with our Filemaker Performance Project?

                                 

                                Cheers, Nick

                                • 13. Re: FileMaker Server - Performance Measuring
                                  NickLightbody

                                  Hi Carsten,

                                  By way of an update on this subject, I have published my recent report on Filemaker Server Performance here

                                  and released our free stress testing tool dsBenchmark here

                                  There are also other resources regarding filemaker performance accessible from the http://www.deskspace.com documents page "Writing about developing"

                                  Cheers, Nick

                                  • 14. Re: FileMaker Server - Performance Measuring
                                    CarstenLevin

                                    Hi Nick,

                                     

                                    Although this started many years ago the issue is of course as relevant today as it was then.

                                     

                                    Thanks for the links, reading now. And maybe other are having input as well?

                                     

                                    4 years later a lot must have changed:-)

                                     

                                    Best regards

                                     

                                    Carsten

                                    1 2 Previous Next