12 Replies Latest reply on Jul 24, 2014 2:37 PM by wimdecorte

    Troubleshooting performance bottlenecks

    Kantala

      Any advice on the best way to troubleshoot server performance bottlenecks?

       

      Setup:

      • FMSA11 on (virtualised) Windows Server 2008 R2 SP1, 64-bit
      • 16GB RAM
      • 2.67 GHz Quad Core CPU
      • Three drives (OS on C, FMS and live data files on D, backups on E)
      • Approx 125 database files, around 100 users connected via Terminal Services

       

      We're experiencing periodic slowdowns at various points, where everything slows to a crawl and client stats go through the roof. This doesn't appear to follow any particular pattern of user activity.

       

      I've checked:

      • backup schedules
      • virus scanning
      • shadow copies
      • RAM utilisation
      • other services running on the server

       

      I've attempted to set up Performance Monitor, but can't seem to get any useful information out of it...

       

      My only idea so far is that it may be down to a lack of space on the C drive. The C drive is 60GB in size, with approx 12GB free space. However, the total size of the database files is around 13GB, which given the recommendation for at least double the total database size in free space on the C drive, is a little lacking. However, the powers that be seem reluctant to increase the size of this drive. Can anyone confirm whether this would be likely to cause the issues we're experiencing, and if so, any idea why it can be ok for hours then suddenly lock up?

       

      Grateful for any thoughts, especially your methods of troubleshooting issues like this.

       

      Matt

        • 1. Re: Troubleshooting performance bottlenecks
          jkrichton

          Are there any other departments that may be tying up network bandwidth downloading or uploading large files?

          You mention 100 users. Are they all terminated to one switch or are there mutiple switches connected?

          I would look at a network monitoring tool to look at the traffic and determine if there is a bottleneck.

          Jeff

          • 2. Re: Troubleshooting performance bottlenecks
            Kantala

            Hi Jeff

             

            Thanks for the advice.  As far as I'm aware there is plenty of network bandwidth to go around.  There are multiple switches within the building, and the load is spread over multiple terminal servers.  The 'network utlization' on the FM Server never goes over 1%.  I will raise the issue with the IT guys though.

             

            Cheers

             

            Matt

            • 3. Re: Troubleshooting performance bottlenecks
              wimdecorte

              Kantala wrote:

               

              and client stats go through the roof.

               

              Which ones specifically?

              • 4. Re: Troubleshooting performance bottlenecks
                Kantala

                Hi Wim

                 

                Specifically Elapsed Time and Wait Time, which I take to mean an issue within the files somewhere, rather than a hardware issue.  I/O Time seems to stay realtively low, although it does peak on occasion. 

                 

                Matt

                • 5. Re: Troubleshooting performance bottlenecks
                  jkrichton

                  Are there any new reports created that might be slowing things down?

                  • 6. Re: Troubleshooting performance bottlenecks
                    Kantala

                    Hi Jeff

                     

                    Not as far I know, although it's difficult to be certain with this many users!  Plus, we didn't develop the system, but merely support it.  I'll investigate, but I gather that users are not doing anything 'out of the ordinary' when the slow downs occur.

                     

                    Matt

                    • 7. Re: Troubleshooting performance bottlenecks
                      wimdecorte

                      Kantala wrote:

                       

                      Hi Wim

                       

                      Specifically Elapsed Time and Wait Time,

                       

                      Those indicate that there is not enough processing power in the machine for what it is being asked of it.  "elapsed time" is what currently being processed and "wait time" is what is in the queue waiting for the processor to become free.

                       

                      There are "expensive" things in the solution that can contribute to this like searches on unstored fields, big summaries,  layouts with many portals,...

                      1 of 1 people found this helpful
                      • 8. Re: Troubleshooting performance bottlenecks
                        Kantala

                        Thanks Wim

                         

                        It's good to get clarity on what those actually mean, as I was a little uncertain.  I guess I've got to bite the bullet then and wade through the 100s of layouts and scripts to track this down, oh joy...

                         

                        Matt

                        • 9. Re: Troubleshooting performance bottlenecks
                          Mike_Mitchell

                          Matt -

                           

                          Not necessarily. Elapsed Time is the time it takes the server to process a request. Wait Time is the time one client waits for the server to process other clients' requests before processing the one he just sent. If they're both high, it means the server is busy. That could mean:

                           

                          1) Insufficient CPU or very CPU-intensive tasks.

                          2) Insufficient RAM or insufficient cache size (too much paging to disk).

                          3) Insufficient hard disk speed (especially when coupled with #2).

                          4) Inefficient database design / processes (dragging down the server processing too much).

                           

                          Looking at your specs, 16GB of RAM is a little thin. But you're right; you are pushing the envelope on the space on the C: drive, too. I would look at my RAM cache and see what it's set to. (General recommendation is 25% of available RAM.)

                           

                          That said, though, since you said the problem is intermittent (i.e., you don't have the problem all the time), but you do have a fair number of clients, it could be that all the users are doing something (or attempting to) at the same time. Is there a pattern to the slowdowns vs. time of day? Perhaps there's a report or other batch operation that multiple users are attempting run at roughly the same time every day?

                           

                          Mike

                          1 of 1 people found this helpful
                          • 10. Re: Troubleshooting performance bottlenecks
                            oldal41

                            How much Memory is allocated to the VM Machine

                            How many processors are allocated to the VM Machine 

                             

                            If Windows Server is using a Swap File try moving it to the E Drive, and make sure it is the right size. 

                             

                            You might try no swap file to a 16GB swap File on E Drive - Isolate System Swap IO's from FileMaker.

                             

                            See links

                             

                            http://search.yahoo.com/search?ei=UTF-8&fr=crmas&p=Windows+Server+2008+swap+file

                            • 11. Re: Troubleshooting performance bottlenecks
                              Kantala

                              Hi Mike

                               

                              Thanks for your reply and suggestions.  CPU never peaks over 50%, RAM utilisation doesn't seem very high, and my limited investigations seem to suggest that there isn't much paging, but I could be wrong.  I'll push them to increase the hard drive space, and may suggest oldal41's idea of moving the swap file.

                               

                              Typically the slow downs occur around the same time of day, so our next step is to try and track down the user(s) that are inadvertantly causing the problems, by looking at the users with the highest elapsed times.  Apparently they've had similar issues in the past, and users have been told not to run certain reports duing office hours(!).  Like I said, we didn't develop the system, just support it.

                               

                              Thanks again for your help.  I'll post back when I have any updates.

                               

                              Matt

                              • 12. Re: Troubleshooting performance bottlenecks
                                wimdecorte

                                Kantala wrote:

                                 

                                CPU never peaks over 50%,

                                 

                                If you are looking at what the OS tools report here then that is not the full picture.  The 50% seems to indicate that half of the processing power is available and it clearly is not.  The processors an be back-loggd with incoming calls but don't need their full power to get through it.  Your slowdowns come from the length of time it takes to clear a thread, not from how much processing power it takes.

                                It's a subtle difference.

                                 

                                What it shows is that you will have more effect from adding more cores to have more threads worked on at the same time.