1 2 Previous Next 21 Replies Latest reply on Aug 20, 2017 5:34 AM by alex.hellier

    Extreme Server wait times on Windows Server

    Paul Jansen

      I already posted this in the community discussions but had no responses.

       

      This is becoming a major issue for a 15 year client and FileMaker user which could result in their leaving the platform...

       

      First here is the setup....

       

      VMware virtual server 10GB RAM, 3 processors, 6 'cores'

      Windows Server 2008 R2 Standard

      FMS 13 single machine deployment

      Typically 15 pro clients mostly LAN, but 1 or 2 WAN

      0 to 15  xml WAN clients through WPE varies through the day. mostly very small text data incoming, but periodically uploading files in to externally stored containers (typically 10-100KB) using base64 encoding and html POST running a FileMaker script to decode and create the record.

       

      Mostly wait times are very low or zero, but periodically, shoot up to several seconds and occasionally 20+ seconds.   There are times when we just get the occasional spike, but others when spiking occurs every few minutes.  These excessive wait times affect all FileMaker Pro users.

       

      stats.png

       

      Network, IO and CPU on server all normal with no corresponding spikes.   IT report no spikes on network.

       

      I thought the slowdown might be related to the file uploads, but the vast majority of uploads complete with no change to wait times and the first of the spikes shown here does not correspond to file uploads.

       

      My question....

       

      How can I identify what is causing these excessive wait times and how can I fix the issue.

       

       

      Thanks

       

      Paul Jansen

        • 1. Re: Extreme Server wait times on Windows Server
          Benjamin Fehr

          weak network performance and/with sporadic drop outs is first possible causes that catches my mind.

          Do you have any tools to check latencies (Ping) between Clients AND Server?

          I got told that any Ping > 50milliseconds causes problems anyway.

           

          Another point to check is if the Wait time occurs with specific FileMaker tasks, Script task, a specific layout, to figure out if something in your solution is slowing down performance.

          • 2. Re: Extreme Server wait times on Windows Server
            Paul Jansen

            We have not been able to determine any pattern as to which layouts are in use and which scripts are running, as it seems to happen randomly.

             

            Ping times on the local LAN are <5ms and from remote WAN 10-15ms

             

            This morning for example we had 4 spikes in wait time between 09:56 and 10:06 and nothing so far in the following hour.

            The 09:56 spike showed a 19 second wait time.   During the period 09:54 to 09:58 there were no image uploads shown in the WPE log, so maybe that's a red herring...

             

            Any other suggestions much appreciated?

            • 3. Re: Extreme Server wait times on Windows Server
              Fred(CH)

              Any other suggestions much appreciated?

              A dummy one : backups or schedules handled by FMS ?

              • 4. Re: Extreme Server wait times on Windows Server
                Paul Jansen

                Fred,

                 

                Thanks for your input.

                 

                We turned off all backups except the end of day and it makes no difference.

                • 5. Re: Extreme Server wait times on Windows Server
                  Benjamin Fehr

                  just some 5 cent thoughts:

                  - any other applications running on the server?

                  - to put focus on the network itself:

                       • anything in the environment stressing the network? Anybody performing some heavy downloads?

                       • any technical issues with the internet router OR the Ethernet Switch *)

                   

                  *) I know somebody had a big issue similar to this caused by a cheap Ethernet Switch. Would you have a chance to replace it temporarily for tests?

                  • 6. Re: Extreme Server wait times on Windows Server
                    Paul Jansen

                    So here is an update...

                     

                    We were able to mitigate the issue somewhat by reducing the virtual cores to one - overall performance slowed, but the excessive wait times reduced considerably in length.

                     

                    This week the server has been upgraded from the original vmware virtual machine to a physical server

                     

                    RAM: 64GB

                    CPU Xeon E5-2650 v3 10 core (2 processors)

                    SAS 15K RAID

                     

                    Everything is very speedy on the new server almost all the time. Unfortunately, we still get occasional spikes in wait time where no-one can do anything for up to 10 seconds (that's the max shown on the stats graph, but the experienced wait time, when FMP is frozen, seems to be a multiple of this).

                     

                     

                    Now that we have an excess of hardware capacity to run FMS and have eliminated the virtual environment.  Does anyone have any thoughts on how we might track down the cause of the slowdown.

                     

                    Thanks

                     

                    PJ

                    • 7. Re: Extreme Server wait times on Windows Server
                      CamelCase_data

                      I realize this may not be easy to test realistically, but it would have been really interesting to see if you still get the same issue with FileMaker Server 15 rather than the now quite old FileMaker Server 13. Any chance you can set up a test copy on a VM?

                       

                      And if you would still get the issue, you could hopefully get some more information via the Top Call Statistics.

                      • 8. Re: Extreme Server wait times on Windows Server
                        Paul Jansen

                        Sorry, I forgot to mention, that the shiny new server is running FMS 15.  This means the issue has been present on FMS 13, 14 and 15.

                         

                        The top call stats does show some things that look strange, but I'm not sure how to interpret them....

                         

                        At the time of the most recent occurrence of the problem, it looked like all users were recorded as having high wait times for a find on the same table at exactly the same moment performing exactly the same task

                         

                        Interestingly, I have also been experiencing the issue raised here Find Loop? of a find progress dialog appearing apparently randomly even when FileMaker is not actively in use but just sitting idle.

                         

                        I wonder if these things may be connected or have a similar root cause.

                         

                        All suggestions will be gratefully received.

                         

                        Thanks

                        • 9. Re: Extreme Server wait times on Windows Server
                          CamelCase_data

                          I'd start by checking on what table this is happening.

                           

                          ExecuteSQL (

                           

                            " SELECT TableName FROM FileMaker_Tables WHERE TableId = ? "

                            ; "" ; ""

                            ; 129

                           

                          )

                           

                          You may also want to check resources liks http://www.soliantconsulting.com/soliant-tv/2016/05/filemaker-15-server-top-call-stats-logging.

                          • 10. Re: Extreme Server wait times on Windows Server
                            TSGal

                            Paul Jansen:

                             

                            When a client performs a find on a hosted file, the find actually occurs on the hosted file.  If a client performs a search on an unindexed field or a wildcard search, depending on the number of records, this could take some time.

                             

                            Could this action be a possibility?

                             

                            TSGal

                            FileMaker, Inc.

                            1 of 1 people found this helpful
                            • 11. Re: Extreme Server wait times on Windows Server
                              Paul Jansen

                              I think this is unlikely.  I have discovered that the freezing for all 20+ clients at the same time seems to coincide with significantly increased disk write activity.

                               

                              Could this be the cause of the client freezing of up to 45 seconds?

                              • 12. Re: Extreme Server wait times on Windows Server
                                Paul Jansen

                                TSGal wrote:

                                 

                                Paul Jansen:

                                 

                                When a client performs a find on a hosted file, the find actually occurs on the hosted file. If a client performs a search on an unindexed field or a wildcard search, depending on the number of records, this could take some time.

                                 

                                Could this action be a possibility?

                                 

                                TSGal

                                FileMaker, Inc.

                                Could this really cause a 20 core server to freeze all the clients for almost a minute?  That would be extremely disappointing.

                                • 13. Re: Extreme Server wait times on Windows Server
                                  TSGal

                                  Paul Jansen:

                                   

                                  From the limited information provided, the previous screen shot of the Top Call Stats showing a Find in progress, the last information of mitigating "the issue somewhat by reducing the virtual cores to one", I was providing one possible scenario.

                                   

                                  With the new information of a "20 core server" (and assuming all 20 cores are being used), this would probably not be relevant.

                                   

                                  What actions are causing the disk writes?  Are all the remote calls performing a Find, or is there any external tables (SQL) involved?

                                   

                                  TSGal

                                  FileMaker, Inc.

                                  • 14. Re: Extreme Server wait times on Windows Server
                                    Paul Jansen

                                    Thanks for your input. 

                                     

                                    I have been trying to get to the bottom of this issue for so long, that I sometimes lose patience!

                                     

                                    To expand on the single entry from the top calls log.  An almost identical entry appears for almost all 24 fmp clients.  My view is that an explicit use initiated search by all those people at the same time is very unlikely.

                                     

                                    There are no non FileMaker external data sources.

                                     

                                    I have now reduced the FMS cache from 5000 to 1000 to see if this makes any difference, but we will not know until about 08:30 BST when the bulk of the users start work.

                                    1 2 Previous Next