8 Replies Latest reply on Mar 20, 2014 6:30 PM by wimdecorte

    FMS 13 occasional sudden slowness and logs question

    user10625

      FMS 13, on Windows Server 2008 R2.

      (Clients are 95% on Windows 7, with a couple of developers on macs).

      30+ users. Everything is V13 now (except for a couple of utility machines with FMP12 on them, they are still using XP OS.)

       

      It's a pretty big/complex solution, with 100+ tables, some with record counts in the 100K+ range (1 table with 3M records).

      We also have ESS . . .our server is talking to another machine on our network with a MySQL database.

      We upgraded to FMS13, from FMS12, a few weeks ago, and it's been largely great.

       

      However, recently (this past week), we've had a couple of instances where every one in the office got the ole' coffee cup/ spinning wheel at the same time, for an extended period . . . more than a few minutes, with no sign of letting up. And so essentially, the system crashed, we reverted from backup, and things were OK after that.

       

      This has not happened more than once per day.

       

      First thing was to make sure there was no active virus scanning. Check.

      Also checked with the users as to whether anyone was doing a particularly nasty find or replace operation at the time. Check. (no).

       

      In poking around on the web, I saw this ( http://help.filemaker.com/app/answers/detail/a_id/12541/kw/filemaker%20very%20slow%20in%20windows ):

       

      • 1.3. If you install FileMaker Server 13 on a system where FileMaker Server had previously been installed, remove or rename all of the FileMakerServer log files before installing FileMaker Server 13. Otherwise the server deployment may have errors when parsing the log files and may cause the server deployment to be slow.

       

      So I did not know this at the time of upgrading from FMS12. My question is, should I go ahead and re-install the server software completely (also doing the log removal), or, can I just turn off the server, delete (or move/rename) all of he log files, and re-start the server?

       

      One other idea . . . we have Progressive Backups running. 7 minute interval. Is this liable to cause occasional performance issues?

       

       

      Thanks,

      Christopher Bailey

      Baytaflow

      Boston MA

        • 1. Re: FMS 13 occasional sudden slowness and logs question
          wimdecorte

          The first thing to turn to are the FMS statistics.  By default they are off but hopefully you turned stats logging on when you installed FMS.  If you have not, turn them on now.

           

          The admin console statistics page may give you some clue in the meantime.  Over in the right-most column, under peak values: what is reported there?  Can you post a screenshot?

          1 of 1 people found this helpful
          • 2. Re: FMS 13 occasional sudden slowness and logs question
            user10625

            I just re-installed the server (got rid of old logs, just in case that was the issue, which I doubt) and have the stats running now, so we'll see if this happens again this week.

             

            Thanks,

            Chris Bailey

             

             

             

            >

            >

            >  Wim Decorte replied:

            >

            The first thing to turn to are the FMS statistics.  By default they are off but hopefully you turned stats logging on when you installed FMS.  If you have not, turn them on now.

             

            The admin console statistics page may give you some clue in the meantime.  Over in the right-most column, under peak values: what is reported there?  Can you post a screenshot?

            • 3. Re: FMS 13 occasional sudden slowness and logs question
              user10625

              Hello all,

               

              Resurrecting this old thread.

               

              This freeze/crash happened today again, twice.  Again, all users are suddenly frozen, at once.  (more details below).

              This time, I got the server stats graph going, and I'm posting the results, in case someone sees something interesting.

               

               

              Here is the crash from 9:30 AM:

              http://www.baytaflow.com/3_20_14_at_9_36_17AM.png

               

              and from 4:30 PM:

              http://www.baytaflow.com/3_20_14_at_4_33_12PM.png

               

               

              Note that in both cases you see spikes of "elapsed time per call" at the end (the green line.)  Along with mini-spikes in the opposite direction for Disk KB/per second.

               

              This system does have inefficient scripts, a lot of TOs, and various other issues, but those have been around for years and have not brought everyone (30 users) to a crashing halt.  I could imagine an errant script or SQL query or whatever crashing a single user (I've certainly seen that happen),  but I wouldn't think it would effect everyone else, except maybe for an instant or 2. 

               

              I've been suspicious that a connection to ESS might be hanging things  . . but that also has been in place for a while, with no issues before the last week or 2.

               

              Thanks for any input,

              Christopher Bailey

              Baytaflow

              Boston, MA

               

               

               

               

               

               

               

               

              >>>>>>>  A week ago, I wrote:

              >>>>>

              >>>>

              >>>

              >>>>

               

              FMS 13,   on Windows Server 2008 R2.

              (Clients are 95% on Windows 7, with  a couple of developers on macs).

              30+ users.   Everything is V13 now (except for a couple of utility machines with FMP12 on them, they are still using XP OS.)

               

              It's a pretty big/complex solution, with 100+ tables, some with record counts in the 100K+ range (1 table with 3M records).

              We  also have ESS  . . .our server is talking to another machine on our network with a MySQL database.

              We upgraded to FMS13, from FMS12, a few weeks ago, and it's been largely great.

               

              However, recently (this past week), we've had a couple of instances where every one in the office got the ole' coffee cup/ spinning wheel at the same time, for an extended period . . . more than a few minutes, with no sign of letting up.  And so essentially, the system crashed, we reverted from backup, and things were OK after that.

               

              This has not happened more than once per day.

               

              First thing was to make sure there was no active virus scanning. Check.

               

              Also checked with the users as to whether anyone was doing a particularly nasty find or replace operation at the time. Check. (no).

               

               

              Thanks,

              Christopher Bailey

              Baytaflow

              Boston MA

              • 4. Re: FMS 13 occasional sudden slowness and logs question
                wimdecorte

                The graphs don't tell me much, Christopher.  The raw numbers underneath that are more useful and the stats.log itself too as it will show that data over a prolonged time.

                • 5. Re: FMS 13 occasional sudden slowness and logs question
                  user10625

                  Hi,

                   

                  Thanks, Wim.  Here is the log file.  http://www.baytaflow.com/Stats.log

                   

                  I'm somewhat embarrassed to say that I had never bothered to poke around in the logs folder, so I wasn't really aware that one could poke around in this file.  Cool.

                   

                  Anyway, the relevant times are 9:35AM today (freeze #1), or thenabouts, and 4:30 PM today (freeze #2), or thenabouts.

                   

                  Thanks,

                  Chris

                  • 6. Re: FMS 13 occasional sudden slowness and logs question
                    wimdecorte

                    Looking at the log it is very clearly a processing power constraint that happens when te load reaches a certain treshold, seems to happen around 27/28 users.

                    Something is happening that uses up all the processing power for an extended period of time, leaving the server unresponsive.

                     

                    A couple of ways out of this:

                    - scrutinize the solution for anything that is "expensive" like finds on unstored calcs, especially in the larger tables, layouts with many portals,...

                    - if this is a virtual server, see if you can get more cores or more processors assigned to it, if it is physical box, see if you can get a more powerful one.

                     

                    To help narrow down the search, you can use the admin console and look at the "clients" tab to see what users generate thw worst stats and then ask them what layout they were on, what routines they were running, generally what they were doing at that time.

                     

                    You can also build a logging mechanism in your scripts so that it logs who runs what when, from what record / size of the found set / what layout,... It's a bit of work but it will help you track down where the pain points are.

                    1 of 1 people found this helpful
                    • 7. Re: FMS 13 occasional sudden slowness and logs question
                      user10625

                      Thanks for looking at it, Wim.

                       

                      One immediate question I have:  the first  couple of times this happened,  I checked the computer's monitoring software, and it looked like the CPU was, IIRC, (I'll check again next time),  below 7% usage or some such . . . way under-utilized--or so it looked like.  Is there somewhere where I can tell the machine to allow FMS to use more CPU "juice", or does that just happen automatically?

                       

                      Interestingly, on the same monitoring panel, I noticed that memory usage spiked from a normal 12G,  to around 17G.  Not sure if that means anything.

                       

                      From your answer, I surmise that there is no way to look at a stats.log-ish file for the individual clients? 

                       

                      Anyway, thanks again for the advise.

                       

                      Chris Bailey

                      • 8. Re: FMS 13 occasional sudden slowness and logs question
                        wimdecorte

                        user10625 wrote:

                         

                         

                        I checked the computer's monitoring software, and it looked like the CPU was, IIRC, (I'll check again next time),  below 7% usage or some such . . . way under-utilized

                         

                        That's not always a good measure.  You have to look at this together with FMS's reported stats.  FMS is multi-threaded but it can not take one thread and split it across different processors.  It take run one thread on one processor and another on another processors.  So if you have long-running threads, things will back up and requests will get queued up.  So more processors / cores will help out a lot in your scenario.

                         

                        There are some actions that seem to tie up FMS more than others and more than what the processor monitor at the OS level indicates.  That's why trying to find what is happening at the time of the hang-up is important.  There will be valuable clues to help streamline the solution.