7 Replies Latest reply on Oct 8, 2016 10:27 AM by philmodjunk

    Host's capacity exceeded by duplicate users

    AdamSnyder_1

      Twice in the past 2 days our server has needed a reboot due to people getting the error "The host's capacity was exceeded, try again later."  Our server is set to 250 connections max, and we have no where near that many happening a once.

       

      When I look at the admin console, I see that each user is duplicated several times.  Activity monitor is showing fmserverd is using up 100% of it's thread.  I disconnected all clients and closed all databases before rebooting, but on restart the log still showed that the files weren't closed properly, and needed consistency checks.

       

      Unfortunately, our ISP is wireless.  Our internet comes from a big antenna on the roof of our warehouse.  We're getting a hard line in a couple months, but they had to trench and lay conduit, etc.  We've been hosting the server via the wifi connection for over a year so far and have had this problem intermittently. 

       

      Is our wireless ISP to blame? 

       

      Is there something I can configure in FM server to prevent this?

       

      Screen Shot 2016-10-03 at 7.14.03 AM.png

        • 1. Re: Host's capacity exceeded by duplicate users
          wimdecorte

          AdamSnyder_1 wrote:

          I disconnected all clients and closed all databases before rebooting, but on restart the log still showed that the files weren't closed properly, and needed consistency checks.

           

          This is obviously bad - do you go back to a backup when this happens?  If not then I would strongly suggest you set up an aggressive backup strategy that will allow you to do so.  It could be that repeated events like create some lurking damage in the file.

           

          AdamSnyder_1 wrote:

          Activity monitor is showing fmserverd is using up 100% of it's thread.

           

          Not sure what this means.  Do you have the FMS stats log turned on under the Logging options?  If not turn it on so that it collect some good data.

           

          Are there any DMP files in the FMS logs folder that correspond to the date & times that you saw this happen (not the exact times, could be a few hours or a day or so older).

           

          Do you have both FMS and the user's priv set configured to disconnect users after a certain amount of idle time?

          • 2. Re: Host's capacity exceeded by duplicate users
            AdamSnyder_1

            Yeah, I'm well aware of how bad it is to continue with a file that might be damaged.  We back up every 30 min.  We have a brand new Mac Pro server with a 1TB ssd, so it's pretty fast to backup.

             

            I didn't have any of the extra logging options turned on, just enabled all of them to help troubleshoot.

             

            There are no DMP files in the FM Server logs folder.

             

            Privilege sets are set to "disconnect users when idle".

             

            I enabled "Set max idle time allowed for clients" and set to 360 minutes.

            • 3. Re: Host's capacity exceeded by duplicate users
              wimdecorte

              AdamSnyder_1 wrote:

               

              I enabled "Set max idle time allowed for clients" and set to 360 minutes.

               

              I would probably set it to 60-120 minutes.  Waiting 6h doesn't really give FMS much a chance to clean house and free resources.

               

              Another thing you could consider (but would take some work) is to write an OS-level script that takes the output from the 'fmsadmin list clients -s' ouput, looks for duplicates and uses the 'fmsadmin disconnect client <id>' against the oldest of the sessions for a given client name.

               

              That may work better than relying on the idle time.

              • 4. Re: Host's capacity exceeded by duplicate users
                philmodjunk

                What version of server are you using?

                 

                We see our FMS 13 server get a bad case of the "slows" from time to time. When we do, we often see the same user slisted multiple times. We've always interpreted the number of multiple instances of the same user as a symptom of the larger problem and yes we had to restart.

                 

                Been looking forward to see if an upgrade to 15 with some "top 25 logging" might help us identify if there are user processes overloading the system--which has been our hypothesis...

                • 5. Re: Host's capacity exceeded by duplicate users
                  AdamSnyder_1

                  We are running FMS 15.  I enabled logging and will update my post if I figure anything out.

                  • 6. Re: Host's capacity exceeded by duplicate users
                    AdamSnyder_1

                    I'm starting to think this is a networking issue.  Maybe TCP connections are being left in a "half open" or "half closed" state? 

                     

                    I have a feeling our crappy wireless ISP is basically DoS attacking our server.

                     

                    "TCP has a vulnerability in that the final FIN packet sent to a client can be potentially dropped by routers/networks resulting in a connection that is half-open when the actual intention was to fully close the connection. This and similar approaches have been popular types of Denial of Service attacks as they do not require a lot of bandwidth, yet potentially eat-up valuable handles, sockets, and threads depending on the server implementation, but they can also happen in the real world with increasing frequency thanks to our shoddy wireless carriers."

                     

                    networking - what is TCP Half Open Connection and TCP half closed connection - Super User

                    • 7. Re: Host's capacity exceeded by duplicate users
                      philmodjunk

                      However, server is supposed to drop clients that are no longer responding. It regularly sends out a signal meaning "are you alive?" and the client responds back "yes". If server doesn't get a response in a specified period of time, it is supposed to disconnect that user and log the event with a "No longer responding" entry.