13 Replies Latest reply on Nov 8, 2016 9:03 AM by justinc

    Error 812 - Exceeded capacity - for PSOS calls

    justinc

      We ran into a problem yesterday with our server.  Our PSOS script calls were failing with error 812 (something like "Server has exceeded capacity - try again later").

       

      We are running FM Server 13.09.  This is on Windows on an AWS virtual machine...I don't know the specifics, though (I don't have access to the desktop, just the admin console).  However, I believe it to be pretty beefy - we host 50+ files for our clients on this server.  The Server is configured for 255 simultaneous script sessions.

       

      There was no visible evidence on the server admin console itself that there was anything wrong:  there were only 19 users logged in, non of which appeared to be 'script session' users in the Statistics or the Client list for each database.  The Statistics tab only showed 20-30 remote calls per second at the peak (I didn't write it down).  And remote calls in progress was 0-5.

       

      Restarting the Server Script Engine appears to have fixed the problem for us:  "fmsadmin restart fmse"

       

      But now we are trying to figure out why it happened in the first place.  With no strange 'script' users hanging around it wouldn't seem that those were causing the problem.  It would seem that there is something not releasing correctly in the Script Engine when a PSOS call completes.  (We have some files on this server that make extensive use of PSOS.)  So how can we track down what script was causing the problem?

       

      Thanks,

      Justin

        • 1. Re: Error 812 - Exceeded capacity - for PSOS calls
          Johan Hedman

          On your FMS in the Admin Console you can set a maximum of allowed PSoS. That is I believe for standard 25. You can change that if you wish

          • 2. Re: Error 812 - Exceeded capacity - for PSOS calls
            Markus Schneider

            since You set up the server to allow 255 script sessions, I got the gut feeling that FMS just needs a restart from time to time... we have had this with earlier versions of FMS (I believe 13.04 was much better than earlier 13) - and we also have had this with FMS14.

             

            -> restarting helps (on FMS14, I don't have access via CLI, so I asked the IT to restart the machine (VM))

             

            Problem is, that one has to make sure that a PSoS is running/finishing...

            • 3. Re: Error 812 - Exceeded capacity - for PSOS calls
              wimdecorte

              justinc wrote:

              However, I believe it to be pretty beefy - we host 50+ files for our clients on this server. The Server is configured for 255 simultaneous script sessions.

               

              The fact that you don't know how beefy is a little worrying.  You obviously rely big-time on PSoS sessions is a game-changer and you can't treat the server as a plain old database server; it has become an application server.  And needs to be configured AND monitored as one.

               

              If you have the FMS stats.log turned on: any clues of things building up there?

              If you do not have it turned on: do so right away (it is off by default and it is the single most important troubleshooting source, together with the event.log).

               

              You would certainly benefit from upgrading to 15 since it is more robust in all server-side processing AND you'll get the benefit of the top-call logging for additional info.  I would seriously consider this especially since it sounds like this one server hosts files for multiple clients - so it is a single-point of failure with a big impact.

              • 4. Re: Error 812 - Exceeded capacity - for PSOS calls
                justinc

                It's a case where I just don't know the specs - I don't have access to the full desktop.  There are others in charge of that, that do know - and they are server savvy.  I have found a roundabout way that it appears to have 14GB of memory (based on the 'Server Cache' setting, where it says the max recommended was ~7GB, half the system memory.)

                 

                Stats.log turned on?  You mean checking the box on the 'Clients' tab of the Statistics page?  (In 15 I believe there is the checkbox on the Logging tab of the DB configuration...and it appears that that box no longer exists on the Statistics page.)

                 

                We do have a server that we are testing 15 on...we haven't switched over this primary one yet, though.

                • 5. Re: Error 812 - Exceeded capacity - for PSOS calls
                  taylorsharpe

                  I used to have a lot of problems with 13 script engine dying and having to manually restart it.  But that was a long time ago and those problems went away with FM 14.  I heard one person here wrote a script timer on FMP on the server to test if PSoS worked and if it didn't, it ran a batch file to restart FMSE.  But upgrading is much easier than doing all that. 

                   

                  FYI, 13 is getting behind the times in security as well as performance.  Highly recommend you upgrade.  I bet your FMSE problems go away like mine did if you upgrade. 

                   

                  You might want to start testing scripts to see start and stop UTC timestamps to see what is taking long and clogging things up.  Also, write PSoS script so that if they error our, they then run locally. 

                  • 6. Re: Error 812 - Exceeded capacity - for PSOS calls
                    wimdecorte

                    justinc wrote:

                     

                    Stats.log turned on? You mean checking the box on the 'Clients' tab of the Statistics page? (In 15 I believe there is the checkbox on the Logging tab of the DB configuration...and it appears that that box no longer exists on the Statistics page.)

                     

                     

                     

                    No, it the toggle on Database Server > Logging, the one that says "usage statistics"

                    • 7. Re: Error 812 - Exceeded capacity - for PSOS calls
                      wimdecorte

                      justinc wrote:

                       

                      It's a case where I just don't know the specs - I don't have access to the full desktop. There are others in charge of that, that do know - and they are server savvy.

                       

                      Allow me to be a bit pedantic here; not aiming to make you feel bad; I just want to drive an important point home about the implications of using PSoS in general.

                       

                      How can you make architectural decisions for your solution (such as how often to use PSoS and for what) without knowing the exact processing / disk / memory / network i/o that you can work with?  That's what I mean when I say that FMS now becomes an application server.  By shifting the workload from the rich desktop client to the server; the server now becomes an integral part of the efficiency of your solution; much more so than if FMS just hosts the file.  The 'point of failure' shifts a bit here and these decisions can not be made in a vacuum.

                      • 8. Re: Error 812 - Exceeded capacity - for PSOS calls
                        justinc

                        Don't worry - no offense taken.  As a developer on this system I agree - I should probably know better what the server configuration is so that I can judge more accurately how much stress it could handle.  But on the flip side of that, there doesn't appear to be a metric or way to measure the impact of a given PSOS call on the server - and thus if it can spare the resources.  it seems to be more of a trial and error situation - get it running and see what impact it has.

                         

                        Back to the particular situation, I did glance at the Server Statistics during this issue, and didn't notice anything that stood out.  The Client list (on the Statistics page) was all human-looking names; on the Server tab I looked at the Remote Calls/s and Remote Calls in Progress and there were not high numbers of either of those:  < 30 for RC/s and ~0 for RCIP.

                         

                        What would you recommend looking at?

                        • 9. Re: Error 812 - Exceeded capacity - for PSOS calls
                          richardcarlton

                          +1 for WIm.  I totally agree with the comments.

                           

                          + 1 For Taylor.  Upgrade... please.  Otherwise plan on randomly just restarting the box when it mis behaves.  FMI won't be providing much support... and certainly not chasing bugs on this...since they were probably already fixed.

                           

                          - RC

                          • 10. Re: Error 812 - Exceeded capacity - for PSOS calls
                            FabriceNordmann

                            We had this problem with almost every 13 and 14 servers we installed. Much less with 15.

                            • 11. Re: Error 812 - Exceeded capacity - for PSOS calls
                              wimdecorte

                              justinc wrote:

                               

                              I did glance at the Server Statistics during this issue, and didn't notice anything that stood out. The Client list (on the Statistics page) was all human-looking names; on the Server tab I looked at the Remote Calls/s and Remote Calls in Progress and there were not high numbers of either of those: < 30 for RC/s and ~0 for RCIP.

                               

                              What would you recommend looking at?

                               

                              The FMS stats.log.  It's off by default unfortunately so if you had not toggled it on before then do so now.  It's on the 'logging' tab of the 'database server' config area.  It's named 'usage statistics'.

                              It will show the same counters as the statistics pane but with a lot more granularity as it collects more measuring points and can give you a much clearer view than the 'current' / 'peak' / 'average'

                               

                              In 15 you can actually measure the impact of an individual PSoS call by using the new 'top call logging' but since you don't have 15 then you have to make do with the stats.log

                              That stats.log is absolutely critical.  It will give you performance data over time so that you can spot trends in any of the 4 potential bottlenecks (processing resources, disk i/o,  network i/o and memory).  It can give you a good baseline to compare against when things start going south.

                               

                              Similarly the 'access.log' can you give you some extra info on what is consuming those PSoS sessions.  It will show what opens what file when in more detail than the 'clients' tab can.

                              • 12. Re: Error 812 - Exceeded capacity - for PSOS calls
                                justinc

                                Fabrice,

                                    Good to hear that your upgrade to 15 helped out the number of problems, but you say 'much less with 15'...implying that you still run into the problem.  Can you determine if you run into that problem because you are hitting the simultaneous limit?  Or does it randomly give that error even though it doesn't look like you have excessive concurrent script connections?

                                • 13. Re: Error 812 - Exceeded capacity - for PSOS calls
                                  justinc

                                  WIm,

                                      The usage statistics is toggled on now...and I believe that it was on when this problem occurred as well, so I will get a hold of that log file and see what I can find.