10 Replies Latest reply on Aug 4, 2017 9:55 AM by monkeybreadsoftware

    FM Cloud - server side scripting engine

    amsc

      I've had a couple of instances where the scripting engine in FM Cloud doesn't start up correctly.  Its only happened twice, and they were not close together time wise (today and again several months ago), so this isn't a "burning" issue at all.  I'm more curious if there's a way to restart FM Cloud without having to restart the AWS instance. 

       

      Here are my observations:

      When I tested this morning, I got error# 812 (host capacity issues) for any PSOS call.   It happened when the first users started to log in for the day.  At that point, there's only 1 or 2 people logged in and only one of those people was attempting to run a PSOS script, so it can't be licensing or PSOS capacity issues.  When I look at the logs there are no obvious issues and in the console there are no "runaway" PSOS scripts logged in.  The instance goes through the recommended nightly maintenance schedule where, among other things, the system is restarted/rebooted; it appears that the engine just didn't come up correctly.   

       

      When I restarted the AWS instance (i.e. reboot the server) the issue was resolved. 

       

      Is there another way to restart the FM scripting engine or FM Cloud itself without having to reboot the instance? 

       

      Thanks!

      Alex

        • 1. Re: FM Cloud - server side scripting engine
          mikebeargie

          Alex,

           

          There's not really a filemaker recommended way to do it. I'd imagine to do so you would have to log in via putty/SSH and do command line actions on linux. However fmsadmin commands do not work like they do on windows or mac in FM Cloud.

           

          Have you tried to temporarily turn off all auto-executed PSOS scripts in order to eliminate that as the problem causer?

           

          Your instance is up to date with the latest update, right?

           

          only one of those people was attempting to run a PSOS script, so it can't be licensing or PSOS capacity issues.

          Well, capacity would be determined by WHAT that script is trying to do. any insight as to what that script does?

          • 2. Re: FM Cloud - server side scripting engine
            amsc

            Mike, thank you for your reply.

             

            On the FM restart, that was what I had guessed.  It would be nice if there were a "restart" option for FM without having to restart the whole instance.  Oh well, as I said its only happened twice in 3 months.  Everything is up to date (just upgraded to 1.15.2.32). 

             

            On the auto-executed PSOS, that was a wake-up call for me to put in the necessary safeguards which I'm actively working on now.  Fortunately, I don't overly rely on PSOS steps.  I have them in place for large record processing steps, data import steps, REST calls, ...

             

            This script in particular was a REST API call to another service.  I also tested PSOS with a generic "record write script" which failed as well.  There were no active processes going on during this time and when I was testing it, I was the only user logged in (the advantages of early in the day and a small user base!).  Rebooting resolved the issues. 

             

            If this becomes more widespread, I'll move FM Cloud to a new instance. 

             

            Alex

            • 3. Re: FM Cloud - server side scripting engine
              amsc

              FM Cloud 1.15.2.32

               

              This issue seems to have happened again; the server script engine didn't startup properly after the midnight maintenance server reboot.  When I was looking at the event log before, I was looking for errors.  This time, I decided to look at all of the line items for the startup of the midnight maintenance reboot compared to the reboot I just did.  They are slightly different.  One obvious differences between the two is when the "Starting FileMaker Script Engine process..." step happens.  In the bad midnight reboot, the script engine starts before the database server step.  Also there are 3 script-related steps missing in the  bad reboot (all plugin / script related steps). 

               

              See attached for a screenshot of the logs.  The left side is the bad reboot, the right side is the successful reboot.  I've highlighted the script engine startup step in yellow and the missing steps in the orange. 

                 

              I'm going to reach out to FM support, but wanted to see if anyone has any ideas as to what could be causing this?  I've not "messed" with the EC2 Linux instance at all, and I upgraded to 1.15.2.32 a couple of weeks ago.

               

              I may move the instance this weekend to see if that resolves the issue.  The only way I know to do that is to downgrade to a smaller instance (or any different instance size), wait a while, then upgrade back to the larger instance.  Is there another way to do it?

               

              Alex

               

              Screen Shot 2017-08-04 at 8.13.17 AM.png

              • 4. Re: FM Cloud - server side scripting engine
                mikebeargie

                Use of a plugin is a new variable you're presenting.

                 

                Given how new the linux plugin engine is, I wouldn't be suprised if something is conflicting there. I assume your PSOS script that is freezing does use that plugin's functions?

                • 5. Re: FM Cloud - server side scripting engine
                  wimdecorte

                  You may want to reach out to monkeybreadsoftware and see if he can help shed a light.  The use of the plugin can be a factor here in why the FMSE dies or refuses to startup.  Do contact FMI support as well; FM Cloud has an internal mechanism (Virtual DBA) that is supposed to monitor the health of all the FMS sub processes and restart them if they fail.  Perhaps FMI support has a way to collect more info from that.

                  • 6. Re: FM Cloud - server side scripting engine
                    amsc

                    All PSOS script steps fail to run regardless of whether they use MBS script steps or not.  In the startup steps in the event log for the bad restart, it never starts the "use of server side plugins" step. 

                    • 7. Re: FM Cloud - server side scripting engine
                      mikebeargie

                      But the FMSE could be reacting to plugins regardless of if they are being called or not. One of the standard troubleshooting methods for users having inexplicable or odd issues is disabling plugins to see if the symptoms goes away.

                       

                      Don’t discount it just because you assume it’s not being used. Like I noted, the linux plugin engine has only been available for a short time, so there is a possibility it’s causing errors. I agree with Wim to contact Christian at MBS and FM tech support to see if they can offer any advice.

                      • 8. Re: FM Cloud - server side scripting engine
                        amsc

                        I've reached out to FMI support and will shoot off an email to monkey bread as well.  I'll let everyone know what I find out.  I don't think its the MBS plugin itself because the startup step to "use server side plugins" never happens.  Also, all PSOS scripts fail to run regardless of MBS content or not.  In a worse case, I could shift the MBS steps to the client; just a lot of data processing that I don't want to do over a WAN connection...  

                         

                        Alex

                        • 9. Re: FM Cloud - server side scripting engine
                          amsc

                          Thank you Mike.  I was posting my next comment as you replied.

                           

                          Will definitely reach out to both.

                          • 10. Re: FM Cloud - server side scripting engine
                            monkeybreadsoftware

                            well, you can check log and see if you see the message from MBS Plugin.

                            Something like MBS Plugin loaded in version x.y.

                             

                            I can provide a debug plugin version which writers much more log entries.

                            Or you use MBS("Trace") command with normal plugin to write log with all calls to MBS functions.

                             

                            I'd expect that a bug in one of our functions can crash FMSE process. But that normally requires to call a MBS function first.