1 2 Previous Next 18 Replies Latest reply on Apr 11, 2014 5:05 AM by dchretien

    Server-side Scripts Time Limit Hangs Server

    dchretien

      Hi,

       

      I have a system that's been running for years. Most recently, it was running on FileMaker Server 11. This sytem runs a scheduled script every half hour during the work day, at staggered times. One set of scripts imports data from csv files generated by the client's AS-400, every 30 minutes. Then, staggered from those times, another scheduled script exports changes made in FileMaker to csv files that are imported into the AS-400 system, also every 30 minutes.

       

      This has been running continuously since we installed the last update to FileMaker Server 11—in other words, quite awhile.

       

      Each of these scripts was set with a 10-minute time limit, and other for very rare occasions, that limit was never reached. When it was, the script was aborted, and would run at the next scheduled time.

       

      Recently we upgraded the server to FileMaker Server 13. The server is running Windows Server 2008 Enterprise, and we had to install Service Pack 1 in order to install FileMaker Server 13. It has 18GB of RAM.

       

      The scripts ran fine for approximately a week, when I received a call from the client saying that the server was unresponsive. When I looked at the Admin Console, it showed that the data import scripts (coming from the AS-400) had timed out, and so had other scheduled scripts.

       

      I saw nothing else odd, but there was a Java update notice on the server. I restarted the server, and everything seemed to work fine. Two weeks later, I received another call. The server was unresponsive, and the same results were showing in the Admin Console. There was another Java update notice. I now suspected that the Java update was somehow interfering with scripts running on the server. I restarted the server, and everything seemed fine again.

       

      Within a week, the server was unresponsive again. This time, there was no Java update notice on the server, but I did note that all of the server script operations that had been scheduled since the timeout were still "connected" to the server. In other words, when the server runs a script schedule named "Import Data," a user named "Import Data" displays in the Admin Console as connected to the server for as long as the script is running. What seems to be happening is that if the "Import Data" schedule times out, every subsequent script schedule (the next "Export Data" schedule, and all of the follwing import and export schedules) attempt to run, but they all time out. In addition, none of those "users" is ever disconnected from the server. After four hours, there will be 8 users connected (four import schedule users and four export schedule users). I'm not sure at what point the server becomes unresponsive, but we haven't hit the connection limit on the server. (That's irrelevent anyway, since this shouldn't be happening.)

       

      I restarted the server. Then, I received a call last Thursday. Same thing. No Java update notice. Restarted server. Call today (one work day in between this time.) Same thing; no Jave update notice; nothing else that I can find. Restarted server.

       

      This leads to two main questions:

       

      1. Why should scripts that ran, in most cases, within a maximum of 7 minutes, now take more than 10 to run?

       

      2. When a script reaches the time limit and is aborted, why is that user not disconnected from the server, and why do all subsequent scripts time-out?

       

      And, finally, has anyone else encountered this or anything related to it, and does anyone have any suggestions?

       

      Thanks.

       

      --Dennis Chretien

        • 1. Re: Server-side Scripts Time Limit Hangs Server
          DavidJondreau

          I've had a similar issue.

           

          My script, sending scheduled e-mails, runs every 20 minutes, sometimes exited in 5 seconds, someimtes, running 5-10 minutes. Every now and then (sometimes once a week, sometimes twice a day), the script would hang the entire server. All users on all files (even unrelated ones) would become unresponsive. Users could force quit and try to log in again, but wouldn't get access. The users would show up multple times in the Clients status tab. The schedule script would run again, and show multiple times.

           

          The entire databse server was stuck. From the admin console, I couldn't disconnect users, closing databases would never complete, etc etc. From the command line, I couldn't do any database functions either. I could run some basic fmsadmin commands, like Help, but nothing around the databse. In the end we would have to restart the machine. We tried re-installing FMS but it didn't help.

           

          In the end, I re-wrote the script from scratch and created a new schedule, and it's be fine ever since.

           

          Maybe it was a corrupt script (though a Recover on the database didn't show any errors), maybe the re-write got it to run faster than before and avoid a time out, I'm not sure.

          • 2. Re: Server-side Scripts Time Limit Hangs Server
            wimdecorte

            dchretien wrote:

             

             

             

            1.  Why should scripts that ran, in most cases, within a maximum of 7 minutes, now take more than 10 to run?

             

             

             

            Extra load on the server (PSoS, other schedules, users running big finds on unstored calcs,...)?  Backup running at the time (especially if it has "verify" on), different data set that takes longer to process?

             

            The FMS stats should paint some sort of picture here and should show you some kind of pattern.

             

            If you have a schedule that is time-critical and it absolutely has to finish inside 10 minutes, then don't rely on the FMS force abort but manage time yourself in your script.  Record the start the start time and as you loop, check the time and exit gracefully at say 8 minutes.

             

             

             

             

            2.  When a script reaches the time limit and is aborted, why is that user not disconnected from the server, and why do all subsequent scripts time-out?

             

             

            The few times that I have seen it was because there is field-level validation and the data was such that the validation failed.  Since there is no UI, there is no way to get out of it.  For that reason (and to be able to exit gracefully as mentioned earlier), scripts that are used server-side, both in schedules and through PSoS should have a lot more error trapping and handling than we typically see for scripts that run on the desktop.

            • 3. Re: Server-side Scripts Time Limit Hangs Server
              dchretien

              This is the exact same behavior as I'm experiencing.

               

              I will try re-writing the scripts to see if that solves the problem.

              • 4. Re: Server-side Scripts Time Limit Hangs Server
                dchretien

                The load on the server hasn't really changed, and I've tested the times with no users connected, and simply running the scripts from the Admin Console.

                 

                There is no looping.  There is a series of scripts, each called in sequence by a main script.  Each script is simply go to a layout, import records from a specified file, to to a log layout, and record the last error reported.  This last step is to let me know where things failed, if something goes wrong, but there is never an entry for the last script that it tries to run, because the time lmit is reached, and everything hangs from that point, on.

                 

                I could check the time before calling each sub-script, and abort if we're nearing the limit, but since the process is the same as it had been, and the data is virtually the same (file sizes vary very little), I don't see any reason for the time to be taking significantly longer than before.  Most imports are now taking almost twice as long as they did before moving from version 11 to version 13. I would have expected comparable or even improved performance, but not degradation this severe.

                 

                As soon as one scheduled script times out, everything is hung from that point, on, as described by wimdecorte, above.

                • 5. Re: Server-side Scripts Time Limit Hangs Server
                  marcodezordo

                  Just my two cents: our FMS 13 hangs (like version 11 did - I'm just now testing v. 12) in exactly the same way than yours, even if we do not use any server-side script. In our case it seems just a matter of stressfull load of work (more than one hundred of very active FMPro clients).

                  • 6. Re: Server-side Scripts Time Limit Hangs Server
                    dchretien

                    There are 35 import scripts in the series.  On FileMaker Server 11, the entire sequence completed, generally, within a maximum of seven minutes.

                     

                    I've managed to avoid the timeout problem by removing the last nine imports from the sequence on the thirty-minute schedule.  These last nine imports do work with the largest file from the AS-400 (just about 9MB), but that was never a problem.

                     

                    Now, these last nine imports, by themselves, take 25 minutes.

                     

                    38 imports in 7 minutes on FileMaker Server 11, compared to nine of those steps taking 25 mintues on FileMaker Server 13.

                     

                    The tests that produced these times were run with no users connected to the system.  Progressive backups are not enabled, so that's not interfering.  Everything else is configured as it was with FileMaker Server 11.  (The schedules were exported and imported into the new installation.)

                     

                    To the best of my knowledge, the only change was moving to FileMaker Server 13, and having to install SP1 in order to do so.  There were no other changes to the server machine.

                    • 7. Re: Server-side Scripts Time Limit Hangs Server
                      DavidJondreau

                      I should add that these problems began after upgrading from FMS 12 to FMS 13. The script would hang, and not time out, which was set for 10 minutes.

                       

                      I don't think the issue is the script taking too much time.

                      • 8. Re: Server-side Scripts Time Limit Hangs Server
                        dchretien

                        In this case, the server was reporting that the scripts had exceeded their time limit, but every script was reporiting that once the server was "hung."  I'm not sure whether the first "time-out" triggered the process, or if something else triggered the time-outs.

                        • 9. Re: Server-side Scripts Time Limit Hangs Server
                          wimdecorte

                          The slower import speed started happening in 12 (compared to 11), I think 13 is a little faster than 12 but not at the same speed as 11 was.

                          • 10. Re: Server-side Scripts Time Limit Hangs Server
                            Vincent_L

                            Waow, terrible experience, my strong simpathies.

                             

                            I'm contemplating moving from 11 to 13 (after skipping 12 for perf reasons) and this doesn't bode well. Are you all on PC, any problems on Mac FMS ?

                            • 11. Re: Server-side Scripts Time Limit Hangs Server
                              dchretien

                              I'm running FileMaker Server 13 on Macs, as well, but none of these installations have server-side scripted imports, as this one does.  I've not had problems with FileMaker Server 13 on Mac OS X or Windows, other than this issue.

                              • 12. Re: Server-side Scripts Time Limit Hangs Server
                                dchretien

                                The problem is worsening.  I'd removed the "larger" imports to a separate script, set to run once a day, early in the morning, before business.  These imports took less than five minutes in version 11, but now, they take nearly thirty minutes, and they did exceed the thirty-minute limit once, already.

                                 

                                Even worse, the remaining imports, which took less than two minutes in version 11 are now exceeding a ten-minute time limit. 

                                 

                                The largest file being imported is approximately 500K.  All of the imports are matching record imports, some of which are into rather large files (50K to 150K records), but, as notied, these imports used to take less than two minutes.

                                • 13. Re: Server-side Scripts Time Limit Hangs Server
                                  dchretien

                                  Here's some data that will demonstrate the problem:

                                   

                                  I have a script that logs completion of each import process, along with any error codes that may have been generated.  The times for each run vary slightly, but usually within a few seconds of all other runs of that script.

                                   

                                  Many of the scripts are running in comparable time spans, but that's a bit difficult to verify, because some of them complete in under one second, so the difference between ten milliseconds and forty milliseconds is negligible, but it does represent a process that's taking four times as long as it did, previously.  A script that shows zero seconds for both version 11 and version 13 does not necessarily indicate that both are taking the same amount of time.  A few scripts show zero seconds in version 11 and one second in version 13.  In real terms, this is negligible, and these variations are well within the normal variation of each run, but some of these zero second to one second increases are consistent.  (All version 11 runs of some of these scripts ran in zero seconds, but all of the version 13 runs take one second.  I’m sure there’s variation at millisecond accuracy, but version 13 does take longer.)

                                   

                                  Focusing on some of the longer processes, though, there is a more obvious increase.

                                   

                                  One script that, in version 11, typically took about 1:45 (1:46, 1:41, 1:43, 1:45) takes more than 21 minutes in version 13.  Strangely, another that took approximately 1:20 in version 11, takes two minutes in version 13.  Both scripts read from the same file, but each match a different subset of records.  (The second is matching against just over 700 records, the first against nearly 21,000 records.)  Nevertheless, what used to take under two minutes now takes more than twenty.

                                   

                                  This seems to be related to running the script as a server-side script, because I did test one of the imports (not the largest one) from a client, on a remote connection through a VPN connection, which traditionally is rather slow.  The script ran in less time than it takes to run from the server, but it did take longer than it used to take to run on the server in version 11.

                                   

                                  Making the matter worse is that because the scripts take so long to run, they’re exceeding their allocated time on the server (I’m trying to limit the number of simultaneous script sessions, and I don’t want one import colliding with the next.)  When the time limit is reached, that session is not terminated, and the server becomes unresponsive.

                                   

                                  I’m hoping that someone from FileMaker will verify if this is a known issue, or look into it to see if there is a problem in general. 

                                  • 14. Re: Server-side Scripts Time Limit Hangs Server
                                    Mike_Mitchell

                                    Are any of the fields in your target table (the one into which you're importing) indexed? If so, this can dramatically affect import speed, and will get worse over time. Suggest checking this and turning off the indexes on any fields that don't absolutely require it.

                                    1 2 Previous Next