13 Replies Latest reply on Jan 30, 2012 4:37 AM by codeus

    FM Server Elapsed Time/call in the millions and hanging

    AntonPopov

      Title

      FM Server Elapsed Time/call in the millions and hanging

      Your post

      Occasionally (sometimes 10 times a day, sometimes 1 per week) my FM stops responding.

      Happens every time Remote Calls In Progress is more than 1. (or maybe the remote calls in progress increase because of this problem)

      In the stats I see stuff like:

      2011-09-08 12:43:06.321 +0200   0       0       0       0       100     0       1       3       0       1       0       0       2       15001481        0       0

      and goes on like this until I restart it.

      Any ideas where I can start looking? I can't locate anything in the logs that is not showing up when there is no problem.

      I use FM server 11.0.3.309 on a Leopard Server 10.6.8

      CPU and memory usage are low at all times.

       

      Thank you for your time,

       

      Anton.

        • 1. Re: FM Server Elapsed Time/call in the millions and hanging
          TSDuck

          Hi Anton Popov:

          Thanks for posting.

          I have a few questions regarding your setup that will help with troubleshooting.

          What is the processor speed and ram on your server?

          When did this issue start occurring?

          How many databases are currently hosted from FileMaker Server?

          What is the maximum concurrent usage that your server reaches?

          Are you running any server side scripts or backup schedules?

          Does your FileMaker Server event log contain any error messages?

          If you’ve not checked your event log, you can find it located at MacintoshHD/Library/FileMaker Server/Logs.  Check the event log for any error codes, and see if any errors coincide with FileMaker Server stopping.  Once I have some further detail on your setup, I can suggest some troubleshooting you can try.

           

          TSDuck

          FileMaker, Inc.

          • 2. Re: FM Server Elapsed Time/call in the millions and hanging
            AntonPopov

            Hi, Thanks for your answer.

             

            First an update: sometimes remote calls in progres is more than 2 and server still runs. So the increase of remote calls in progres is a result and not a cause.

             

            I have 2x2.8 quad core xeons, 8G 800MHz DDR2 RAM

            FM is a tool that our HR have used for years (a handfull of people) but during the last months we have extended its usage to almost everyone in the company and that's when the problems started.

            I have 2 databases

            I seldom have more than 10 concurrent users.

            I have an hourly backup job but that does not seem to be related since the problem almost never occurs when the backup is running. no other scheduled jobs seem to be related.

            We have a bunch of server side scripts, I have contacted all the people that have used it during a lockup and asked them to re-do exactly what they were doing but I was not able to reproduce the problem. As of yet, I don't know how to trigger it on purpose.

            I almost never have any events before it locks up, for example:

            2011-09-08 15:05:24.526 +0200   Warning 661     filemaker        Client "name name (name name MacBook Pro) [1.1.1.1]" authentication failed on database "db.fp7" using "Admin [fmapp]".
            2011-09-08 15:28:21.676 +0200   Warning 661     filemaker        Client "name2.name2 [10.248.217.11:0]" authentication failed on database "db.fp7" using "name2.name2 [fmiwp]".
            2011-09-08 15:46:06.477 +0200   Information     518    filemaker        Administrator connected: "macadmin [10.0.10.3]" (sadmin).

            2011-09-08 15:36:30.335 +0200   0       0       0       13      100     0       7       3       0       1       0       2       0       265     0       0
            2011-09-08 15:36:45.335 +0200   1       0       0       0       100     0       7       3       0       1       0       5       0       271     0       0
            2011-09-08 15:37:00.337 +0200   1       11      1       13      100     0       7       3       0       1       0       6       0       635     0       1
            2011-09-08 15:37:15.337 +0200   0       0       164     0       100     0       7       3       0       1       0       0       1       6693801 1473    5421
            2011-09-08 15:37:30.340 +0200   0       2       171     0       100     0       7       3       0       1       0       1       2       2052570 160748  477
            2011-09-08 15:37:45.340 +0200   0       188     0       3       100     0       7       3       0       2       0       14      3       200512  9099    0
            2011-09-08 15:38:00.343 +0200   0       0       0       0       100     0       7       3       0       2       0       0       3       15001366        0       0
            2011-09-08 15:38:15.343 +0200   0       0       0       0       100     0       7       3       0       2       0       0       4       11521870        0       0
            2011-09-08 15:38:30.346 +0200   0       0       0       0       100     0       7       3       0       2       0       0       4       15001392        0       0
            2011-09-08 15:38:45.346 +0200   0       0       0       0       100     0       7       3       0       2       0       0       4       15001300        0       0

            Thanks again,

             

            Anton.

            • 3. Re: FM Server Elapsed Time/call in the millions and hanging
              TSDuck

              Anton Popov:

              What are the file sizes on the 2 databases you have hosted?

              How often are the server side scripts running?  What exact script jobs are they doing?

              Does either database use global fields in any calculations?

              The first thing you will want to look into is the databases themselves.  Pull both of your databases off the server and open them in FileMaker Pro.  Go under the file menu into manage external data sources.  If a database has multiple paths for one source, it can cause a delay as FileMaker searches for the valid reference.  You want to remove any invalid references from here so there is only one reference between both databases.  The second thing to do is create a compacted copy of each database file.  From the file menu select “Save a Copy as” to make a compacted copy.  In the “Save a” drop down choose compacted copy smaller and click save.  Upload the compacted copies and use those from the server.  Let me know if the issue continues after trying these steps.

               

              TSDuck

              FileMaker, Inc.

              • 4. Re: FM Server Elapsed Time/call in the millions and hanging
                AntonPopov

                Hi,

                I moved the server to a windows 2008 r2 and I observe the same results.If I try to close the database in question, it just stays at closing for hours.

                I compacted the databases with no result.

                The external data sources are ok - no multiple paths for one source.

                 

                Is there something else I can check?

                 

                Edit: there was a reference to an non-existent file in one of the entries for external data sources. I have removed it now, will comment again after I monitor this for some time.

                 

                Thanks,

                Anton.

                 

                • 5. Re: FM Server Elapsed Time/call in the millions and hanging
                  AntonPopov

                  Hi,

                   

                   No changes. I still observe the same behaviour - elapsed time/call goes up and stays threre, database is not responding, I can only reboot the server to bring it back up.

                   

                  Is there something else I can try?

                   

                  Thanks,

                   

                  Anton

                   

                  • 6. Re: FM Server Elapsed Time/call in the millions and hanging
                    philmodjunk

                    You might try taking the file (or a back up) down off the server and running a recover on it to see if any problems are reported by the recover operation.

                    Things to keep in mind about Recover:

                    1. Recover does not detect all problems
                    2. Recover doesn't always fix all problems correctly
                    3. Best Practice is to never put a recovered copy back into regular use or development. Instead, replace the damaged file with an undamaged back up copy if this is at all possible. You may have to save a clone of the back up copy and import all data from your recovered copy to get a working copy with the most up to date information possible.
                    • 7. Re: FM Server Elapsed Time/call in the millions and hanging
                      AntonPopov

                       Thanks for your response,

                      I ran the recovery and at the end I have

                      2011-12-02 10:34:13.300 +0100   icm.fp7 0       File blocks: scanned and rebuilt 482451 block(s), dropped 0 invalid data block(s)

                      2011-12-02 10:34:13.322 +0100   icm.fp7 0       Schema: scanned fields and tables; some problems were found...

                      2011-12-02 10:34:13.341 +0100   icm.fp7 0         fields created to match record data: 0

                      2011-12-02 10:34:13.363 +0100   icm.fp7 0         field values deleted due to invalid ID or repetition: 1

                      2011-12-02 10:34:13.383 +0100   icm.fp7 0         records deleted due to invalid ID: 0 2011-12-02 10:34:13.405 +0100   icm.fp7 0       Structure: scanned; 17 item(s) modified

                      but it's hard to tell what exactly was modified, lines read like:

                      2011-12-02 10:23:10.053 +0100   icm.fp7 0         Recovering: library '' (2)

                      2011-12-02 10:23:12.282 +0100   icm.fp7 8461      Deleted invalid field data, ID or repetition invalid

                      I'll try to make some sense out of this.

                       

                      Thanks again,

                       

                      Anton.

                      • 8. Re: FM Server Elapsed Time/call in the millions and hanging
                        philmodjunk

                        Hope you have a back up that recovers cleanly. This is your best option if you have it. You can import the data from your recovered copy into a clone of such a back up file.

                        • 9. Re: FM Server Elapsed Time/call in the millions and hanging
                          truji_

                          Same thing is still happening to me. I created a post about 2 months ago and did all of the above but nothing fixed the issue.

                          Server is still crashing, stuck remote calls, elapsed time in the millions. The only way to get the server back up is ending the process and starting all over.

                          I merged all the data, scripts, value lists, etc into a new database, tested all scripts, but  nothing makes the database purposely crash...

                          Users and of course me, are extremely frustrated... we hired a company to take a look at our problem but they cant find anything weird.

                          The server running is Win 2008 R2 (64bit), 12gb ram, Xeon 2.13 Ghz

                           no external data sources.all users use IWP to access the database.All scripts are compatible with IWP, excepet when remoteScripter is triggered.

                          We don't know what else to do...

                          • 10. Re: FM Server Elapsed Time/call in the millions and hanging
                            philmodjunk

                            If you have not already done so, please make a report of this in Report an Issue. This may be a bug in the software and bringing it to Filemaker Inc's attention there may get them searching their bug report database for similar reports and any workarounds/fixes that they might have on hand. Do any of your systems where this occurs use ODBC or JDBC?

                            • 11. Re: FM Server Elapsed Time/call in the millions and hanging
                              AntonPopov

                              Thanks,

                              I submitted a report.

                              I do not use odbc or jdbc and they are disabled.

                              Regards,

                               

                              Anton.

                              • 12. Re: FM Server Elapsed Time/call in the millions and hanging
                                codeus

                                Hi there,

                                We have just started to have this issue (since approx December 7th) on a database we have had running perfectly since September.

                                We are running FMSA11.0.3.309 on an HP Proliant Xeon Quad 3.1GHhz / 4GB RAM.  OS is Windows Server 2008 R2 SP1.

                                The database with the problem is approx 700MB, has around 15 tables and circa 50,000 records. Server loads are generally <10 concurrent users.

                                Our database is an education tool and the problem in our case seems to relate to users logging on via Filemaker Go and uploading images of pupils work to the file.  The database is accessed via Client, IWP and FM Go.

                                I noticed one user with a stuck call today so then logged in on my iphone. I was able to use the file fine until I tried to upload an image (iphone camera to container), at which point my connection also went to 1 stuck call and I got the spinning wheel of doom on my iphone, mean time users of other database are still able to do so freely.

                                So far we have tried this: -

                                -Uninstall FMSA11

                                -Uninstall IIS

                                -Clean install both above

                                -Revert to a file from before it started happening

                                -Performed a successful clean recovery of the file

                                -Imported records in to the clean clone.

                                The thing that bothers me the most is it only started happening last week.  I am wondering about Windows Defender updates perhaps (since these are the only changes that occured on the server), or a new client with a network issue at there end?

                                I would be interested to hear if the other guys with this problem use FM Go or significant Container uploads?

                                Regards,

                                Ed

                                 

                                Note, in the image below, iPhone5 was the first to hang, he closed the app and opened a new connection and now shows 2 entries, one hung and one ok.

                                • 13. Re: FM Server Elapsed Time/call in the millions and hanging
                                  codeus

                                  Any news on this from FM?  Any likely fix coming in FMS12 maybe?