1 2 3 Previous Next 35 Replies Latest reply on Dec 3, 2015 1:56 PM by TSGal

    Filemaker Server 13 randomly crashes, mostly while doing scheduled backups

    pavle

      Summary

      Filemaker Server 13 randomly crashes, mostly while doing scheduled backups

      Product

      FileMaker Server

      Version

      Current Installed Version: 13.0.5.520

      Operating system version

      10.8.5

      Description of the issue

      Hi,
      let me join the group of people that are reporting an issue with FMS crashing randomly.
      My company is one of the bigger Filemaker customers. We have approximately 200 users using the system all the time, 6 days a week. Our business is a retail store, so we write to the system consistently.
      We have a connection with mySQL/PHP part of the system, where we  replicate a lot of the FM data through robot clients. We are using ODBC Actual Technology drivers for that purpose.
      Our backups are running on 30 minutes intervals. The size of all the dbs is around 20Gb spread in around 80 dbs. Most of our dbs are one db with one table. The reason for that is an easy and fast recovery in case of a damaged file.

      We have recently moved from FM11 to FM13, primary because of the memory leak we have noticed on our old server that was leading to server crashes. As soon as we hit 8.5Gb of Real Mem usage for the fmserverd on the Activity Console, there was a good chance we would get a crash, followed with verification data and corruption on some of the opened dbs.
      We have worked with couple of people from Filemaker on that issue, but they were not able to find the reason for the memory leak. The workaround was to restart the server as soon as it hits the 8.4Gb real mem value.
      We thought that with FM13, 64 bit app, and all the new features, our crashes will be part of the history, but we are not that lucky.
      Our current FM server is on Mac Pro, 3.2. GHz Quad Core Intel Xeon, with 32 GB DDR3 RAM, and the HD is 500Gb SSD, with 300Gb available space.
      Our clients are on many different types of computers, but no Go devices yet. OS on those devices are ranging from 10.8.x to 10.10.x
      The memory leak that we had from FM11 is probably still with us in FM13, just a bit more intense maybe. Because of that, we still are restarting the server on timely basis, just at different real memory amount. When we restart the FM 13 server the start up real mem reading is 8.5 Gb. We have noticed that the server was crushing when we reach 14.5Gb and up. To reach 14.5Gb threshold, it takes around 7-10 days. For FM11 to reach the 8.4Gb threshold it took 17-21 days, so we have to restart the server more frequently than before.
      One very interesting fact is that in lot of our crashes, when we open the console it is unresponsive. One can try to disconnect clients, close dbs, stop the server/web publishing engine, but nothing is responding or happening visually. Then we need to restart the computer itself.
      Another thing I have noticed is that when we get a crash, on the console often it says that a backup is still running, and the time of the backup’s start is maybe 20+ minuets prior to the crash. Usually our backups finish in 2-3 minutes, so it saying that it is still running after 20 minutes is maybe a lead to some of the reasons for the crashes.
      I am concerned that the system is not able to un-pause the dbs while it’s backing them up, and that is putting it in some kind of loop, that leads to a crash.
      I can submit the log files to the interested parties. I have two logs’ locations:
      /library/Filemaker Server/Logs - 4.22Gb total, due to one of the log files stderr being 4.08Gb and it's being constantly updated.  
      /Library/Logs - where the DiagnosticsReports folder is located (1.1 Mb total)
      I have tried to open the stderr file, but none of the app I have tried are opening it (Text Edit, Word, TextWrangler).

      Please, let me know which log files will be beneficial for me to upload?

      I apologize if this issue was already discussed in some other thread. I have found some threads about FM Server crashing, but was not able to see the same reasons as in my case.

      Thank you.
      Pajo

      Expected result

      Solution to prevent crashes

      Actual result

      None yet

      Workaround

      none so far

        • 1. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
          TSGal

          pajonate:

          Thank you for your post.

          I will look at your FileMaker Server log files and system log file.  Check your Inbox at the top of this page for instructions how to get these files to me.

          TSGal
          FileMaker, Inc.

          • 2. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
            TSGal

            pajonate:

            Both of the RCConnection::Send crashes on 01-25 and 01-14 happened during a shutdown of the database files.  Testing would like to know if you are shutting down the server in any special way?  This may help us reproduce the crash by performing the same exact steps while some action is being sent to the client at the same time (for example, perform a large find request, disconnect network cable of client, stopping database very quickly, etc.).

            The "clock_get_time() failed" messages may be due to an Apache port leak issue.  See the Apple discussion at:

            https://discussions.apple.com/thread/5242300

            TSGal
            FileMaker, Inc.

            • 3. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
              pavle

              TSGal,

              sorry for not answering your messages, but, it's been crazy lately with lot of assignments here at work.

              Regarding your question, I have no special way of shutting down the server. What we try to do, before the crash, or after the crash is to disconnect all clients (which usually does not work, the web console is iresponsive ), then we try to close the files (also usually the console is deaf to this action), then I disable all the schedules, and limit the client access to 0. When I save after this last change I get "safe successful", but in reality I am not sure, because after the hard restart the server is not limiting the clients. Before the hard restart, I try to flip the switches for the Fielmaker server and Web Publishing Engine. All of this is not producing any helpful outcome, so the next thing is "Apple/Restart/Cross your fingers and pray" button.

              The logs are not telling me if someone is doing a large find, or someone disconnected themselves or similar, so as I said with so many active clients, I am not sure what may be the cause at the user's end.

              Pavle

              Tekserve

              • 4. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                TSGal

                pajonate:

                Thank you for the additional information.  I have sent this back to Testing for review.  When any new information becomes available, I will let you know.

                TSGal
                FileMaker, Inc.

                • 5. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                  TSGal

                  pajonate:

                  Development has replied with the following information:

                  When the customer wonders whether the Admin Console is working or not, have the customer go to Databases view and Close or Pause one of the database files.  If the database view doesn't change right away to Closing/Closed or Pausing/Pause, then this means the Admin Console is not really connected to the database server.  Run the command line "fmsadmin restart adminserver".  This often fixes this problem, although it will require the customer to login to the Admin Console again.

                  If the Admin Console doesn't respond at all, or a database file gets stuck at Closing or Pausing, then try issuing the command:
                     fmsadmin list files -s

                  If that works, then issue the command:
                     fmsadmin close

                  ... to close all files before shutting down.

                  TSGal
                  FileMaker, Inc.

                  • 6. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                    pavle

                    Thanks, TSGal,

                    I haven't tried "fmsadmin restart adminserver" in case of console not reacting, even that I knew about it. I had to use it recently, when I was getting an error 500 when I wanted to open the Admin console from Safari (the rest of the server processes were fine, just I could not get to the console). That was a bit scary, because the first online info I was able to find about it was "you need to re install the server in order for this to work". After couple of more searches, I have found a suggestion to use this command, and it worked.

                    It is good to know about the other commands, too.

                    In one of our emails or threads, there was a something about Apache's memory leak, and a suggestion of restarting it every night. Do you recommend that, since our fmserverd memory use is climbing in quite a fast pace after each restart. At the restart point it reads 8.5 Gb, but in week or so, it climbs to 14.5Gb. We are concerned that when it reaches that threshold, there is a good chance of FM Server to become  not responsive and to crash.

                    Thank you and take care.

                    Pajo

                    • 7. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                      TSGal

                      pajonate:

                      Reading the Apple discussion link, the one customer used the link https://codedmemes.com/lib/newsyslog-automatic-log-rotation/ to rotate the log files and restart Apache at 12:30 AM each night.  This is what I was referring to.  It helped this one customer, but I can't guarantee it will solve the issue.

                      TSGal
                      FileMaker, Inc.

                      • 8. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                        pavle

                        I may try it, to see if my fmserverd real mem usage amount will get higher slower that it does now.

                        • 9. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                          pavle

                          Hi, TSGal,

                          unfortunately we had another instance of FM server locking us out and net being responsive.

                          Pretty much the same as I described in my original post.

                          At 3:37pm yesterday, users started reporting that FM client app is not responsive, beach balling, etc. NO disconnection, just not responsive. At the same time we noticed that the Web Publishing Engine is down, and we thought that it may play a contributing role for this problem.

                          I accessed the server console, and of course, it was not responding well to my commands. I wanted to disconnect the users, which it kind of did, visually, but then looking at the open dbs, all of them were reporting +100 users as still having them open as clients. After that I wanted to close the dbs, which got stuck in "closing" for all of them, but none as "closed". After couple of minutes, a flipped the switch to stop the server and the WPE, but nothing happened. Next thing was restarting the server machine, which brought up the server successfully, with couple of dbs verifying for (luckily) short time. 

                          Now to the one issue I have noticed last time and I assume might be the biggest issue here, and that is the backup process. As I said first report happened at 3:37pm. We have backups running at every half four at 00 and 05 mins (3:00, 3:05, 3:30, 3:35 etc.). Those are two different backups running every 30 mins. When I looked at the console, it was reporting that one of the backups is still "running", all the others were "OK". What was weird is that it was reporting "Last completed" as 3:05pm, and "Next run" as 3:35pm. even that it was 3:55 when I had to restart the machine. Usually, it takes less than a minute to finish that backup (only 3 files, 8Gb total). That's our archive, which is not used actively, just on rare occasions.

                          I have gathered the logs folders from the library/Filemaker server/logs and library/logs and I will send you the link for them, and will appreciate if your technical team can retrieve anything from them, to help us understand what is going wrong.

                           

                          Thank you and take care.

                          Pajo

                          • 10. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                            TSGal

                            pajonate:

                            Testing has looked through your event.log and stats.log, and in the event.log just prior to FileMaker Server going deaf, the Elapsed Time column spikes to 24 seconds.

                            The reason for the Server going deaf is unclear, but from page 120 of the FileMaker Server Help (https://fmhelp.filemaker.com/docs/13/en/fms13_help.pdf):

                            Elapsed Time (µs)/call
                            Time (in microseconds) to process a remote call.  For example, a value of 1 million means an average of 1 second per call.  If the average Elapsed Time/Call is high, but the average Wait Time/Call is low (which it is in your case - 0 seconds), ask clients whether the operations they are performing on the server have changed.  Adding indexes for more fields or other solution changes may be required.

                            TSGal
                            FileMaker, Inc.

                            • 11. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                              pavle

                              Thanks, TSGal,

                              unfortunately asking over 200 users what they have looked up/run as script, etc  at that time is close to impossible.

                              What I have noticed out of the ordinary is that our backup was also labeled as running on the Server console, even that it was suppose to be done some time ago. Can your team see if anything with the backup stalled the whole system. We have 2 backups running every 30 mins. One is triggered at 00 and 30 mins of the hour, and the other 05 and 35 mins of the hour. Please, check my notes in the previous note for more details. I am scared that the system may have not be able to un-pause the dbs that lead to a problems to everyone else.

                              Thank you for your efforts.

                              Pajo

                              • 12. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                                TSGal

                                pajonate:

                                I did notice the event log at 2015-03-09 15:35 when the backup schedule started.  The log also showed that it was attempting to pause up to 18 databases, and in the next few minutes, the server went "deaf", because at 15:40, the server schedule was disabled, and two minutes later, the client "W" (keeping your data confidential) was then forced to close the connection, and no further connections were allowed.

                                However, at 15:32 is when the spike in Elapsed Time started occurring, so Server was already in trouble before the 15:35 backup started.

                                I have asked for more information from the Tester.  I will continue to keep you posted.

                                TSGal
                                FileMaker, Inc.

                                • 13. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                                  pavle

                                  Thanks, TS Gal,

                                  any help is appreciated.

                                   

                                  Take care.

                                  Pajo

                                  • 14. Re: Filemaker Server 13 randomly crashes, mostly while doing scheduled backups
                                    TSGal

                                    pajonate:

                                    Sorry for the late response.

                                    The tester says the cause is unknown.  If you could send in your system logs for that time frame, it may provide a clue.  I know quite a bit of time has passed, so if the system logs are not available, let me know the next time this issue occurs, and then send in the FileMaker Server logs as well as the system log.

                                    TSGal
                                    FileMaker, Inc.

                                    1 2 3 Previous Next