13 Replies Latest reply on Mar 6, 2017 12:29 PM by wendy

    FM Server - What is the log telling me?

    wendy

      Good day, all!

       

      Our database was apparently not available one morning when users arrived. Unfortunately, I wasn't called, and when  I arrived several hours later I was limited to log analysis and quizzing users. I did serve a backup from 4:00 a.m. but am baffled by what I'm seeing in the log. Two screen captures attached.

       

      What happened:

      5:17:56 a.m. - a backup completed.

      5:25:46 a.m. - The log shows "Starting Filemaker Server processes" and additional automatic steps. There is no evidence of a power outage. Early users say the server had power, but was not accepting logins. I was unable to determine exactly what anyone tried (except a confused report that "we tried to log in but the password didn't work". Later story was that people typed the password incorrectly). Those here at that time wouldn't know to do anything except try to log in to a prompt or reboot the machine. (Why I wasn't called is another discussion I won't go into here.)

      8:30:47 - My analysis is that they let it sit until someone tried a reboot at this time, but then didn't log in to the console.

      9:24:11 -  Someone arrived who knew the password, and was able to log into the console and click on the button in the console to start up the server and open the databases. (I did take it down when I arrived a little later, and served a backup from 4:00 a.m., which I figured was "safe".)

       

      Questions:

      - Any idea why user's couldn't log in when they first arrived in the morning? Something quit working, but I'm not sure what.

      - I'm baffled by the "Starting Filemaker Server processes" at 5:25 when there is no evidence that anything was down, and no log entries showing anything went down. All I have is users saying "We couldn't log in."

      - How to determine the closest "safe" backup to re-serve when there has been a server problem; what to look for in the log? (I'm the only admin, but I'll be training "volunteers" for future emergencies, so I have to keep it simple.)

       

      3.jpg4.jpg

      System Details:

      - Filemaker Server 15.0.3.308, upgraded recently from FMS13.

      - iMac running 10.11.06

      - The server runs 24/7. We have a UPS battery backup set to run for 4 minutes after a power outage before a shutdown. Computer AND display sleep settings are both set to "NEVER".

      - The previous server was set to serve upon reboot by a previous admin--I've ended that practice.

      - Later in the day, I did a "pull the plug" test. The UPS performed as expected, and I saw expected "shutting down" entries in the log.

       

      Thank you.

        • 1. Re: FM Server - What is the log telling me?
          wimdecorte

          You'll need to look at the various system logs that OSX provides, pretty sure the machine went down hard.  If so there should be some clues in the various logs.  Open the Console.app (not the FMS one) and check the various logs on the left hand side.

           

          If you have the stats.log active in FMS (and you should, but it is off by default), then you'll probably have a better idea of when exactly the server went down).

          1 of 1 people found this helpful
          • 2. Re: FM Server - What is the log telling me?
            wendy

            Thanks, wimdecorte.

             

            I investigated the Console All Messages log. That doesn't quite go back to the crash date; I took too long to get back to this issue. But I appreciate the direction should there be another.

             

            I followed your advice to look at the stats log. It isn't showing much activity; there weren't any users on the system that early in the morning. However, (see screen capture) I'm seeing a point at which the number of served databases went from 21 to 0. So does it make sense to direct others to use this point as a reference for a usable backup?

             

            Dbase OK at 5_24 I think.png

            The FMS events log on the server hard drive only goes back 15 days, which surprised me; I thought it would log until I reached the 40MG size--and then save an "old" version. (I don't see an "old" version on the hard drive.) I've attached a screen capture of the current log tab settings, please advise if there is anything I'm overlooking here.

            Log settings.png

             

            Thank you!

            • 3. Re: FM Server - What is the log telling me?
              wimdecorte

              wendy wrote:

               

              I'm seeing a point at which the number of served databases went from 21 to 0. So does it make sense to direct others to use this point as a reference for a usable backup?

               

              5:24 is the last recorded stats event, after that there is a big gap to 9.  So any backup that completed before 5:24 should be safe.

              • 4. Re: FM Server - What is the log telling me?
                wimdecorte

                wendy wrote:

                 

                The FMS events log on the server hard drive only goes back 15 days, which surprised me; I thought it would log until I reached the 40MG size--and then save an "old" version. (I don't see an "old" version on the hard drive.) I've attached a screen capture of the current log tab settings, please advise if there is anything I'm overlooking here.

                Log settings.png

                 

                 

                That *is* weird and slightly worrying.  Keep an eye on that events log for the next few weeks.

                 

                I do notice that you've changed the stats logging interval from its default 30 seconds to 60.  I would recommend setting it back to 30 so that it logs more stats measuring points.  Was this change done for any particular reason?  Performance considerations?

                • 5. Re: FM Server - What is the log telling me?
                  duncanbaker

                  Wendy, the FileMaker Admin Console will only show you the last 15 days worth of logs in the Log Viewer side menu item. The actual log file located here: /Library/FileMaker Server/Logs/Event.log should contain more than the past 15 days. I'd suggest copying that file to the desktop and then open it like you did the Stats file and perhaps you'll find the logs you're looking for.

                  • 6. Re: FM Server - What is the log telling me?
                    siplus

                    IMHO your server restarted automatically after being hit by a kernel panic.

                     

                    AFAIK kernel panics mostly occur because of a malfunctioning hardware attached to your mac.

                     

                    See here. And also here.

                     

                    I've already had this kind of problem at a couple of clients and it stopped happening after we uncoupled external HD's (in one case a Pegasus).

                    1 of 1 people found this helpful
                    • 7. Re: FM Server - What is the log telling me?
                      wendy

                      duncanbaker, I actually was looking at the log on the hard drive. Right now it only goes back to the evening of 2/1/2017 (the crash occurred early that morning) at 18:26:48.

                       

                      I'd deliberately pulled the plug on the UPS that evening, to test the battery and try to understand more about that process. But strangely, the log now starts right in the middle of closing the databases at 18:26:48, with 17 of 21 files listed as closing, but everything before that is gone. I see the expected lines about the various components shutting down after that, and normal entries (as far as I can tell) up to present time.

                       

                      I'll be keeping an eye on it to see if it keeps accumulating as time goes on.

                      • 8. Re: FM Server - What is the log telling me?
                        wendy

                        siplus, that's interesting. I wonder if it has anything to do with the two connected external drives (both Seagates). One has been in use for a long time. The other was added in December when we upgraded the server to El Capitan (wiped the drive and started clean). I used it to run a TIme Machine backup, then turned TIme Machine off. (It will remain off unless I make a major system change, in which case I'd run TM once during an off time.) I will eject that drive--there's no reason to have it connected until/unless I'm actually running TM.

                         

                        I looked in Library/Logs, but see only a "Diagnostic Reports" folder, which is empty, and a setup log which was apparently created back when the system was overhauled.

                        • 9. Re: FM Server - What is the log telling me?
                          wendy

                          Thanks, wimdecorte, the info about the "gap" is helpful.

                           

                          Yes, I did change the logging interval from 30 to 60, with the vague notion, at the time, that recording less might mean the log would go back further. Which sidesteps the issues as I now understand them. I'll change it back to 30.

                          • 10. Re: FM Server - What is the log telling me?
                            wendy

                            One other note: in the Statistics tab/viewer, my choices of which statistics to view (e.g. Cache Hit, Elapsed Time) keep unchecking themselves. (It may be when the FM Server Console logs itself out; as far as I know there is no way to stop that logout.)

                            • 11. Re: FM Server - What is the log telling me?
                              wimdecorte

                              Those settings are not meant to persist and they don't affect what FMS actually logs to the stats.log file.  That log file will have data on all the counters it keeps track off, regardless of what you look at in the admin console.

                              1 of 1 people found this helpful
                              • 12. Re: FM Server - What is the log telling me?
                                wendy

                                Oh good, that's the answer I wanted...!

                                 

                                Thanks.

                                • 13. Re: FM Server - What is the log telling me?
                                  wendy

                                  Update: I temporarily set our log size to a few megabytes over its current size to see if it saved an "old" version of the events log and created a new one, as expected. It did, so that now appears to be working correctly.