1 2 Previous Next 23 Replies Latest reply on Apr 27, 2012 6:44 AM by Norsult

    No Event Logs and Server Unexpectedly Quits

    Linda@compu-books.com

      My client is running FMSv11.04 (latest version) and I am not seeing ANY event logs. The event logging is on by default and cannot be changed, yet I'm not getting any event logs. Is anyone else having this issue?

       

      The reason I was prompted to look at event logs was because their FM Server unexpectedly quit twice in the last 2 weeks and I'm looking for why that happened. Does anyone know if Carbon Copy or Retrospect backup apps would have any conflict with FM? They have been setup to only backup the 'backup' folder, but wondering if there could be an issue. The system crashed when a scheduled script was running to save documents from a web based location. The documents file ended up getting corrupted. Not sure if the script hanging caused the crash, or the crash caused the script to hang, but either way the end result was a corrupt file. I was able to restore the file from backup, but still trying to discern why the crash happened. Any insight would be appreciated. If you can copy your reply to me at my email address that would be great!

       

      -------

      Linda McFee, Compu-Books

      www.compu-books.com

      Linda@compu-books.com

        • 1. Re: No Event Logs and Server Unexpectedly Quits
          Stephen Huston

          Always be suspicious of backup software, though it sounds like you have properly omitted the FM Server data directories from the area external backups can touch.

           

          However, if the script is attempting to save a document to a directory that is being backed up, a conflict could occur if the backup and the save-script collide. Some backup software tries to grab   directory info from where it's backing up and doesn't deal well that info changing mid-stream.

           

          I would assume you have some control over the backup utility schedule. And the "scheduled script" is, I assume, a Server-based FM script, so it should be timed to not run near when that backup utilitiy will kick in.

           

          Or it may be something else entirely. Do the permissions and settings of the target directory for the Server script need to be part of the backup area? If not, try removin them. Are the permissions of that directory still correct for FM Server script access? Has the backup utility modified permissions (shouldn't, but who knows when something goes wrong.)

           

          As for FM Server crashing (really?) --  there is a chance that if the script failed and became hung-up, the server would eventually fail. This isn't common (I haven' seen it happen myself except with the IWP module), but you might want to put in some error capture routines with the Save script so that if that script fails it has an out:

           

          (pseudo-script code)

          -

          Set Error Capture [ON]

          Save/export/whatever step here

          If Get(LastError) ≠ 0

               Do this instead (send email to DB manager?)

          End If

          -

          And remember when revising a Server-side script to verify that all script steps are still server-compatible when you're done editing. Then try testing it as a client-called script to be sure it's still working as expected and able to write its output.

          • 2. Re: No Event Logs and Server Unexpectedly Quits
            Linda@compu-books.com

            Thank you Stephen. I did find out that the Carbon Copy backup was including the 'databases' folder and their IT guy has corrected it.  We also rescheduled the backups to be fewer and the server scripts to be running less often as well.

             

            What about the issue with not having any event logs?  Have you seen that?  It should be saving logs for backups and other scheduled events, but it's not.  There doesn't seem to be any way to turn that on or off.

             

            -------------
            Linda McFee, Compu-Books
            www.Compu-Books.com

             


            • 3. Re: No Event Logs and Server Unexpectedly Quits
              Stephen Huston

              Hi Linda,

               

              Like, I said, always suspicious about backup utilities. Glad that got sorted.

               

              Regarding Serer Logging: Generally I check the scheduled events to see when server scripts and backups were run, or set the server to email me notifications of completed server scripts so I know a backup was completed.

               

              As for logs, I leave them turned off as they impose a real overhead on the server. I turn them onn in the admin panel if I want to watch something happening live for troubleshooting, but I don't attempt to leave them on. At one time, I thought I had heard that the full log disabled itself when not being viewed, but I simply don't know if that is the case or not, having not attempted to run them in the backgound. We avoid things like that which add to the server load and response.

               

              Someone else here should be able to answer about server logging; I simply don't attempt it unless I'm troubleshooting.

               

              Sorry I can't be  more authoritative regarding the log issue.

               

              Stephen Huston

              • 4. Re: No Event Logs and Server Unexpectedly Quits
                Stephen Huston

                And keep in mind that having allowed the backup utility to hit the live data directory (though now corrected) could have already introduced new file problems (corruption). It may be worth taking a copy of the current files off the server and run the Recover on them to see if the log indicates anything obvious.

                 

                Don't use the recovered file, just examine the recover log to see if any problems were found. It's not a gurantee that they're clean, but it could tell if something specific went wrong.

                • 5. Re: No Event Logs and Server Unexpectedly Quits
                  Linda@compu-books.com

                  Yep, I did have corruption in one file and was able to recover from a backup.  And I did pull down a copy of all their files to test for corruption as well.  Thanks for your feedback Stephen!

                  • 6. Re: No Event Logs and Server Unexpectedly Quits
                    sporobolus

                    on 2012-03-21 12:55 Stephen Huston wrote

                    And keep in mind that having allowed the backup utility to hit the live data directory (though now corrected) could have already introduced new file problems (corruption).

                     

                    i can't think of how simply backing up the live database could introduce

                    corruption in the live copy; the backed up version would be at risk of

                    corruption, but the live version should not be altered nor should ongoing

                    writes be affected

                    • 7. Re: No Event Logs and Server Unexpectedly Quits
                      Stephen Huston

                      I don't know how or why either, but I recall many years ago in a FMI product briefing  we were warned that "copying" a live/open file can corrupt both the source and the resulting copy.

                       

                      They didn't say why, but  it's certainly been reported. That's the ultimate danger of letting backup utilities run on the served files, being left with nothing that's safe.

                      • 8. Re: No Event Logs and Server Unexpectedly Quits
                        sporobolus

                        on 2012-03-22 18:04 Stephen Huston wrote

                        I don't know how or why either, but I recall many years ago in a FMI product briefing  we were warned that "copying" a live/open file can corrupt both the source and the resulting copy.

                         

                        They didn't say why, but  it's certainly been reported. That's the ultimate danger of letting backup utilities run on the served files, being left with nothing that's safe.

                         

                        i still don't buy it; it sounds like urban legend or someone who was a little

                        too zealous in warning; there is no plausible mechanism for this

                         

                        the only reason not to back up live files is because you can't trust the

                        backups; if at some point you realize your backup system hasn't been configured

                        to skip the live files, just don't restore those files (and fix the configuration)

                         

                        i believe the corruption Linda reported must have been caused some other way

                        • 9. Re: No Event Logs and Server Unexpectedly Quits
                          Linda@compu-books.com

                          Steve, are you new to FileMaker?  Backing up live copies has been a known issue to cause corruption since the first version.  Read the manual.  I've been developing with FileMaker over 20 years and I've seen it happen plenty of times. I have the hardest time convincing IT people NOT to backup live databases. FileMaker is different than other software in this way, but you will probably have to crash and burn to believe it.  Only FileMaker Server can safely backup live files to a 'backup' folder.  Then you can have other backup software backup the 'backup' folder, but never, never, never backup the 'databases' folder with the live copies!!!

                           

                          Hope this saves you the bad experience...

                           

                          Best -

                          Linda

                          ------

                          Linda McFee, Compu-Books

                          www.compu-books.com

                          • 10. Re: No Event Logs and Server Unexpectedly Quits
                            sporobolus

                            on 2012-03-23 15:01 Linda@Compu-Books wrote

                             

                            Steve, are you new to FileMaker?  Backing up live copies has been a known issue to cause corruption since the first version.  Read the manual.  I've been developing with FileMaker over 20 years and I've seen it happen plenty of times. I have the hardest time convincing IT people NOT to backup live databases. FileMaker is different than other software in this way, but you will probably have to crash and burn to believe it.  Only FileMaker Server can safely backup live files to a 'backup' folder.  Then you can have other backup software backup the 'backup' folder, but never, never, never backup the 'databases' folder with the live copies!!!

                             

                            no, i'm not at all new to FileMaker; and i'm not advocating backing up live

                            database files, if you read it that way, please read again

                             

                            what i am saying is that the act of backing up the database file will not make

                            the original — live, current, open, operating — database file corrupt; it is

                            only the backup copy that may become corrupt; making such a backup is not

                            inherently harmful, it's just not a good backup; this is not at all unique to

                            FileMaker

                             

                            so if you are using FMS to schedule backups, and you have backup software that

                            makes further copies of those backup files, then it is not such a big deal if

                            the live files were also included in the backup; it's a good idea to fix the

                            backup configuration, but the portion of your backups that were made properly

                            are safe, and your live files are too

                             

                            if you believe otherwise i'd like to hear your explanation of how that could

                            happen, or to see you prove it experimentally

                            • 11. Re: No Event Logs and Server Unexpectedly Quits
                              Stephen Huston

                              As for what we "believe" making any difference, it never does.

                               

                              I believe that  FMI has warned repeatedly for many years that this can cause corruption and they must have some reason rather than to spread fear among developers, so some of us who are prudent take that warning and act as if FMI might be right. Not believing it can happen is inadequate protection for my files.

                               

                              So, No, I don't know why corruption may happen from this, but I assume it can if FMI warns me it can. That's  safer than believing the FMI engineers are wrong.

                              • 12. Re: No Event Logs and Server Unexpectedly Quits
                                ch0c0halic

                                Steve may be working with Zombie files,

                                 

                                Backing up, indexing, or "virus protection checking" a live FMP file is bad. It will eventually corrupt the LIVE file.

                                 

                                FMS requires complete and total control over the file when performing any action with that file. If the file is being touched (even reading the file) by another application or service it prevents FMS from getting that complete control. That is an OS level status the FMS relies upon for file integrity. If FMS does not have complete control when it access the file it will shut down the file, the file is improperly closed, and all cached data is lost.

                                 

                                The key here is timing. If the other application or service accesses the file while FMS is NOT accessing it then FMS doesn't know its been tampered with. BUT, you cannot guarantee that that will not happen because the FMS access is asynchronous with all other accesses. Eventually they will collide.

                                • 13. Re: No Event Logs and Server Unexpectedly Quits
                                  sporobolus

                                  on 2012-03-26 12:13 ch0c0halic wrote

                                  Backing up, indexing, or "virus protection checking" a live FMP file is bad. It will eventually corrupt the LIVE file.

                                   

                                  FMS requires complete and total control over the file when performing any action with that file.

                                   

                                  thank you for offering a technical explanation; i'm conversant with but not

                                  expert in OS fundamentals (particularly OS X/*nix), so please help me

                                  understand the actual mechanism …

                                   

                                  the only way i can relate "complete and total control" to what i know of file

                                  system operations would be if FMS were to request an exclusive lock on a file,

                                  which would fail if the file were open by another process; but from the

                                  behaviors (particularly the fact that so many people accidentally back up live

                                  files) it doesn't seem that FMS uses exclusive locks ... what am i missing here?

                                   

                                   

                                  If the file is being touched (even reading the file) by another application or service it prevents FMS from getting that complete control.  That is an OS level status the FMS relies upon for file integrity.

                                   

                                  so other than an exclusive lock what specific "OS level status" do you mean by

                                  "total control"? what does reading a file prevent FMS from doing?

                                  • 14. Re: No Event Logs and Server Unexpectedly Quits
                                    ch0c0halic

                                    Steve may be picking new locks,

                                     

                                    I'm pretty sure it is an exclusive lock. It happens when FMS accesses the file and is released when it finishes whatever transaction was performed. So the timing of this access is very important. The more data in cache, the less new data being entered, the fewer cache flushes performed makes the intersection point get smaller. But, people who backup live files eventually have their files get corrupted, or they go off-line, and otherwise have 'unexplained' bad things happen to their good files. §^=(

                                     

                                    I have seen instances of Spotlight indexing files in OSX and having a FMS file suddenly disappear from the list of available files.

                                     

                                    Its all about the timing. I have heard people say they back up live files. But, typically the data set is small, is almost all stored in Cache, and the new data being entered is minimal. They get away with it for a long time until suddenly the amount of data reaches a level requiring more disk access and the system 'suddenly' has problems they've never seen before. That's when they learn the messy reality when backing up, copying, indexing, virus scanning live files.

                                     

                                    Some people do things like restart the computer every day or every week because strange things happen on the weekend. Virus checking day(s)?

                                     

                                    BTW, did I mention its about the timing? Yes, you can get away with it a lot of times. Especially when the backup is performed in the middle of the night when FMS is basically not accessing files because there has been no activity.

                                    1 2 Previous Next