7 Replies Latest reply on Jul 3, 2012 1:50 PM by TonyWhite

    Batch ³Advanced Recover² XPOST

    TonyWhite

      Hi All,

       

      Question at the end. First the use case...

       

      Client reported that one of their files hosted on FileMaker Server 10 had

      closed by itself.

       

      I was able to use the FileMaker Server Admin Console to open the file.

       

      Naturally, I wanted to test the file to see if there were any issues:

       

      1. Ran FMDiff: no issues found

      2. Ran a backup schedule with ³verify backup integrity² turned on: no

      issues found

      3. Ran File->Recover...->Check Consistency button: no issue found

      4. Ran File->Recover...->Select File->²Use Advance Options² checkbox: here

      is where it gets interesting...

      (Please note that I have NO plans to use the removed file. I am running a

      diagnostic recovery on a throw away copy of the file.)

       

      ...at the bottom of the report was the message:

      ³2012-06-28 10:26:27.749 -0400 FileName.fp7 8495 WARNING: problems

      were detected while recovering the database. The recovered file should NOT

      be used going forward; copy only the most recent work from it into a backup

      copy of the original file.²

       

      4 custom function and 4 layouts were flagged in the recovery log. Below is

      one of each.

      2012-06-28 10:26:25.244 -0400 FileName.fp7 0 Recovering: custom

      function 'myTrFile_CreateFile' (22)

      2012-06-28 10:26:25.245 -0400 FileName.fp7 8477 Calculation

      modified

      2012-06-28 10:26:25.245 -0400 FileName.fp7 8476 This item

      changed

      ...

      2012-06-28 10:26:25.880 -0400 FileName.fp7 0 Recovering: layout

      'LayoutName - seems to be working OK' (473)

      2012-06-28 10:26:25.880 -0400 FileName.fp7 8462 Deleted invalid

      layout part

      2012-06-28 10:26:25.881 -0400 FileName.fp7 8497 Updated last

      layout part key (1)

      2012-06-28 10:26:25.892 -0400 FileName.fp7 8487 Reset table view

      2012-06-28 10:26:25.895 -0400 FileName.fp7 8476 This item

      changed

       

      I suspect that the 4 Custom Functions were flagged because the plug-in they

      reference was not installed.

      I went to each of the 4 layouts in both the browse mode and the layout mode

      and they did not crash, etc.

       

      The clients system consists of 108 files with 3,639 layouts, etc.

       

      I would like to run a diagnostic recovery on each of 108 files and do the

      same on a number of early backups.

       

      Is there a way to run a batch ³Advanced Recover² or we going to have to

      build one with AppleScript, QuicKeys and/or scripting the user interface?

       

      Thanks for any thoughts on this.

       

      All the best,

       

       

      Tony White

      Tony White Designs, Inc.

      Tel: 718-797-4175

      tony_white@twdesigns.com

      http://www.twdesigns.com

        • 1. Re: Batch ³Advanced Recover² [XPOST]
          wimdecorte

          Hi Tony,

           

          The one big thing that I'm missing in your troubleshooting efforts is back-tracking through the event log to see when the file was closed and what event caused it. There will be something there and it should provide you with a solid clue.

           

          HTH

           

          Wim

          • 2. Re: Batch ³Advanced Recover² [XPOST]
            TonyWhite

            Hi Wim,

             

            Thanks, good question. I should have addressed that in my original post.

             

            The file closed was on 2012-06-28 and I did not see any server events on or

            around that date.

             

            I just exported the entire month 2012-06 for all 6 Module Selections:

            • Server Events

            • Server Access

            • Publishing Engine

            • Publishing Engine Access

            • Web Server

            • Web Publishing Core

             

            All 3 types of logging have been on for years Configuration->Database

            Server->Logging:  (FMS 10)

            • Event

            • Access

            • Usage Statistics

             

            There are no Server Events for the entire month (lots of Server Access).

             

            It is interesting that the file can pass the ³verify backup integrity² and

            the ³Check Consistency² tests and fail the ³Advanced Recover² test!

             

            The good news is, I have a good pattern of 99 hourly, 30 daily and 52 weekly

            backups.

             

            The file in question has...

            • 4 tables

            • 61 Table Occurrences

            • 437 fields

            • 57 Relationships

            • 471 Scripts

            • 10,786 Script Steps

            • 148 Layouts

            • 4,516 Layout Objects

            • 42 Value Lists

            • 36 Custom functions

            • Custom Menus

            • Security that I can reconstruct relatively easily

            • 7,184 Calcs

            • Lots and Lots of Blood, Sweat and Tears.

             

            ...and is under active development.

             

            I hope to never have to rebuild the file!

             

            Therefore, I on a mission to figure out the best way to run more ³Advanced

            Recover² to generate more data points to base an action plan on.

            I wonder if there is a command line equivalent?

            Running Advanced Recover on 108 files n times (even 1 file 993052 times)

            calls out for some form of automation!

             

            I would rather that the file had not acted up (acted down?)

            That said...it does bring up interesting core questions.

             

            Open to additional suggestions from any and all.

             

            Thanks again.

             

            All the best,

             

             

            Tony White

            Tony White Designs, Inc.

            Tel: 718-797-4175

            tony_white@twdesigns.com

            http://www.twdesigns.com

            • 3. Re: Batch ³Advanced Recover² [XPOST]
              wimdecorte

              TonyWhite wrote:

               

               

              The file closed was on 2012-06-28 and I did not see any server events on or

              around that date.

               

              There are no Server Events for the entire month (lots of Server Access).

               

              That is something I have never seen before.  Is this a Mac or Windows server?  What's the OS on the box?

               

              FMS is very dilligent in loggin events, I've never ever seen it NOT do it so I'm very suprised to hear that you can' find any events.

              Are you looking only from the FMS admin console or looking in the FMS logs folder?

              • 4. Re: Batch ³Advanced Recover² [XPOST]
                TonyWhite

                FMS Machine specs:

                • FileMaker Server 10.0.2.206

                • 10.4.11

                • Xserve 2 x 2.66 GHz Dual-Core Intel Xeon

                • 8 GB RAM

                 

                Good tip, I was looking at the logs only from the FMSAC filter and that was

                what was displayed!

                When I looked at the Logs folder...

                /Volumes/Boot/Library/FileMaker\ Server/Logs/

                ...events.log directly and zoomed in on the time when the file was closed I

                see these 2 lines:

                 

                2012-06-28 09:49:24.204 -0400    Error    618    fms10    Database

                "FMTEMPFM4fc3d4c7006d" is damaged and has been closed.

                2012-06-28 09:49:24.204 -0400    Error    681    fms10    Closing database

                "FileName" due to damaged temporary file...

                 

                In the future I am going only going to look directly at the log files!

                 

                Thanks again.

                 

                Tony

                • 5. Re: Batch ³Advanced Recover² [XPOST]
                  wimdecorte

                  Hi Tony,

                   

                  When you re-opened the file do the logs indicate that it went through a consistency check?

                   

                  The damage is with the temp files so carefully inspect the machine to see what could be touching the machine (virus scanning, indexing, OS-level shares,...)

                  • 6. Re: Batch ³Advanced Recover² [XPOST]
                    TonyWhite

                    Hi Wim,

                     

                    More good questions...thanks...back soon.

                     

                    In the meantime...

                     

                    Regarding running a FileMaker Batch ³Advanced Recover² as a diagnostic

                    tool...

                     

                    Spoke to FileMaker, Inc. Systems Engineer, Matthew O'Dell yesterday and he

                    reminded me that "Recover" is available as a script step (thanks!).

                     

                    Added the files to a folder and loaded the file names in a table.

                    Wrote a simple FileMaker loop script that processes the found set of files.

                    First batch done and reviewed.

                     

                    Some good news:

                    - the recover file is able to contain the output of multiple file recovers

                    meaning the batch script can be pure FileMaker.

                     

                    - the log file that is generated seems to have all the information that I

                    would get by running manually regarding questionable FileMaker objects. Lots

                    of data points to help evaluate the clients solution.

                     

                    - not every error message is a real concern

                    -- if there is a missing plug-in, recover will report an error when there

                    might not be one 

                     

                    -- there are some false positives for certain grouped objects

                     

                    -- a custom FileMaker database is really good at formatting log files (and

                    lots of other things too!)

                     

                    - COR020_Alexei Folger_File Maintenance and Recovery - Tools and Best

                    Practices (DevCon 2010) is a good read.

                     

                    Demystifying FileMaker Pro File Recovery (file_recovery_fm10_en) is up for a

                    re-read.

                     

                    More later.

                     

                    All the best,

                     

                     

                    Tony White

                    Tony White Designs, Inc.

                    Tel: 718-797-4175

                    tony_white@twdesigns.com

                    http://www.twdesigns.com

                    • 7. Re: Batch ³Advanced Recover² [XPOST]
                      TonyWhite

                      Hi Wim,

                       

                      I looked in the server events log file (imported into FileMaker, etc.) for

                      the entire day and I do not see any: Error Code 194: Database was

                      not closed properly, performing consistency check...

                       

                      My guess is that is good news.

                       

                      I have requested an audit from the client of the FileMaker Server machine to

                      see if anything was added or changed since it was set up, along with a best

                      practice reminder. Waiting to hear back...

                       

                      More soon.

                       

                      Thanks.

                       

                      Tony

                       

                       

                      http://help.filemaker.com/app/answers/detail/a_id/6550/~/filemaker-server-ev

                      ent-log-messages

                      Database was not closed properly, performing consistency check...

                      The specified file was in use when FileMaker Pro or FileMaker Server shut

                      down abnormally; therefore, FileMaker Server performs a file consistency

                      check to examine the file¹s blocks.