7 Replies Latest reply on Feb 8, 2012 5:56 AM by ChrisVanBuren

    Corruption - Is Recovered File Safe to Use

    ChrisVanBuren

      Hi All,


       

      Thanks in advance for any help that can be offered. Here’s the situation:

       

       

      -I have a perfectly good FileMaker file which is under constant development always hosted by FileMaker Server. I believe this file has never been crashed.

      -I was going to use this file as the basis of a new template for myself. Because of this I tested this file with Recover.

      -The file failed Recover even though the file works perfectly.

      -My last good backup without the corruption is Nov 2, 2011. There has been many days of development since then.

      -Recover says that I should NOT use the file going forward.

      -Recover log report:

       

      8496  Balanced out begin and end folder markers

       

       

      -Clone fails recover with the same issue.

      -Compacted copy fails recover with the same issue

      -Deleting all scripts gives an error message “This script cannot be found or has been deleted” There is a folder called “Common” which also remains (cannot be deleted) so I believe the problem is either a phantom script inside Common or a problem with Common itself.

      -The recovered file seems to work fine.

      -The recovered file can be recovered itself with no problem.

      -All scripts can be deleted from the recovered file (so problem appears to be fixed by recover).

       

       

       

       

      FMDiff Check

      Running check described at http://fmdiff.com/fm/filecorruption.html yields:

       

       

      Test 1 - vs compacted copy

       

       

      Errors (indicates some kind of damage)

       

      No apparent errors detected.

      Block 41 orphaned.

      Warnings

       

      First data block is not block 2.

      No apparent errors detected.

       

       

       

       

       

       

       

       

       

      Test 2 - vs recovered file

       

       

      Errors (indicates some kind of damage)

       

      No apparent errors detected.

      Block 41 orphaned.

      Warnings

       

      File was recovered 1 times on Mac OS (Intel).

      No apparent errors detected.

      Overview

      [–] Deleted

      [~] Modified

      [+] Added

      Base Tables

       

      1

      Scripts

       

      1

      Graphics

       

      1

       

      Base Table ^

      New: Cliverly.20120113.0 Recovered.fp7

      Old: Cliverly.20120113.0.fp7

      [+] 32798

      Recovered Library

      --

       

      Table Occurrences ^

      New: Cliverly.20120113.0 Recovered.fp7

      Old: Cliverly.20120113.0.fp7

      [+] 13631615

      Recovered Library

      --

      Fields ^

      New: Cliverly.20120113.0 Recovered.fp7

      Old: Cliverly.20120113.0.fp7

      [+] 1

      Recovered Blob

      --

      [+] 2

      From Table

      --

      Scripts ^

      New: Cliverly.20120113.0 Recovered.fp7

      Old: Cliverly.20120113.0.fp7

      [+] 33011

      --

      --

      Graphics ^

      New: Cliverly.20120113.0 Recovered.fp7

      Old: Cliverly.20120113.0.fp7

      [~] 15

      --

      --

       

      [4 bytes]

      [4 bytes]

      modificationId

      45

      44

       

       

       

      Based on what Winfried Huslick (at http://fmdiff.com/fm/filecorruption.html) says I guess those are actually passes.

       

       

       

       

      QUESTION

       

      Can I just use the recovered file? The FM UI on recover says no but the log file says “Balanced out begin and end folder markers” which says there was a minor problem and recover fixed it for me. So my guess is that I the recovered file is safe to use. Love to have other opinions.

       

       

       

      Lessons

       

      -You cannot trust any FileMaker file to be free of corruption. You must check every FileMaker file for corruption after every x hours of development. The value of x depends on how careful you want to be.

       

      -A simple check with Recover is a good idea but I full check with FMDiff would appear to be a better bet (http://fmdiff.com/fm/filecorruption.html). Again it depends on how careful you want to be.


       

      Argh. I hate corruption but I let it bite me and I did know better. When’s the last time you checked your working perfectly files for corruption?

       

       

      Cheers,

       

       

      Chris Van Buren

      ---------------------------------------------------------

      e: chris@zeropointdata.com

      t: +44 (0)1291 689966

      www.zeropointdata.com

       

      Abbey Lodge, Tintern, CHEPSTOW, NP16 6SF, UK

      ---------------------------------------------------------

        • 1. Re: Corruption - Is Recovered File Safe to Use
          RayCologon

          ChrisVanBuren wrote:

          QUESTION

          Can I just use the recovered file?  The FM UI on recover says no but the log file says “Balanced out begin and end folder markers” which says there was a minor problem and recover fixed it for me.  So my guess is that I the recovered file is safe to use.  Love to have other opinions.

           

          Hi Chris,

           

          The fact that you're asking this question indicates that you understand (or at least suspect) that there is a level of risk attached to using the recovered file. You're correct.

           

          The reason there is risk is that the possible forms of corruption (which might be described a random variance from valid file content), at least theoretically, approaches infinity, whereas the recover procedures are necessarily finite. So the recovered file passes a finite list of checks for validity/integrity, but by its nature the process is not exhaustive.

           

          Moreover, the Recover procedure will omit blocks that are suspect, to achieve a valid structure for the file - but it has no way to confirm that without the omitted content your file will still function as intended. So a file that passes integrity checks may not be fully operational in other respects.

           

          If you plan to invest a good deal more in the file in question, and/or if its reliability in production will be of paramount importance, you would be wise to go to lengths to ensure known provenance and to avoid depending on questionable code. I believe it's on this basis that the caution against using the recovered file appears in the message returned by the Recover procedure, and I agree with it.

           

          I dare say you stand to receive a range of replies with opinions all the way to reckless abandon, but in your situation, FWIW, I would be taking a cautious approach.

           

          Regards,

          Ray

          ------------------------------------------------

          R J Cologon, Ph.D.

          FileMaker Certified Developer

          Author, FileMaker Pro 10 Bible

          NightWing Enterprises, Melbourne, Australia

          http://www.nightwingenterprises.com

          ------------------------------------------------

          • 2. Re: Corruption - Is Recovered File Safe to Use

            Hi Chris,

             

            Thank you for also contacting me directly and pointing to this thread.

            Let me note that your test was done with an older version of FMDiff.

            This version issued a false positive about the first data block.

            Despite the fact that FMDiff 2.0 does even more error testing it is no substitute for a professional file check which we can perform on the actual file with our own tools.

             

            Unfortunately I don't even have the complete FMDiff report so I have to ask

            - how big is the file?

            - with what version was it created?

             

            From what you describe you could try on a Finder copy:

            - Open the file with version of FileMaker 9 or older

            - rename all scripts starting with "--"

            - open with FM 11 again and see whether the script issue is solved.

             

            Orphaned blocks often indicate the file is still open with FileMaker during the test with FMDiff.

             

             

            If this does not solve the problems then to really go to the roots of the problem (and possibly repair the file) I need the original file here.

            Recover seems to produce two Recovered Libraries and one Recovered Blob. This indicates some lose ends that may stem from unexpected closures of the file. To find out what caused this the file from Nov 2, 2011 is also needed for comparison.

             

            You may remove all records from the file for privacy or size reasons. Use a Finder copy NOT save as Copy (or Clone) and zip the files. If you can't send them by email, please contact me for other solutions to transfer the files.

             

            Kind regards,

            Winfried

            • 3. Re: Corruption - Is Recovered File Safe to Use
              Oliver_Reid

              If you wnat to make a table is not corrupt in sense that in can crash FM (or worse, FM Server) or function weirdly, rthaer than that some teh data may be wrong, exprot to XL and re-import.

               

              The XML import rejects any invalid chartcters and the import will stop.

               

              You can edit the XML in Text Wrangler and fix the corrupt values. Afaik this is the only way remove invalid characters in data fields. FM's own recover does not do that, I have found.

              • 4. Re: Corruption - Is Recovered File Safe to Use

                Oliver,

                 

                The problem here is NOT the data but the file structure. And by the way, there is no such thing as "invalid character" in FileMaker data. FileMaker is capable to work with ANY character without any problems. There may be UNDESIRED characters - please read <http://fmdiff.com/fm/cleantext.html> on that subject.

                 

                While you may copy and paste many elements in FileMaker there are others you can't. And there is a big problem with the element IDs, as pasting elements may use new IDs, hence breaking things.

                 

                All these "tricks" are just possible if a file is not really damaged, but more often than not you can't access elements or even the complete file. So while copy and paste may help in some cases, you can't be sure whether the file has other flaws or is missing essential elements that cause problems down the road.

                 

                If you got a problem, don't care what it is

                If you need a hand, I can assure you this

                I may help...

                 

                See <http://fmdiff.com/repairs.html>.

                 

                 

                Winfried

                • 5. Re: Corruption - Is Recovered File Safe to Use
                  Oliver_Reid

                  Well, I have had actual exprience with fields with illegal characters in data feilds that caused FMP to freeze when the record concerned was loaded.

                   

                  The XML export/reimport fixed the problem.

                   

                  My reasoning is of course empirical rather analytic, but I have done this more than once so the odds are agaimst it being a coincidence.

                  • 6. Re: Corruption - Is Recovered File Safe to Use
                    ChrisVanBuren

                    Some new information.  On FM 8.5 folders are represented by:

                     

                    --

                    <folder name>

                     

                    In the case here the -- cannot be deleted or renamed.  On FM11 though everything works fine.  I can rename the folder.  Checking the folder with recover (after renaming in 11) still gives the error.

                     

                    To be honest, I really think Recover is just fixing a simple problem.  If I could get information on or a FMI comment on the error message:

                     

                    8496            Balanced out begin and end folder markers

                     

                    that would be really useful.  Anybody know of any resource detailing recover log errors and/or any way to get FMI to comment?  THANKS. 

                    • 7. Re: Corruption - Is Recovered File Safe to Use
                      ChrisVanBuren

                      Just to close this thread off.  I have recovered the file and am using it.  Everything with it is fine so far.  Problem with scripts appears fixed in recovered file ( I can delete all the scripts which I cannot do in the corrupt file ).

                       

                      Cheers,

                       

                       

                      Chris Van Buren

                      ---------------------------------------------------------

                      e: chris@zeropointdata.com

                      t: +44 (0)1291 689966

                      www.zeropointdata.com

                       

                       

                      Abbey Lodge, Tintern, CHEPSTOW, NP16 6SF, UK

                      ---------------------------------------------------------