1 2 Previous Next 17 Replies Latest reply on Feb 24, 2010 10:36 PM by RickWhitelaw

    When to "Recover" and "Is there a problem, really"

    RickWhitelaw

      Title

      When to "Recover" and "Is there a problem, really"

      Post

      <!--   StartFragment   -->

      My apologies if this post ends up being a bit of a polemic . . .  I’m actually writing it off line. Recently I had a problem involving a certain layout (as it turns out). Error 401 wasn’t being captured after a find without results and I couldn’t figure out why. Laretta posted a very fine example , to me, of extended error checking. Comment suggested the problem might be corruption. I saved a compacted copy . . . no change in behavior. Then I did a “recover” and viewed the recover log. It reported no changes in indexes. . . the data seemed fine. However the log reported two “items changed”. In the original file I found a field dependant on a non-existent custom function,corrected it, and ran Recover again. The log reported only one error or“change”. Finally I, in the original file, deleted the offending layout and created a new one, adding the required fields. The problem was solved.

       

                  Out of curiosity I recovered the last three “ordinal” backups of my solution. By ordinal I mean significant changes. On the first one different tax codes were introduced . . . another may have introduced a different use of drop down menus etc. I keep these on CD/DVD. I use Time Machine and Super Duper as well, so I feel reasonably secure with my backup strategy. All three recover logs of these three backups gave me the same message . . . there were problems. The thing is that, excepting the failure to capture the 401 error, there really weren’t anyproblems I could discern. The error capture failure had made me nervous though,so I found and solved the problem.

                  My first question . . . when would a person normally use “Recover File”? When a file doesn’t open? When data is scrambled? I had at least had a clue that something was amiss even though it wasn’t crucial . . . a window that should have closed didn’t and that tipped me off.

                  Second question . . . prevailing wisdom indicates a recovered file should never be used regardless of the message delivered. “Use a recent backup and update records in it from the Recovered File” is what I’ve heard. In the case I’ve described the data seemed intact. This one particular file has 107 tables (I’m a bit of a normalization freak). Does this suggest I should import records from the recovered file into a recent backup? For that many tables? And further, the problems encountered did not involve DATA corruption.

                  Thirdly,do developers routinely create scripted strategies to recover files and import data from the recovered file into a recent, and assumed stable, backup? 

                  Finally,how much attention should I be paying to this? The only problem I’ve ever encountered was one corrupt layout. As a result I’m aware that problems can occur. I’d like to be ready should a real disaster happen.

       

      Thanks for reading this,

       

      Rick.

       

       

      <!--   EndFragment   -->

        • 1. Re: When to "Recover" and "Is there a problem, really"
          philmodjunk
            

          Yes, file corruption is a challenge as it can lurk undetected in your file for quite some time. This can require working back through multiple back ups until you can find a "clean" copy. (For what its worth, I've encountered the same issue with MS Access and their recovery tool is practically a joke compared to Filemaker's. It really doesn't tell you anything unless recent updates to that product have improved it. You don't even know if a problem was detected with that application's "repair" utility. You know it repaired something only if you can see that the file itself now functions as expected.)

           

          Obviously, performing a recover on a file should be done if there are problems. I recommend also running recovers on all your files periodically. If it's a big file, you can pull the previous nights back up and run a recover on that copy overnight. Essentially, you're using Recover as a "cat scan" to see if any problems pop up.

           

          If problems are detected, the safest option is to test successive backups until one recovers cleanly, clone it and then import all the data from your most recent, recovered file into this clone. This can take a long time. It is possible to set up a script that imports from each table in turn and then updates next serial value settings on any auto-entered serial numbers. Such scripts usually are easy to set up, but can take a very long time to run if your file stores massive amounts of data. In those cases, you may have to import in two stages, first from a most recent back up file, then the most recent day's added/changed records. You can run the first stage overnight and then run the second import at close of business for the day. The trick here is to be able to identify any new or modified records. Auto-enter fields that record modification and creation time stamps can help you with this.

           

          You want to avoid using recovered files if at all possible as the recover may have changed your file in an unpredictable fashion which might cause a serious problem if you try to use it.

           

          Typical signs of a possibly corrupted file:

          • Filemaker crashes when you open or use the file.
          • Finds or sorts fail that previously worked correctly.
          • Layout elements fail to function correctly.
          • Scripts that have not been modified suddenly fail to work correctly.
          • Graphic objects don't display correctly as they did previously.

           

          In all of the above situations, it's possible that file corruption is not the cause so you have to investigate and/or run a recover to see what happens.

          • 2. Re: When to "Recover" and "Is there a problem, really"
            DLW-BPEX
              

            Phil,

            I often have this same general concern. An added one, though: What are the implications with runtime solutions? Often the user will not have native FM (that being much of the point of a runtime in the first place). Yet the developer may be disconnected from the product, except for essential support.

             

            Your thoughts?

            David 

            • 3. Re: When to "Recover" and "Is there a problem, really"
              philmodjunk
                

              My initial thought is to be greatful that I don't have a lot of Run Time solutions to be responsible for. :smileywink:

               

              Supporting the tech needs of your clients is a built in responsibility you accept when you choose to distribute run time solutions. You'll need to include support language encouraging regular backups at the very least.

               

              Here's an interesting "spin" on the discussion: I'm of the opinion that files undergoing development are much more likely to incur such problems while files that are just simply in use are less likely. That's assuming that neither database has suffered an uncontrolled shut down or crash of course. If true, then this would be less of an issue for your run time users.

               

              However, that's just a gut feeling on my part without any hard evidence to back it up. Anyone else have a take on that to offer?

              • 4. Re: When to "Recover" and "Is there a problem, really"
                RickWhitelaw
                  

                My sole FM solution has six files. It's large in terms of structure but is not truly large in terms of the amount of data stored. Even though I started building it (and using it) almost four years ago, I still tinker with it regularly, and on occasion make major changes. Two of the six files are quite complex and it's these two that undergo most of the changes. This morning I recovered all six files so I could read the logs. Four reported no problems. The two complex files reported problems . . . "items changed" I believe. In one case a "Recovered" table was created; no fields though, and no records recovered into it. All indexes fine, schema fine . . . "structure" was the area in question.

                 

                So . . . my evidence seems to support what Phil is saying. Files under development seem to run into more problems than those simply in use. The issue that leaves me with some doubt about all this is that I'm not having any problems whatsoever with these files. I've done a DDR, examined scripts, calculations and relationships carefully and uncovered nothing. At one point I was wondering if the structure problem had something to do with one of the external files and this was another reason to recover those files. No problems reported. I wonder if a dodgy (and unused) relationship existing between the files would generate this sort of report. I say "unused" because nowhere in any of the files is there an "unknown field" or "field missing" in any calculation or layout.

                 

                At this point I'm resigned to doing nothing but keeping my eyes open in case I DO encounter an issue.

                 

                RW 

                • 5. Re: When to "Recover" and "Is there a problem, really"
                  philmodjunk
                    

                  "The issue that leaves me with some doubt about all this is that I'm not having any problems whatsoever with these files."

                  That's a very key point. The problem is that you simply don't know. Is there really nothing serious wrong with our file or have we simply not done the one thing needed that triggers disaster? Since we have no way to tell, replace with back up remains the best, safest option even if nothing appears to be wrong with the file.

                   

                  Explaining this to a customer who wants to know why you want to bill major $$ to completely replace their damaged file when nothing appears to be wrong can be a major challenge sometimes.

                  • 6. Re: When to "Recover" and "Is there a problem, really"
                    DLW-BPEX
                      

                    Phil,

                    Again I agree. And your perspective on likelihood of failure is encouraging -- particularly with respect to runtimes, which by definition are single-user. (My own gut feeling is that a multi-user solution is more likely to have something go wrong than is a stable, single-user solution. But that also is with no hard evidence.)

                     

                    David 

                    • 7. Re: When to "Recover" and "Is there a problem, really"
                      RickWhitelaw
                        

                      Phil wrote:

                      "That's a very key point. The problem is that you simply don't know."

                       

                      Beyond that, I'm getting the same reports for every backup I've recovered. I'm wondering how far back I might have to go! 

                      • 8. Re: When to "Recover" and "Is there a problem, really"
                        philmodjunk
                           Sometimes you can get lucky and by studying the recover log to figure out what was changed, you can delete a layout or other element of your table and then get a copy that recovers "cleanly". Sometimes you can even narrow it down with successive recover tests to a single layout object. Then you can delete and replace the offending item to get a clean copy. That's not my first choice for fixing the file, but if you can't find a clean copy...
                        • 9. Re: When to "Recover" and "Is there a problem, really"
                          RickWhitelaw
                            

                          Thanks Phil,

                           

                          The logs are reporting, under "Structure" (not Schema) "1 item changed". It doesn't get more specific though.

                           

                          RW 

                          • 10. Re: When to "Recover" and "Is there a problem, really"
                            RickWhitelaw
                              

                            Interesting . . . I recovered a file and the log reported 3 modifications along with the usual warning not to use the file. I copied all the files onto a spare drive, opened the recovered file and found 1 table named "Recovered" and two fields, one in that table, named "Recover".  I deleted these two fields and one table. I then recovered THIS file and got the "All o.k." message at the end. I would have thought that deleting these would have made the file identical to the original, but apparently this isn't the case. I'm beginning to suspect the file was fine to begin with and that FM isn't "truly" deleting fields, tables or objects etc. as the developer deletes them in the normal process of building a solution, or at least is not releasing the page or blocks. The recovery process creates fields and tables it evaluates as missing and names them "Recover". This is nothing but speculation in the end since I know so little about this specifically.

                             

                            At this point, since I seem to have a perfectly functional file, and do heed the warnings to never use a recovered (or "RecoveredRecovered) file, I'm going to carry on with my originals.

                             

                            RW 

                            • 11. Re: When to "Recover" and "Is there a problem, really"
                              LaRetta_1
                                

                              You can come up with reasons to fit any belief if you search for them hard enough.  It is NEVER safe to keep working on a file after recover has been ran on it.  It has been repeated over and over by many top developers and even TSGal, although I'm still waiting to see if there is ever a response to Stella Luna's post New Recover process in vs. 10

                               

                              You can keep working in a file which seems fine and then unexpectedly, you will find out 6 months later something had trashed and you will have to go backwards in time anyway.  It is never worth the risk.   And copying layout objects or scripts from a recovered file can bring the corruption right into your good file.  A person can self-lie because they don't want to redo their work but better to redo part of their work from past few days (prior to last backup) than redo 6 months worth.  I know; I've self-lied a few times and paid a very high price.

                               

                              It seems you are stuck but I don't believe you are.  Pay the price now or pay a higher price later.

                              • 12. Re: When to "Recover" and "Is there a problem, really"
                                RickWhitelaw
                                  

                                LaRetta,

                                 

                                To be clear . . . I'm not using a recovered file. And I'm not copying any objects or elements from a recovered file. My original problem, as you know, was about a lack of error capturing, which was due to a corrupt layout. That particular problem has been sorted out on the original file and it checks out fine. One thing you said intrigues me: "It is NEVER safe to keep working on a file after recover has been ran on it." Are you saying that running a Recover on a file actually affects the original file? I understand about not using a recovered file, but surely continuing with the original is not any riskier after running Recover than it was before, no? I'm not saying it's ideal, but the error I've seen on the last 20 or so backups is identical, and in this case on this file the problem has been solved. Of course I intend to find a backup that checks out fine and update it with the (many) changes I've made since then . . . to be safe. Running Recover is actually new to me. Because it was suggested to me I tried it and it's led to much discussion! Frankly in the past if I've had a layout or any element behave strangely I've simply trashed it and rebuilt the element. 

                                 

                                Finally, I'm not trying to convince myself of anything. I'm simply trying to judge whether or not I'm over-reacting to the results of the Recover Logs, especially since I'm running them on files that are working perfectly. As well, I think my last post brings up an interesting point: When you recover, fix the "problem" (in the recovered file) and then recover the recovered (and "fixed") file, no problems are reported even though the "RecoveredRecovered" file is (or should be) effectively identical to the original file which DID report problems.

                                 

                                RW 

                                • 13. Re: When to "Recover" and "Is there a problem, really"
                                  LaRetta_1
                                    

                                  No, Rick, I wasn't saying YOU were trying to convince yourself.  But these discussions (between you and Phil) are played out in many Developer's minds every time they need to run recover because it is always painful to trash their work and go to backup plus need to migrate all their data.  We all wonder whether we should risk it.  Don't entertain the thoughts - just recover, migrate your data, rebuild what you've lost and move on!  FileMaker has never made it clear for us, as pointed out in the link above. And that is why many of us have repeatedly asked for clarification on whether it is ever safe to use a recovered file. 

                                   

                                  Recovery AGGRESSIVELY attempts to repair a file and it doesn't always display in the log. I have recovered files (in several versions but not yet in vs. 10) where custom functions are trashed, layouts and layout elements crash out but the logs indicate they are fine.  There have been thousands of discussions of this type through several forums over the years. And that is why we all keep asking!

                                   

                                  It was reported that vs. 10 has improved the Recovery process but what does the 'new and improved' recovery options mean in terms of trusting a Recovered file?  Go to Knowledge Base and search for 7026 http://filemaker.custhelp.com/cgi-bin/filemaker.cfg/php/enduser/std_alp.php?nav=support-kb

                                   

                                  It does not indicate that the recovery process itself has improved and can now be trusted and they say so themselves (again search Knowledge Base for 1580 for How To Organize The Recovery Process When Recovering Related Databases, Recover and Rebuild Item #4).

                                   

                                  The problems are two-fold: 1) FM contradicts itself about the safety of reusing a recovered file and 2) FM doesn't make the information IN YOUR FACE and when something is critically important, it should be.  The import bug is hurting customers every day by trashing their data and people use Recover and trust the dumb recover message and keep using the file, not really understanding that they are practicing unsafe sex by doing so.  The risk to use or don't use recovered files belongs to the customers (once they understand the risks) but making people aware of the seriousness of these issues falls completely on FileMaker. 
                                  • 14. Re: When to "Recover" and "Is there a problem, really"
                                    philmodjunk
                                      

                                    I agree that one should never use a recovered file. You simply have no way to know what changed. This has been a good thread and a useful one one to point people to if they have questions about File Corruption and the recover process.

                                     

                                    Just one point of clarification on my earlier posts: You can use the recover process as a crude "cat scan" of your file. Sometimes you can spot a line in the recover log where a specific layout was changed during the recover. I've found that sometimes, I can edit the original, unrecovered copy to delete this layout and then the file recovers cleanly. In some cases, that's a less drastic method of getting your file back up and sound than importing into a cloned back up.

                                     

                                    Sometimes, if a client hands you a damaged file and can't find any back ups, it may be your only option short of rebuilding the file.

                                    1 2 Previous Next