10 Replies Latest reply on Jul 20, 2010 9:50 AM by philmodjunk

    Records disappearing randomly FMP 11

    galumay

      Title

      Records disappearing randomly FMP 11

      Post

      Hi, One of my clients is using FMP 11 and on random occasions we are losing records, it occurs across a number of databases and it mainly appears to affect recently created records, although with over 6,000 records in the main database its hard to confirm.

      FMP is running on an X-Serve and the other clients, which are all iMacs running 10.5 or 10.6, access the database through a remote connection on the server. 

      The clients have lost faith in FMP, they want to me to revert to an earlier version of FMP, but I am very doubtful this will resolve the issue as I dont believe that the upgrade to 11.0 was specifically to blame.

      I have talked to FMP support in Australia and they sent me info about repairing and restoring databases, but I can only do the first two stages as the records have embedded images in them and its not feasible or possible to completely rebuild the databases with the image files. The two initial steps did not resolve the issue and basically support told me it is likely a corruption of the databases and there is nothing else that can be done to fix the issue!

      Currently I am frequently creating incremental backups that I save separately to ensure that in the event of further loss of records, not too much work is lost.

      As the database forms the archival database of 20 plus years of artwork from a major indigenous Art centre, they regard it as their crown jewels and they are less than impressed that it has become so unstable and unreliable.

      Has anyone else had experience with FMP 11 deleting recently created records and does anyone have any suggestions for preventing future losses?

      Cheers, Rick

      EDIT - I should add that they dont have a backup of the databases that predates the issue occuring.

        • 1. Re: Records disappearing randomly FMP 11
          conor

          We have been running FMP 11 on Windows for months without experiencing this kind of problem.

          Have you tested chained deletion via portals or other such potential side effect operations, was there any change in the OS etc.

          There is usually an explaination for this kind of behaviour, reverting back to the older version might be a good option until you can figure out what is happening.

          Better to regain the users confidence.

          • 2. Re: Records disappearing randomly FMP 11
            philmodjunk

            With regards to file corruption and rebuilding a file (Which may not be the cause of your issue here)...

            You may be able to rebuild the file's indexes and resolve the problem. If you save a clone of your file and import the data into this clone, it should rebuild all indexes during import and this might resolve the issue.

            When you ran a recover on the files, were any problems reported by the recover process?

            "as the records have embedded images in them and its not feasible or possible to completely rebuild the databases with the image files." How are these images embedded? If they are store by reference container fields, you can import these references into container field in the new file just like any other data.

            • 3. Re: Records disappearing randomly FMP 11
              galumay

              "When you ran a recover on the files, were any problems reported by the recover process?"

              No, it didnt report any issues.

              "How are these images embedded? If they are store by reference container fields, you can import these references into container field in the new file just like any other data."

              I am wary of attempting this, they are images in container fields on the data base records, but many of the original images no longer exist except within the records. Given the history with these databases its seems unlikely I would be able to rebuild with the images intact.


              "There is usually an explaination for this kind of behaviour, reverting back to the older version might be a good option until you can figure out what is happening."

              I am sure there is an explanation!! I just wish I knew what it was, I am reluctant to revert back to an older version because I fear it may add more issues rather than reduce them, and I would still have no confidence that we would not suffer further data loss in the future.

              I am more inclined to migrate all the data out of FMP and get something like an SQL database built from scratch at this stage.

              Its really not possible to continue on with the constant threat of data loss, so unless I can identify a cause and rectify it then I will have to look at more stable and secure alternatives.

              Unfortunately the lack of meaningful support from FileMaker limits the options, essentially they have told me that this happens 'sometimes' that data base corruption is usually the cause, and have offered no more that the standard PDF on data recovery. Given the absence of a 'clean' backup that we know is un-corrupted and the difficulties posed by the images in the container field that document was really no help.

              • 4. Re: Records disappearing randomly FMP 11
                conor

                Thats drastic!

                If you believe its a corruption issue and need to get the images, then you could create a script to extract all the images, and capture the image path for each record.

                Once you have extracted the images you can clean out the container fields, clone as per PhilModJunks suggestion and re import the image again using another script, this time you can consider using  a container reference rather than embedding the images, if you think it would help.

                • 5. Re: Records disappearing randomly FMP 11
                  galumay

                  Thanks Conor, I guess I could try doing all of that with another copy of the databases on one of my personal computers so that it was completely isolated and if successful then migrate it all back to the server.

                  Do you have any ideas where to start with a script to extract the images and image paths? I have not looked at how FMP handles images within the database.

                  I really appreciate you taking the time to assist me with this.

                  • 6. Re: Records disappearing randomly FMP 11
                    philmodjunk

                    I would try importing the images first before trying more labor intensive approaches. Since you are importing images into a new copy, if it doesn't work as it should, you have lost nothing as your current file is unchanged. If it works, you've saved yourself hours of work. I really doubt you'll have trouble importing the images if they are present in the original file to begin with.

                    If recovering a file doesn't result in a message telling you problems were found it is most likely that there were no problems with the file structure itself. It's possible that the recover simply didn't detect the problem--recover isn't perfect--but it's more likely that you may have a problem with an index and importing all your data into a clone of your file will rebuild all your indexes and this may very well correct the problems you are having.

                    It's also a good idea to play detective and try to identify ways that records or data may be deleted in unexpected ways:

                    1. Incorrectly implemented scripts with Go To Related Records can modify or delete large numbers of records in one swoop.
                    2. Delete options in relationships can delete related records when a record in a related table is deleted
                    3. Layout based script triggers can fire unexpectedly in the middle of another script's execution, such as when another script uses Go To Layout to switch to a layout where an OnLayoutLoad trigger is defined.

                     

                    There's no magic way to identify these causes, but if you analyze your design and ask your users questions, you might spot a possible cause for your disappearing records.

                    For more on the hazards of Go To Related Records:  The Complete Go To Related Record

                    • 7. Re: Records disappearing randomly FMP 11
                      galumay

                      Thanks Phil, I am perservering with the trouble shooting! I will investigate all your suggestions. 

                      Should I be running FMP Server on the server? Does it offer any real world advantages over the standard install for the type of use I have described?

                      • 8. Re: Records disappearing randomly FMP 11
                        philmodjunk

                        Server provides a number of advantages:

                        1. Better protection against possible file corruption if a client crashes
                        2. Automated Backups that can be performed without closing the file
                        3. Better accesse control
                        4. Support for large numbers of users
                        5. Faster response times

                        It's also much more expensive and requires a server with suitable hardware specs (You can find these in the knowledge base). So you get more but you also pay more.

                        • 9. Re: Records disappearing randomly FMP 11
                          nolak37

                          Does anyone know if this was ever resolved?

                          I have a very large DB holding a few hundres records and have run into a similar problem. we recently discovered that once a record has been created and a few days later the billing department gets arround to creating the bill for that record its no longer there.

                          This is causing many problems since we can not bill and this ends up in a big loss for the company. Fortunately we are keeping daily backups from where we cwn recove the record but having to do this on a weekly basis is just not practical.

                          • 10. Re: Records disappearing randomly FMP 11
                            philmodjunk

                            As galumay did, you'll need to do some careful analysis of your database design and function to check all possible causes. There are database features that if improperly implemented, can delete or hide records in unexpected ways. After you've ruled out those causes, you can consider whether a corrupted index or other file damage might be at fault... Don't forget to consider and investigate user error also.