1 2 3 Previous Next 36 Replies Latest reply on May 4, 2016 1:04 AM by rrrichie

    Simple advice on flushing the cache, please

    chazzo

      Apologies for what might turn out to be an idiot question, but I seem to have a problem with automatic saves to disk, and I'd appreciate advice. This is with FileMaker Pro Advanced 14 running on a single Mac, and occasionally shared to another Mac.

       

      I have been running this database under FM 10 for years, right up to El Capitan. I know that's not supposed to be a good idea, but it has worked well for me and I cannot remember a single crash.

       

      Then the other day something bad happened – I think it was a crash on trying to close the file – and I lost several days' worth of data. The file is stored in Dropbox and backed up with Time Machine, so I didn't anticipate any problems recovering it. However, I was horrified to find that the latest disk version dated back to the last time the file was closed. I've become so used to FileMaker's rock-solid behaviour that I had left the file open for several days.

       

      I've always left the app to "save cache contents during idle time", and never taken any other steps to flush the cache explicitly. After this incident, however, I upgraded to FileMaker 14 and added a "flush cache" instruction to a script that I use frequently. I also tried changing the flush interval. But I still get the same behaviour: the disk file changes only when I close the database, not when I run the script.

       

      Have I completely misunderstood the way FileMaker works, and only been protected from disaster by the fact that it's very stable? If so, is there a way to update the disk file regularly without closing the database? If not, any ideas what odd things might be happening on my system?

       

      Thanks

      Charles

        • 1. Re: Simple advice on flushing the cache, please
          DavidJondreau

          I don't think the script step Refresh Window[] will do what you're trying to do here. I think those are two different "caches".

           

          You can set a backup to run at night using the OnTimer script rather than relying on an On Open script.

           

          But your "idle time" save cachee should be working, so I don't really know what the problem is. Was the crash caused by a remote user? Are you sure you're using the correct file (and not the backup)?

          • 2. Re: Simple advice on flushing the cache, please
            chazzo

            Thanks David. The crash was on my own machine, with no other users connected. After that, I tried my best to rebuild the file, though it wasn't obviously damaged, and FM did not report any errors or lost data during the rebuild.

             

            Yes, I could write a script to save a backup (and I was interested to read some people on this forum saying that backing up open files is a bad idea). But as you say, I was hoping that FM's built-in mechanisms would allow me to rely on my normal backup routine of Time Machine and Carbon Copy Cloner.

            • 3. Re: Simple advice on flushing the cache, please
              Mike_Mitchell

              chazzo wrote:

               

              The file is stored in Dropbox and backed up with Time Machine, so I didn't anticipate any problems recovering it...

               

               

              As you've discovered, this is a Really Bad Idea. Here's why:

               

              When you open a database on a remote drive (like Dropbox), you're becoming the host of the system. IOW, your FileMaker client becomes like a little mini-server. All traffic for that database has to pass through your client, and every time you save the database, it goes through your local temp file (which is not the same thing as the cache, although it performs a similar function).

               

              In addition, all communications, requests for data, record saves, etc., have to go "over the wire". That means any interruption of power or network, especially while you're in the process of updating a record, can - and often does - corrupt the database. In fact, it's the most frequent cause of file corruption I know of.

               

              Flushing the cache merely saves the data back to your temp file. The temp file isn't reconciled with the stored file until you close the database. This behavior is different than using FileMaker Server, where the server acts as the host.

               

              I would strongly recommend you abandon the practice of working on the Dropbox file and instead, either work with it locally and upload to Dropbox when you're done, or use one of the excellent hosting services available to put the file up on FileMaker Server (assuming you don't want to run a Server installation yourself).

               

              HTH

               

              Mike

              • 4. Re: Simple advice on flushing the cache, please
                Mike_Mitchell

                chazzo wrote:

                 

                The file is stored in Dropbox and backed up with Time Machine...

                 

                BTW - This is also a bad idea. You should never let anything, including backup software or virus checkers, to touch an open database. The database should be excluded from any Time Machine backups and backed up manually, when you don't have it open.

                 

                (Yet another advantage of using Server.)

                • 5. Re: Simple advice on flushing the cache, please
                  keywords

                  Re: "The file is stored in Dropbox and backed up with Time Machine..."

                   

                  Further to Mike's pont here, I may be wrong, but I don't believe Time Machine will touch any file while it is currently open, so any Time Machine backup will precede the current FM session—which, in your case, lasted some days. Open it, use it, close it is best practice unless the file is hosted on FMServer.

                  • 6. Re: Simple advice on flushing the cache, please
                    databuzz

                    I've looked into this a few years ago as I was seeing different results to what FileMaker Inc. advised about Time Machine backups and locally stored open files:

                     

                    FileMaker and Time Machine | FileMaker

                     

                    Even though it states "Time Machine ignores the open FileMaker Pro file " I found that not to be true in my tests. I opened some local files, did a Time Machine backup and found those same open files in the backup. I haven't run any tests since then but I would be wary of Time Machine running at the same time as you have locally stored open FileMaker files.

                    • 7. Re: Simple advice on flushing the cache, please
                      rrrichie

                      How is your dropbox setup?  When doing your cache tests, I would disable dropbox for the moment.  I assume you have dropbox installed and have a local copy on your machine?

                       

                      Filemaker creates temp files and synchronizes those with the "server".  The temp file only gets cleaned up when you close the file.  The idle cache is about RAM cache and that gets written to the temp file and then synced with the main file.

                       

                      Reading your problems, it sounds like the syncing is not working (maybe dropbox is preventing that somehow).

                      • 8. Re: Simple advice on flushing the cache, please
                        chazzo

                        Thanks everyone.

                        Mike_Mitchell wrote:

                         

                        Flushing the cache merely saves the data back to your temp file. The temp file isn't reconciled with the stored file until you close the database.

                         

                        That's really interesting, and quite different what you might guess from looking at the settings on the Preferences > Memory tab (I'd just assumed that the whole file was saved whenever the cache is updated). Is that mechanism of any help in recovering from a crash, do you think? In those circumstances I wouldn't have much faith in a temp file.

                         

                        So in other words nothing has changed on my system in terms of the way FileMaker behaves. Most of my backups will have been days old, but I didn't find that out because I never had to use them.

                         

                        databuzz wrote:

                        Even though [FileMaker Inc] states "Time Machine ignores the open FileMaker Pro file " I found that not to be true in my tests.

                         

                        Thanks for the link. Again, really interesting. I think I've got the message now about not backing up open files :-)

                         

                        I'm getting the sense that you professionals prefer FileMaker Server even for applications that are supposedly within the scope of FileMaker Pro's sharing abilities. It's disappointing to hear that, because for a tiny office like mine the ability to share files without the expense of Server is a killer feature.

                        • 9. Re: Simple advice on flushing the cache, please
                          chazzo

                          Sorry, rrichie, I missed your reply.

                          rrrichie wrote:

                           

                          How is your dropbox setup?  When doing your cache tests, I would disable dropbox for the moment.  I assume you have dropbox installed and have a local copy on your machine?

                           

                          Filemaker creates temp files and synchronizes those with the "server".  The temp file only gets cleaned up when you close the file.  The idle cache is about RAM cache and that gets written to the temp file and then synced with the main file.

                           

                          Reading your problems, it sounds like the syncing is not working (maybe dropbox is preventing that somehow).

                           

                          I was certainly wondering about how Dropbox actually works here. Basically I have a desktop Mac and a laptop, and – as you do – I have been relying on Dropbox to keep files in sync between them. So each time I open the database, I'm opening a local file that has been updated via Dropbox after I have closed the local copy on the other machine.

                           

                          In other words I'm never opening a file in the cloud, i.e. across the network. And AFAIK Dropbox will not back up a file until it changes, which according to Mike_Mitchell and others will only happen when it's closed. So according to that reasoning there shouldn't be a problem with Dropbox.

                           

                          All this seems excessively complicated in an age where people use Dropbox to sync machines all the time. I use it for my 1Password database, for TextExpander settings – all kinds of tasks that are specifically supported by app developers. It does seem a shame if FileMaker can't do the same.

                          • 10. Re: Simple advice on flushing the cache, please
                            rrrichie

                            In past, had a several clients working that way.  But they would close filemaker at the end of the day, and let the backup system do it's work. ZIP drives haha...  I used to make a startup database that opened all databases and created backups at the opening of the databases, as well as compact/cloned copies when closing the files. In those days people (and this was more common back then) people actually shut down their computers when going home.

                             

                            If you want a database available 24/7 and remote-access and your data is important... Server is the best option.  Next down is a dedicated machine with FileMaker Advanced on it and with FM Scripts/Applescript create a backup system.  You can script backup copies in a timed script for example. 

                             

                            In your case I would suggest creating some backup scripts yourself. Use the commit Records to ensure you get the latest records.  Don't know how big your database is, but have it run at lunch time :-)

                             

                            Happy Coding!

                            • 11. Re: Simple advice on flushing the cache, please
                              chazzo

                              Thanks rrichie. Sound advice there. I don't think I can shell out for FileMaker Server at this point, so I'll take your second option and do some scripting. This particular database is critical to my work but only 10 MB and a few thousand records, so taking time out for backups should not be an issue.

                              • 12. Re: Simple advice on flushing the cache, please
                                Mike_Mitchell

                                Don’t forget the option of commercial hosting. It’s cheap ($30 - $50 a month) and gives you all the benefits of Server without the pain of managing a server yourself.

                                • 13. Re: Simple advice on flushing the cache, please
                                  chazzo

                                  Thanks. That's beyond my budget – at least, I probably have other priorities for IT spending – but sounds like excellent value in the right circumstances. No license costs, and less worry about backups. Definitely something to think about in the future.

                                   

                                  Oh, and I live in the back of beyond, with less-than-wonderful internet access. I have done some maintenance on a FileMaker Server solution run by a customer in the USA (I'm in the UK), and it was rather slow. That might not be typical, but I think it would be an obstacle for me.

                                  • 14. Re: Simple advice on flushing the cache, please
                                    Mike_Mitchell

                                    There are some UK hosting companies, but bandwidth and latency definitely play into it. If your connection is ultra-slow, it might not be feasible without careful optimization of the solution.

                                     

                                    Still, it's better than losing data...  

                                    1 2 3 Previous Next