1 2 Previous Next 19 Replies Latest reply on Oct 26, 2012 9:16 AM by ch0c0halic

    Microsoft's Volume Shadow Copy Service (VSS)

    DanielShanahan

      I'm working with a local IT firm who has set up a client's server. I installed FileMaker Server on the C drive and I've mentioned several times that this directory: Program FilesFileMakerFileMaker ServerDataDatabases needs to be outside of any backup routine.

       

      The backup routine is Microsoft's Volume Shadow Copy Service (VSS). I'm not familiar with it and in fact, setting up servers is not my strong suite (which is an understatement).

       

      The IT firm has described the process in this way:

       

      The imaging software produces two separate types of “backup.”

      1) There is a real-time system imaging/recovery, which creates a weekly live system base-image, with daily incremental image updates. These image files are of the entire system, in its current running state (at the time of the image), and are designed so that the entire server can be restored from a Bare-Metal recovery, or individual drive partitions can be recovered in the event of a system failure.


      2) There is also the conventional Archival Backup, used to recover individual files/folders from accidental deletion or unwanted modifications. For this backup configuration, I will have the VSS Shadow Copy feature (for SQL database backup) disabled. I will also exclude the live database directory from the job; but I will create a local folder on the root of the primary data drive (which I think is the C: drive on this server, but I will let you know if that is incorrect), which will be called “FilemakerDB.” This folder will be included in the conventional archival backup schema, so please set your SQL maintenance schema to store a copy of any flat-file backup and/or log files copies, in that directory.

       

       

      I'm still a bit worried that the live databases will be copied (or "imaged"). Is this just ignorance on my part (which I readily acknowledge)? Or, should I be concerned?

       

       

      Thanks.

        • 1. Re: Microsoft's Volume Shadow Copy Service (VSS)
          BobTheCodeBuilder

          Daniel,

           

          I'm a long-time systems administrator and have managed many Microsoft Windows Server installations - hundreds of servers over the years. It sounds to me that the IT firm is excluding the FileMaker DB from the VSS process and their archive backup routine. They are providing you a separate folder called FilemakerDB which you can use as a destination for any database backups you perform. The FilemakerDB folder is then backed up so as to protect those files and provide the capability for those files to be restored if needed. The FilemakerDB folder is a place where you can put your application's critical backup files. Make sense?

           

          Sounds like they are giving you what you want - not to have any automated backup process from the operating system or third-party software interfere with your FileMaker database.

           

          Bob

          • 2. Re: Microsoft's Volume Shadow Copy Service (VSS)
            DanielShanahan

            Thanks for your reply, Bob.  What mostly concerns me is this phrase in #1:

            These image files are of the entire system...

            Doesn't that mean that the hosted database directory is being imaged?  If so, is this ok?  In other words, is imaging different from backing up?

            • 3. Re: Microsoft's Volume Shadow Copy Service (VSS)
              BobTheCodeBuilder

              It sounds as if they will still be imaging the entire server including your database files. Typically, I find those applications harmless and not likely to destroy any files. However, the question is whether or not their application will be able to make a good copy of open database files to where they are useful if they need to be recovered. That I don't feel comfortable with. It may be why they are providing you a separate folder for you to deposit any backups you perform. My feeling is that the imaging software will not cause you any problems, but it might result in errors for them should the application they are using not be able to deal with FileMaker database files.

               

              If you can find out what imaging application they are using, that would be helpful. You could then contact that company directly regarding your concerns.

              • 4. Re: Microsoft's Volume Shadow Copy Service (VSS)
                DanielShanahan

                Thanks Robert.

                 

                I'm still a bit unsure.

                Robert Boice wrote:

                 

                It sounds as if they will still be imaging the entire server including your database files. Typically, I find those applications harmless and not likely to destroy any files.

                My understanding is that a backup software will corrupt the hosted data file.  I'm not sure if this is symantics or if there really is a difference between corrupting a file and destroying a file.  If they are different, then it seems that a file could be corrupted but not destroyed.

                 

                However, the question is whether or not their application will be able to make a good copy of open database files to where they are useful if they need to be recovered.

                There will be both scheduled and progressive backups and the disk imaging will presumably capture that.  Those are the files used for recovery.

                 

                If you can find out what imaging application they are using, that would be helpful. You could then contact that company directly regarding your concerns.

                 

                According to the IT firm:

                 

                The Dell AppAssure Backup & Recovery is what leverages the Microsoft VSS.

                 

                I appreciate your comments, Robert.  However, I'm still not convinced that imaging the hosted database file is ok.

                • 5. Re: Microsoft's Volume Shadow Copy Service (VSS)
                  BobTheCodeBuilder

                  I can understand your reluctance especially given that you are not familiar with Windows Server and VSS. I just don't see how an application reading anything related to FileMaker will cause any damage. Imaging is reading the data and copying it to another location. No writing to the actual files being read/copied/imaged. The concern should be when the copy is made. The destination (copy) may be unuseable because of the inability of the imaging software to handle open files. That is why a separate backup of your database should be performed and saved to the folder they provided.

                   

                  You can always test the situation using a sample database. Personally, VSS has never destroyed any files on my server by simply reading them.

                   

                  Good luck.

                  • 6. Re: Microsoft's Volume Shadow Copy Service (VSS)
                    GordonShewach

                    What I've always heard is that NO process should touch the live FileMaker database files (read or write). I believe there are verified examples of database corruption based on other processes touching live FileMaker files. It's not common, but it is possible and has happened if the circumstances are right. And with the consequences being this dire, I'd stick to your guns, Dan.

                     

                    I agree with you that #2 is OK, but #1 is not.

                     

                    Gordon Shewach

                    Desktop Services

                    Ann Arbor, MI

                    • 7. Re: Microsoft's Volume Shadow Copy Service (VSS)
                      BeatriceBeaubien

                      Hi Dan,

                       

                      There is no distinction between corruption and destroying of FileMaker databases, so I echo Gordon's warning. No process apart from FileMaker Server should touch the open databases. Another process reading the files will interfere with FMS and potentially cause problems. As Robert suggested, you could get a VSS image before you loaded any real databases (and rely on a backed up version of the sample database rather than the one VSS touched). But as soon as the server is in production, backups should only be looking at the FileMaker Server Backup folder(s). There is no need to backup anything else, provided you follow best practices and only have FMS running on that server.

                       

                      Best wishes,

                       

                      Beatrice Beaubien, PhD

                      i2eye, Toronto, Canada

                       

                      FileMaker Business Alliance

                      FileMaker 12 Certified Developer

                      Knowledge Translation Certified Professional

                       

                      (Jive To Address modified from fmi-1871805712-1mz-2-22yc@fmdev.filemaker.com to fmi-1871805712-1mz-2-22yc@filemaker-inc.hosted.jivesoftware.com. This is getting old fast.)

                       

                      On Oct 9, 2012, at 16:54, GordonShewach wrote

                       

                       

                      created by GordonShewach in Server and Server Advanced - View the full discussion

                      What I've always heard is that NO process should touch the live FileMaker database files (read or write). I believe there are verified examples of database corruption based on other processes touching live FileMaker files. It's not common, but it is possible and has happened if the circumstances are right. And with the consequences being this dire, I'd stick to your guns, Dan.

                       

                       

                      I agree with you that #2 is OK, but #1 is not.

                       

                       

                      Gordon Shewach

                       

                      Desktop Services

                       

                      Ann Arbor, MI

                       

                      • 8. Re: Microsoft's Volume Shadow Copy Service (VSS)
                        DanielShanahan

                        Thanks Robert, Gordon, and Beatrice.  I really appreciate everyone's input.  The basic question I have is this: With regards to FileMaker Server, how is disk imaging different from system backup? 

                         

                        I'd love to hear from a FileMaker System Engineer!  Short of that, I am going to err on the side of caution and instruct the IT firm NOT to include the hosted database directory in the imaging process.

                        • 9. Re: Microsoft's Volume Shadow Copy Service (VSS)
                          wimdecorte

                          Of the two backup types that they describe:

                           

                          - the VSS based whole-disk/system image: that would include the open hosted files and I can tell you from first hand experience that when you try to restore a full system image including the hosted files, those files will be a minimum improperly closed and at worst corrupted.  So the full system image is not a valid backup strategy when it comes to FM files.  It is still valid for other parts of the system but not for FM files.  There is one way that a VSS snapshot can be made to be valid: using the fmsadmin pause command just before the VSS snapshot and fmsadmin resume command right after.   Since no user interaction from the FM clients is possible during a pause, it will depend on the length of the backup to decide whether that is acceptable in your deployment.

                           

                          - archival backup: you will need to investigate whether or not that process is smart enough to NOT touch open files.  Best practice here is to explictely exclude the live folders from this backup process.

                          • 10. Re: Microsoft's Volume Shadow Copy Service (VSS)
                            ch0c0halic

                            Daniel may be asking for the long and short of it,

                             

                            Its the paradox of Schrödinger's cat. With FMS if you snapshot the live files, they could be dead (missing data or 'corrupted').

                             

                            FMS demands exclusive access to its hosted files. If at the time FMS wants to read/write to the file the snapshot is taking place then FMS will drop the file. No cache flush so data is lost! If the snapshot is good the files may be incomplete. There can still be data in the FMS cache that has not been flushed do disk. Once again missing data in the backup. Also, when a file is opened it has a bit set indicating it is 'in-use'. Any snapshot of a live file will have that bit set and will be considered as 'improperly closed' when opened.

                             

                            The live files must be paused either by the FMS backup schedule or via the CLI before they can be safely copied, by any process. Best practice is to never touch the live files. That includes but is not limited to: Virus checking, file indexing, copying, and snapshots.

                            • 11. Re: Microsoft's Volume Shadow Copy Service (VSS)
                              DanielShanahan

                              Thanks Wim and ch0c0halic for chiming in.  I appreciate the added information.

                               

                              I spoke with the tech from the IT firm today.  The conversation was cordial though there was a bit of frustration on both parts.  He was very patient and asked me if I was familiar with x, y, z.  It went something like this: "Let's see how I can explain this.  Are you familiar with x?"

                              "No."

                              "Y?"

                              "No."

                              "Z."

                              "Afraid not."

                               

                              I know he was frustrated and I really appreciate his patience!

                               

                              For my part, I was a bit frustrated because it seemed that he presumed to understand FileMaker Server based on various SQL database backup procedures (He mentioned Quickbooks, PeachTree, MySQL though I don't recall hearing SQL Server, Oracle, or DB2).

                               

                              In the end, he agreed not to run the imaging.  However, he will run it once to get a baseline.  I'm not sure what the point is of getting a baseline if we are not going to run it again but I didn't pursue that.  I closed the databases so the imaging will take place when they are closed.  I'll open them after the tech tells me the imaging baseline is completed.

                               

                              Let me just reiterate my appreciation to all who commented here - even Robert who is the one dissenting voice.  As a sole prop without vast experience in this area, I am grateful to hear from seasoned developers/technicians.

                               

                              Many thanks!

                              • 12. Re: Microsoft's Volume Shadow Copy Service (VSS)
                                ch0c0halic

                                The reason for the initial image is for a quick recovery from hardware failure. If there is an image then the entire system can be re-established very quickly. Then you move in the current backup files, replacing the 'initial" files. Restarting puts the FMS back on-line. Even if you've completely replaced the computer you have the same IP and network settings so its up and running on the network and everyone is happy.

                                 

                                If you have a multiple computer deployment you will have to redeploy to tell the computers to talk to each other.

                                 

                                BTW, FMS is not like any of those applications. FMS is a Service, no logged in user is required for it to operate.

                                Its much more like the Big Iron DBs (like Oracle) in that it has its own backup routines.

                                • 13. Re: Microsoft's Volume Shadow Copy Service (VSS)
                                  DanielShanahan

                                  Thanks, ch0c0halic.  That helps.

                                  • 14. Re: Microsoft's Volume Shadow Copy Service (VSS)
                                    DanielShanahan

                                    I thought this issue was resolved on my end, but alas, it is not.

                                     

                                    A few more communications emails (it seems that we aren't communicating very well) with the IT firm and I have a C drive with AppAssure on and ready to be imaged every hour and on the D drive AppAssure is off with only conventional backup on all directories, save the database directory.  The only problem is that I've installed FMS on the C drive!

                                     

                                    While not wanting to get into "He said, he said", I will quote the IT guy:

                                     

                                    In short; the database cannot reside on C:\  -- the database engine can, but not the database files.

                                     

                                    The simple reason for this is that the server is configured (by Dell) with a 40GB OS partition, and the D: drive is the Data repository and target installation directory.  By design, there is not sufficient room on the C: drive to accommodate program installation files or any data files.

                                     

                                    ServerPartition.png

                                     

                                    FileMaker Server is already on the C drive and the database (which is 864 MB) is there as well, though currently closed/un-hosted.

                                     

                                    My questions:

                                     

                                    1. Should I have requested a different partition spacing before this whole process began?
                                    2. Should I just un-install FMS from the C drive and re-install it on the D drive?
                                    3. Could I keep FMS on the C drive and map the backups to the D drive?
                                    4. Change careers and become a shoe saleman?
                                    1 2 Previous Next