7 Replies Latest reply on Nov 28, 2014 10:04 AM by jnouwen

    Container contents renamed and now "missing"

    jnouwen

      Title

      Container contents renamed and now "missing"

      Post

      FileMaker Server 13.0.4.400, Mac OS X 10.9.5 (13F34), Mid 2010 Mac Pro
      FileMaker Pro Advanced 13v4, Mac OS X 10.10.1 (14B25), Late 2013 Mac Pro
      Container field is set to external storage

      We have a table that we use as a document store. We're tracking 61,646 documents and I discovered some strangeness with 9 of them.

      We have a nightly script that builds a combined PDF of certain documents in our system. It does so by looping through the pertinent records in our document store table calling Export Field Contents. Every now and then, the script will get blocked by an error dialog stating "Container fields cannot be exported". This bypasses the Set Error Capture [On] I have at the top of the script (and it's also worded very generically and seemingly not related to that specific record).

      I finally managed to track down one of the records in our document store that was causing this. It's because the container field is showing the document is missing. Going to the location on disk where the document is actually stored, I see the PDF is indeed there. But the filename had "_1" inserted into it. So if the original file uploaded was named "146523.pdf" the file on disk is "146523_1.pdf". The two auto-generated thumbnails do not have the "_1". And presumably this is why FileMaker thinks the file is missing. Removing the "_1" from the file on disk and refreshing the record does NOT result in the container now "seeing" the file.

      By searching around on disk for other files with "_1" in the name, I found a few others. I even found one with "_2" added to the name that was exhibiting the same problem. I also found one with "_1" added to the name that DID still work.

      The documents in our system are added on a layout with an interactive container field that the users drag and drop the files into. Wanting to get our data back into place, I grabbed the files off the disk and reinserted them (via drag and drop) into the original FileMaker records. For the first record, after removing the "_1" from the filename I dragged the document into the container but FileMaker still showed the file as missing. The reuplaoded file, however, appeared on disk with the proper name and the thumbnails refreshed. The "_1" copy of the file remained in the folder.
      So I entered the container field, hit delete, and everything cleared out of the record as expected along with the reuploaded file and its thumbnails being removed from disk; the "_1" file remained (which makes sense, since FileMaker claimed to not know where it was). Manually deleted the "_1" file from disk, reuploaded the file again, and everything was fine.

      For the next record I figured I'd delete the contents of the container field first and was presented with the Yes/No dialog "The external file can not be deleted from disk. Do you want to delete the reference to the file anyway?". Clicking yes removed the two thumbnail images but left the "_1" file there. I deleted that manually and reuploaded the file and all was well. I followed this procedure to fix the remaning missing documents.

             
      • Has anyone seen this sort of behaviour before?
      •      
      • Why is FileMaker renaming the files on disk in the first place? It seems to be doing it after the thumbnails are generated.
      •      
      • How is it then losing the reference to the file it renamed?
      •      
      • Why did one of the files it renamed remain accessible in the container field?
      •      
      • Why did it only happen on 9 out of 61,646 documents?
      •      
      • How much wood could a woodchuck chuck?

        • 1. Re: Container contents renamed and now "missing"
          philmodjunk

          Am I correct that your container field specifies external storage? And that your file renaming is taking place inside the folder to which FileMaker copies the files after they are inserted via drag and drop?

          What exact options have you specified for your external storage location?

          • 2. Re: Container contents renamed and now "missing"
            jnouwen

            Correct, it's set for external open storage at a calculated path of subfolders grouped on the unique ID of each record (2,500 documents per top level subfolder, and each record's document and thumbnails are within its own folder). There are no validation or auto-enter options enabled for the field.

            And yes, the renaming appears to be taking place inside the folder. The thumbnails are named correctly and are still findable by FileMaker (as illustrated by them being deleted when I click through the dialog when attempting to delete the contents of the container from within FileMaker).

            • 3. Re: Container contents renamed and now "missing"
              davidanders

              Server is on a 2010 Mac Pro, the hard drive is the same age?
              Do you have a backup that you could test?

              Have you used Disk Utility to verify the hard drive? (tests, does not repair)

              • 4. Re: Container contents renamed and now "missing"
                jnouwen

                Hi David,

                Our data drive is a 1TB OWC Mercury Accelsior PCIe SSD card that we picked up earlier this year, so it's definitely not as old as the machine!

                We have not done a verify on the drive just yet, but will later on this afternoon. I'll have one of our ops people pull some backups as the "missing" files were added back in August and September.

                • 5. Re: Container contents renamed and now "missing"
                  chrishass

                  Could this be a duplicate file problem?

                  On my macs running OS X anytime a duplicate file is downloaded it is appended with _1.
                  For example - If I download my October bank statement from the bank's website it goes into the Download folder as 10312014stmt.pdf.  A few hours later when I have forgotten that I downloaded it and download again it is named 10312014stmt_1.pdf. If I forget again and download again to the Download folder it becomes 10312014stmt_2.pdf and so on.


                  Does the container field allow duplicates or is it set to require uniqe data?
                  Is it possible for the script to create a duplicate file?
                  Can two different users 'drag and drop' the same file?

                  • 6. Re: Container contents renamed and now "missing"
                    jnouwen

                    It's possibly a duplicate file problem, but if it is it's because FileMaker's doing it.

                    I created a new record in our document store and dropped a PDF into the container. It showed up on disk on the server along with the thumbnails. Dropping the same PDF into the container resulted in no change. Dropping a screenshot into the container removed the PDF and its thumbnails and put the screenshot on disk. Dropping the PDF back in removed the screenshot and put the PDF and two thumbnails on disk. I took the same PDF to a different machine and dropped it into the container. No change.

                    So that all seems to be doing the right thing. Except for the 9 times it didn't a few months ago.

                    The containers aren't set to require unique data, but the folder path I use in the Container Storage settings has the documents stored in their own folders so putting the same file into two different records wouldn't result in a collision.

                    And the script is a process that downloads a bunch of PDFs from the containers, combines them into a new file, and inserts that into a new record's container, so that wouldn't be it either.

                    • 7. Re: Container contents renamed and now "missing"
                      jnouwen

                      We've run a verify on the data drive and it came back clean.

                      We pulled the nightly backup from the day one of the missing files was added. File was added at 9am and the 7pm backup has the file with "_1" in the name, so it happened that same day.

                      I ran fseventer and dropped a PDF into the container of a new record in our data store, and there were no temp files created by FileMaker which were then moved to the final location; the PDF and its thumbnails were written directly to the final location (and those files then nearly immediately appeared in the progressive backups folder).