8 Replies Latest reply on Aug 17, 2017 12:00 PM by Vincent_L

    Compressed container issue : functions applies to the compressed data and not to the original file

    VincentL

      Summary

      Compressed container issue : functions applies to the compressed data and not to the original file

      Product

      FileMaker Pro

      Version

      13.0v3

      Operating system version

      Mac OS X 10.9.5

      Description of the issue

      Compressed container Attributes and Base64Encode, applies to the compressed data and not to the original file.

      MD5 and Base64decode wil apply to the compressed data rather than on the original file if you insert a file using the compression feature

      I made a sample solution :

      https://www.dropbox.com/s/n4ql12sp67bc389/Compressed_container_issue.zip?dl=0

      The exact same file Apple.txt (a dump of Apple's webpages), is put in 2 container fields : Without compression on left, With compression (insert file compress check box checked) on the Right.

      MD5 is different, though it is exactly the same file. That's because, functions that applies to containers seem to apply on the compressed data, rather on the original file.
      Base64Encode also applies to the compressed data and not the original file.

      That seems to be an undocumented bug, because the compression feature should be transparent. That may lead to unwanted duplicate files (because MD5 is ≠) even depending on how the file was inserted.
      Also, that prevents usefull base64 operations.
      NoWorakaround, because no way to uncompress file internally before function calls

      Steps to reproduce the problem

      Insert a file using the compression feature.
      MD5 function and Base64Decode will be different from the same file inserted in a container without compress checkbox checked.

      Expected result

      MD5 and Base64 decode should produce the same result as if the container was inserted without compression.
      The filemaker compression feature seems to be transparent, but while it should be, it's not, because containers function applies to the unexposed compressed data rather than of the original file.
      Those function calls should trigger on the fly file decompression if file is compressed.

      Actual result

      Different MD5 / Base64Decode with compressed container vs same file stored as uncompressed container

      Workaround

      No workaround

        • 1. Re: Compressed container issue : functions applies to the compressed data and not to the original file
          TSGal

          Vincent L:

          Thank you for your post.

          I have forwarded your post and sample file to our Development and Testing departments for review.  When I receive any feedback, I will let you know.

          TSGal
          FileMaker, Inc.

          • 2. Re: Compressed container issue : functions applies to the compressed data and not to the original file
            TSGal

            Vincent L:

            According to Development and Testing, this is not considered an issue.  Some users want MD5 of the compressed file.  Suggestions include performing the MD5 on the uncompressed file and then throwing away the storage of the uncompressed data.  Or, have the calculation work on another Container where the file is saved as a reference, which would also give you the desired results without enlarging the file size much.

            I would recommend entering this suggestion into the Feature Requests web form at:

            http://www.filemaker.com/company/contact/feature_request.html

            The entries into this web form populate a database file that is hosted and monitored by Product Management and Development.  All entries are read, discussed and considered for possible implementation in a future release.  Although I could easily copy your post and paste it into the web form, there are a few contact questions asked on the form that only you can answer.

            TSGal
            FileMaker, Inc.

            • 3. Re: Compressed container issue : functions applies to the compressed data and not to the original file
              VincentL

              I disagree for several reasons :

              1. If it's a feature it should have been documented, which is not, at the very least documentation should be modified

              2. If it's a feature then we need to have a reliable way to tell wether or not if it's compressed. Filesize differences alone is not reliable (cause sometimes compressed can be > than uncompressed).

              3. MD5 is not the main problem, Base64encode is. With compressed containers, the base64 output is a compressed archive in an unknown filemaker format that you can't do much with it. So if you plan to use Base64encode you can say goodbye to compressed containers, and as dev can change mind as needs evolved, you can end up with thousands of file stored compressed for years, that you won't be able to use later with base64decode.

              So, If some user rely on something undocumented (the curent behavior), for some reasons, at the very least we should have container functions that applies on the compressed data or the non compressed one at developper will. But, modifying the functions and have the dev to think about it, and do some testing in his code to determine ≠ behavior according to the compressed state or not, seems a lot of work both for you and for us, for a feature (the compression) which should be completely transparent.

              Since the export container function, always output the uncompressed file (which is good), at leaf all output function (like base64 decode) should work by default upon the uncompressed file.

               

               

               

               

              • 4. Re: Compressed container issue : functions applies to the compressed data and not to the original file
                VincentL

                 Suggestions include performing the MD5 on the uncompressed file and then throwing away the storage of the uncompressed data.  Or, have the calculation work on another Container where the file is saved as a reference, which would also give you the desired results without enlarging the file size much.

                Both suggestions doesn't work container that are already in filemaker or requires a complex export / reimport process which is slow and not live, and doesn't work in server side scripts

                • 5. Re: Compressed container issue : functions applies to the compressed data and not to the original file
                  TSGal

                  Vincent L:

                  As documented, Base64Encode is to return the contents of the specified Container field as text in Base64 format, and Base64Decode returns Container content from text encoded in Base64 format.  As it stands now, FileMaker Pro will not uncompress a compressed file for these functions.  That is why I recommended entering this as a Feature Request.

                  The manager of Product Documentation has been notified about these functions giving different results for files inserted into Container fields with and without the Compress option.

                  Please also enter a suggestion into the Feature Requests web form to display if a file is compressed in a Container field.

                  TSGal
                  FileMaker, Inc.

                  • 6. Re: Compressed container issue : functions applies to the compressed data and not to the original file
                    VincentL

                     

                    As documented, Base64Encode is to return the contents of the specified Container field as text in Base64 format, and Base64Decode returns Container content from text encoded in Base64 format.  

                    See, that's where the crux of the problem is, for users (end solution user AND us, FMP dev), the contents of a container is what we've put inside it, which the file in its unarchived way. So the container contents is a file, not an archive of it.

                    I understand that for FMI engineers it's more straightfoward the way it is, but this it's not the useful way, and it's not at all the obvious way from users standpoint. 

                    The compressed file is (a cool, and much needed) mean of storage, it's not the contents, the uncompressed file is. Be sure to clear up that contents definition in updated doc (if it should sadly stay the way it is).

                    Moreover, beside that logic argument, the way it is as serious usage issues and lots of unnecessary work for devs :

                    MD5

                    You can't make sure you don't have duplicates of files because you can have them in both compressed and uncompressed form an their checksum will be different. So, juts like that, one usage of the MD5, de-duplication is gone.

                    That means that to use MD5 for deduplication, you must obsoletely forbid users to choose compression or not, so everywhere you've a container you have to script it's insertion. That's a lot of dev work that could be avoided.

                    Worse, if you have a lots of container that already exists, and have been inserted in some uncontrolled ways, since 12's release, you won't be able to know for sure, even in your updated solution that you don't have duplicates.

                    Finally, even if we would have a way, or we get a way, to know, reliably, if the container was compressed or not, it will be no use for MD5 in reduplication, because you won't be able to get the uncompressed MD5 from a compressed container to compare.

                    Base64Encode

                    If you insert a jpoeg file and choose compression, then it will be compressed (but be more heavy), and Base64decode will translate in a unvievwable file, do that you will not be able to embed it. So base64 embedding is now dead.

                    Furthermore, even if we get to know if that file is a compressed container, even though that's useful to avoid to display gibberish, then we're basically toasted, because we can't do anything about it to get the real base 64 which "depicts" the real file.

                    So that current implementation, even though makes sense from an engineer standpoint, destroys 2 usefull usages (usage that may be part of the reasoning of those 2 functions existence). That low level thinking would be ok in C++ environment, but we're in Filemaker, a RAD, made for simplicity. 

                    Possibles ways to fix this 

                    - Container MD5, both compressed and uncompressed should be both computed and stored in filemaker when inserting or updating the container. Because it makes no sense from a performance standpoint to re-compute them on the fly. So for files, the GetContainerAttributeMD5 should be read and not computed (maybe that's already the case today, I don't know).
                    Of course, getting the MD5 on non files containers, like fields, on the fly computing should stay. I use it heavily

                    - a new attribute should be added to the GetContainerAttribute function : MD5_Compressed. It will return the compressed MD5 that have stored when the file was inserted, updated.

                    - a inserted_method attribute that would tell if file was inserted with compression on or off (faster than comparing the 2 MD5/MD5_compressed)

                    - To avoid previous code breaking,  Base64EncodeUncompresed function should be created, it will uncompress on the file and create the Base64Encode from the uncompressed file. That function should work the same on compressed or uncompressed containers, so it will save the dev to have to test for compression or not, because otherwise every dev in the world will have to do this tests and that will be moe code and less efficient. Maybe it should be called Base64EncodeContents.

                    Other container needed features

                    Exporting and inserting containers need to be server side compatible. Containers are just files, and there's no reasons server side scripting couldn't do it. And that would be very useful (for instance mails with attachments)

                     

                    • 7. Re: Compressed container issue : functions applies to the compressed data and not to the original file
                      TSGal

                      Vincent L:

                      Thank you for the additional comments.

                      Although I have attached your comments to the original issue, be sure to also these suggestions into the Feature Requests Web Form, as it will be visible by Product Management and key people in Development.

                      TSGal
                      FileMaker, Inc.