1 2 Previous Next 17 Replies Latest reply on Feb 11, 2013 7:11 AM by gdurniak

    Possible bug, wondering if anyone else can duplicate this...

    ClevelandConsulting

      All,

       

      We have discovered what may be a bug, but before I go off reporting it I am hoping someone else can see if they can reproduce this.

       

      It involves using a container field with global storage turned on, and the save as compacted feature.

       

      The way I believe it is reproducible is this:

       

      Create a new file, any name and make one field, a container field and bake it a global.

       

      Now add a file to the container field, any method, but make it a large-ish life, say a 10 meg file. It makes it easier to see the problem.

       

      Now if you close the file and check it's size you'll see it is about 12 meg or so, about what you'd expect, although FileMaker seems to bloat attachements in a container field, so open it up again and save a copy as compacted. This is where something odd I believe starts to happen. If you close the file you'll see it is a bit smaller, nearer the 10 meg you'd expect. But if you open the file and delete the contents of the container field it get odd. Closing the file will now show it's size to still be 10 meg or so, even with the container field empty...

       

      You can repeat this process as often as you want, adding a file, saving compacted, removing the file adding a new file, save as compacted and removing it and eventually you'll get, as we have had happen, a file that is a few hundred meg, with one empty field. Oddness abounds. Deleting the field does not seem to help, nor does recovering. However deleting the table does remove the hidden bloat.

       

      If anyone can help me confirm this I'd appreciate it. Oddly enough if the field is not set as a global this does not occur, everything behaves as expected.

       

      Thanks to anyone who can help me check the issue, we're avoiding global container fields for the moment until we can track it down.

       

      Court

       

      Court Bowman, CEO

      Cleveland Consulting, Inc.

       

      Visit us on the web at http://www.clevelandconsulting.com

        • 1. Re: Possible bug, wondering if anyone else can duplicate this...
          JeffYurka

          Court,

           

          I can confirm.

          MacBook Pro

          Mac OSX 10.8.2

          FileMaker 12.0 v3

           

          -Jeff

          • 2. Re: Possible bug, wondering if anyone else can duplicate this...
            BruceHerbach

            Court,

             

            I tried this. and had similar results.  Then I went back to the original file, and deleted the data in the container field and compacted it.  This time the result was 57 kbytes, 

            So I think it could be a bug of sorts.  Apparently compacting a compacted file is less effective then compacting the original file.

             

            My suggestion is go ahead and use the global container fields.  I don't think this is going to be to big of a deal.  Global container fields can't use external storage but then again you only have 1 field per table.  So it will only enlarge the file as much as the largest item you put in it. 

             

            Bruce

            • 3. Re: Possible bug, wondering if anyone else can duplicate this...
              kaostika

              I think it saves a checksum of some type.  To see if the file is changed etc.

               

              Oreste

              • 4. Re: Possible bug, wondering if anyone else can duplicate this...
                mbraendle

                Hmm. I think you should save as compacted AFTER you have deleted the container content ...

                • 5. Re: Possible bug, wondering if anyone else can duplicate this...
                  BruceHerbach

                  I tried that.  If I did it on the already compacted version,  it didn't make any difference in the size.  If I did it on the original it droped down to 57 kbytes in size.

                   

                  Bruce

                  • 6. Re: Possible bug, wondering if anyone else can duplicate this...
                    gdurniak

                    I first saw this problem in FileMaker 6 with a global container

                     

                    The only way to clear the bloat was to "save as clone"

                     

                    greg

                     

                    > You can repeat this process as often as you want, adding a file, saving compacted, removing the file adding a new file, save as compacted and removing it and eventually you'll get, as we have had happen, a file that is a few hundred meg

                    • 7. Re: Possible bug, wondering if anyone else can duplicate this...
                      BruceHerbach

                      Court,

                       

                      How often do you plan on compacting this file?  Will the file be on an FMS server or running on your local computer?

                       

                      Bruce

                      • 8. Re: Possible bug, wondering if anyone else can duplicate this...
                        ClevelandConsulting

                        All,

                         

                        OK, thanks everyone who helped confirm this.

                         

                        The issue for us is with the development of products. We often store plugins and such in global container fields for easy deployment. If you develop over time, replacing content with each version, you can end up with a file that is a few hundred meg pretty easily.

                         

                        For example with our newest product, CCPivot 2 we had a version break 200 meg, when we deleted and rebuilt that table it dropped to 25 meg...

                         

                        The save as clone is a nice solution but for use to solve one table it was easier to rebuild.

                         

                        We have switched to a single record table for this for now until it can get fixed.

                         

                        Court

                         

                        Court Bowman, CEO

                        Cleveland Consulting, Inc.

                         

                        Visit us on the web at http://www.clevelandconsulting.com

                        • 9. Re: Possible bug, wondering if anyone else can duplicate this...
                          mbraendle

                          Court,

                           

                          sorry to insist: Frankly, I don't see point where the bug should be. See my previous message.

                           

                          According to the FM help:

                          "Saving a compacted copy

                          When you save a compacted copy of a file, FileMaker Pro recreates the entire database, fitting as much data into each block as possible. This copies the logical structure, or arrangement, into the new file and reclaims unused space. (...)"

                           

                          When you delete a record or a field content, the occupied blocks in FileMaker file are not released, but only marked as deleted (which is what is called the "unused space" in the FM help). If one adds further content, this will not overwrite the "unused space", but be stored in new blocks. Save as compacted will remove the blocks that are marked as deleted and try to stuff as much of the active, undeleted content into as few blocks as possible.

                           

                          However, as you had described it in your original post, you did not save as compacted after you had deleted the container content, but before. Now go figure.

                           

                          It has always been like this since the early FileMaker days (at least FileMaker 3).

                          • 10. Re: Possible bug, wondering if anyone else can duplicate this...
                            gdurniak

                            it was a problem for me ages ago

                             

                            it's just amazing that odd behaviors like this go on for years and years

                             

                            greg

                             

                             

                            > sorry to insist: Frankly, I don't see point where the bug should be

                             

                            Message was edited by: gdurniak

                            • 11. Re: Possible bug, wondering if anyone else can duplicate this...
                              ClevelandConsulting

                              Martin,

                               

                              I understand your point, and for me, since I have a workaround, the issue is if not solved, at least resolved...

                               

                              To help others understand though, let me explain it from my point.

                               

                              The save as compacted is doing what it says, but only for the current contents, not for what seems to be "past" contents. Like I mentioned above. If I place a large file in the container field, save as compacted with the data in there, then remove the contents, the file size will never go down. A clone will remove the unnecessary data size, but noting else besides deleting the table.

                               

                              For an exercise there is a file here:

                               

                              https://dl.dropbox.com/u/5388355/SizeTest.fmp12.zip

                               

                              for you to play with. You can see that it is (extracted) 400 mb. One table, one empty field no records. You can get rid of the "hidden" bloat only two ways. First, you can save a clone (Thanks to Greg!) or you can delete the table. Save as compacted will not do it once the data has been "buried" by the save as compacted step.

                               

                              Certainly not earth shattering since there are workarounds, but I feel it is undeniably a bug.

                               

                              Court

                               

                              Court Bowman, CEO

                              Cleveland Consulting, Inc.

                               

                              Visit us on the web at http://www.clevelandconsulting.com

                              • 12. Re: Possible bug, wondering if anyone else can duplicate this...
                                ch0c0halic

                                Actually I think FMP (the developers) may be a lot smarter than we give it credit for.

                                 

                                The image may not be stored in contiguous blocks. Could be there has to be a minimum number of free block changes to become 'meaningful' before he compacting changes the file structure. A single image removal may effect multiple blocks that also contain data for the records. There would be a formulae to determine number of resulting 'free' blocks and if it is below a minimum then compacting doesn't change the file. While the Compacting stores 'as much data as possible' in the block it does NOT fill it up. There is always a little free space in every block. So an image spread out over many blocks may not change any one block below the minimum that flags it for rebuilding.

                                 

                                Also, FMP keeps an internal image table of all internally stored images. For example if you add an image to a layout then FMP only keeps one copy and when you use the same image elsewhere it reuses the same image. Even if you import the same image again FMP knows it and will not store two copies. Caveat: if you edit the image before the second import then FMP sees it as a new image and will store the new image.

                                 

                                So another possibility (just thinking out loud here): The image stored in the internal table may not be deleted immediately. If FMP has a 'timer' on the image, maybe it assumes it could be reused within a "short" (?) time, then it wouldn't be removed (yet).

                                • 13. Re: Possible bug, wondering if anyone else can duplicate this...
                                  gdurniak

                                  Very interesting theories.

                                   

                                  It's been around for so long, there just might be a reasonable explanation somewhere

                                   

                                  Since Recover looks for "stranded" ( un-used ) image objects, you would think Recover might see this, but it doesn't

                                   

                                  greg

                                   

                                   

                                  >Could be there has to be a minimum number of free block changes to become 'meaningful' before he compacting changes the file structure.

                                   

                                  If FMP has a 'timer' on the image, maybe it assumes it could be reused within a "short" (?) time, then it wouldn't be removed (yet).

                                  • 14. Re: Possible bug, wondering if anyone else can duplicate this...
                                    ClevelandConsulting

                                    All,

                                     

                                    For what it's worth on the two points, the problem was identified for us from a file that grew over the course of several months, at least partially debunking the timer theory.

                                     

                                    And for us, recover did not free the space up.

                                     

                                     

                                    Court

                                     

                                    Court Bowman, CEO

                                    Cleveland Consulting, Inc.

                                     

                                    Visit us on the web at http://www.clevelandconsulting.com

                                    1 2 Previous Next