12 Replies Latest reply on Jul 9, 2017 9:44 AM by gofmp15

    Large number of external containers crashing FM server

    dbail22@comcast.net

      I am being told that our large number of external containers, 114,000, is crashing our hosting companies server.  The scanning file with a single container for 114,000 individual records has been recovered 4 times without finding a problem and has been rebuilt by importing to a clone once without incident. It took several hours remotely.  I have never read or heard of any maximum number of external files (most jpegs and some PDF's) or of a large number being a problem for Filemaker. Using FM14 from northern California, hosted in  southern California on FM 14.  Is there any reading or guidelines anywhere to resolve this kind of problem? Will I have to goto a webviewer instead of a container?

        • 1. Re: Large number of external containers crashing FM server
          bigtom

          I think you should be more clear about what the crash is. Is the entire OS crashing or just FMS? I am not aware of any external container record limit in FM. FM should handle this just fine. Look at any logs that are available for FMS and the OS to see why it is happening.

           

          It is possible that the server itself may be having an issue. Maybe it is crashing for lack of memory or other resources. Failing hardware?

           

          Are you on shared hosting?

          • 2. Re: Large number of external containers crashing FM server
            dbail22@comcast.net

            That's a problem. We are hosted by a company 600 miles away.  Only one file (the one containing one container and references all of the external images) becomes non-responsive.  The server is till running for the rest of our files and a few other companies that share this server.  The only fix (according to them) is to kick everyone off of the server and restart the server since they cannot close and restart this file.

            • 3. Re: Large number of external containers crashing FM server
              bigtom

              Are the files being indexed by the OS? Hard to say without some sort of log on activity or errors.

               

              Maybe TSGal has some insight.

               

              Might be time to move away from shared hosting. My only suggestion is to import into a fresh new file and not a Clone of a file that has an issue.

              • 4. Re: Large number of external containers crashing FM server
                gofmp15

                Currious Question:

                 

                What do you mean by 114,000 external containers?

                 

                Reading your next response seems to indicate that you mean one container field and 114,000 records with the field stored externally.

                 

                Any file can be closed on a server at any time without affecting the others. I have had shared hosted files where I could open and close my own files without bothering anyone else.

                 

                The fact that you have recovered the file a number of times and resorted to a clone is some clue that something is wrong with your file, your layout, your field, your host, etc.

                 

                Was the clone you used made from a recovered file rather than your clean golden master?

                • 5. Re: Large number of external containers crashing FM server
                  bigtom

                  I think it's obvious there are 114k records with one image each.

                  • 6. Re: Large number of external containers crashing FM server
                    philmodjunk

                    What kind of backups are set up for this file?

                    • 7. Re: Large number of external containers crashing FM server
                      wimdecorte

                      bigtom wrote:

                       

                      Are the files being indexed by the OS? Hard to say without some sort of log on activity or errors.

                       

                       

                       

                      Agreed; to begin troubleshooting we'd need a lot more information about what is going on.  Ask for the logs and start there.   We have customers with many more remote container data.  One of the things to watch out for is that path to the RC data is constructed so that not all files for all records end up in the same folder.  Most OSes don't do well with a massive number of files in one folder.  Many subfolders are fine.

                      • 8. Re: Large number of external containers crashing FM server
                        dbail22@comcast.net

                        Thank you all for the responses.  The hosting company seems hesitant to provide logs that have any meaning.  What I mean by that is for a specific time and date the problem occurred.  To be more specific there are 114,000 records each of which has a single scan of a medical, treatment, adoption record of a rescued animal.  The type of crash is this single file that stops responding, cannot be closed and can only be fixed by restarting the FM Server.  Any number of recover, rebuild functions have found no problems of any kind.  It seems the offending file can continue on for a few weeks without being recovered or rebuilt. This is the same amount of time it will work for a rebuilt file.  The hosting company says it will not matter if we create something like 24 files using the last name of a client, the fact is there are still 114,000 containers that filemaker must deal with.  Two choices seem likely right now. Change to a webviewer or change to another host and see if the problem persists since no one can find fault with the file.    Thanks again.

                        • 9. Re: Large number of external containers crashing FM server
                          dbail22@comcast.net

                          The sentence should read.  The hosting company says it will not matter if we create something like 24 individual directories using the last name of a client......

                          • 10. Re: Large number of external containers crashing FM server
                            bigtom

                            How many users are there and how often are the containers accessed?

                             

                            It is odd the hosting company will not give you at least a few log entries relating to your file in the server. I understand not providing all the server info.

                            • 11. Re: Large number of external containers crashing FM server
                              wimdecorte

                              dbail22@comcast.net wrote:

                              The hosting company says it will not matter if we create something like 24 folders using the last name of a client, the fact is there are still 114,000 containers that filemaker must deal with.

                               

                              That is an odd statement from them.  The number of container files doesn't really matter.  FMS does not 'deal' with them all the time, It's like the the number of records.  You can have millions of records and have a fast database, you can have 10,000 and have crappy performance.  It's all about how it is built and used.

                               

                              So to repeat bigtom's question: how are those containers used?

                              And how is the storage path for the container specified. Can you post a screenshot?

                              • 12. Re: Large number of external containers crashing FM server
                                gofmp15

                                Second Thoughts:

                                 

                                You say scan of a medical record which I interpret to mean scanning a printed piece of paper.

                                 

                                These scans begin about 300k and can grow much larger depending upon the dpi, content of the page such as color, photograph, etc.

                                 

                                300K images would produce a potential total of  114,000 x 300,000 or rounded 300,000,000,000 (bytes or bits). or 300,000 megabytes. (Did I do the math correctly) Using 600 dpi scans would make that 12,000,000 megabytes and 1200 dpi scans 50,000,000 megabytes.

                                 

                                So, the problem might not be FileMaker's ability to create a db array to hold the addresses of 114,000 records but your hosts ability to provide sufficient disk space to hold the files. Or your hosts drive beginning to encounter corruption and needing a good cleaning and updating. Just like we use software to delete unwanted files and make the files contiguous rather than in bits and pieces scattered about the drive. Really bad drives might chatter in the past as the heads bounce about picking up pieces here and there.

                                 

                                I have so much free space on my drive that this is not a problem. But your host may be having free space problems and with databases creating and deleting and .... file continuity may be a problem and this could lead to corruption.

                                 

                                Computer files aren't like animate objects such as glasses of water or marbles that have an identifiable chunk of matter. They are created from magnetic impulses stored here and there on hard drives (a generality) and then massaged by software to become an imaginary real thing.

                                 

                                A shared host that might have 50 clients each with 125 files each of which might have 500 tables with 600 fields and 1,000,000 records and 3 external container files each... well, that's a lot of stress on one copy of FileMaker Server. FileMaker seems to realize that by eliminating beginning with 15 the ability of a company to use one copy of FIleMaker server and offer shared hosting.

                                 

                                Or, to offer a suggestion: find another host.

                                 

                                Or consider whether or not you really need to host the file on the Internet. Do you need to provide anywhere access to these files or is it just in house employees working in an office. An in house server would be faster but would require a bit of hands on expertise.