13 Replies Latest reply on Sep 14, 2011 8:48 AM by harlowtech

    FMSA 11 is taking all available memory

    NWbutcher

      Title

      FMSA 11 is taking all available memory

      Your post

      I just moved the majority of my db's to a new FMSA 11 server running IWP from a FMSA 8 server that died.  I noticed I wasn't getting the performance out of the new server that I was expecting, and was quite a bit more sluggish than the old one.  After looking around I found that it was using all of the memory the server had, will processor utilization was in the low single digits.

      Server:

      64 bit Windows Server 2008 r2

      Xeon Processor 8-cores

      12 GB of memory

      FMSA 11.0.1.95

      The server is about a year old, and had the new Network OS installed cleanly.  It runs nothing else besides Symantec Endpoint Protection.  In my opinion the server is slight overkill, but who am I to complain.

      I do have a few very large files (in excess of 10 GB, much is due to a large amount of embedded PDF's), but rarely have more than 3 clients connected.  I did not have this problem on the old server, which only had 2 GB of memory.  The memory usage stays pegged out even if there are no client connections, or if I have closed all of the databases.  The only way to get the memory usage down is to either restart the machine, or to stop/restart the filemaker service.  Memory usage will climb slowly as it is running, or seems to peg out if the server runs a schedule like a backup or verification.

        • 1. Re: FMSA 11 is taking all available memory
          philmodjunk

          Are you using JDBC? If so, this bug may explain the issue:   FileMaker Server

          This is one of many acknowledged bugs that can be found in the Known Bug List here in the Report an Issue section of the forum.

          It can also be downloaded as a database file from:   http://www.4shared.com/file/8orL8apk/FMP_Bugs.html

          • 2. Re: FMSA 11 is taking all available memory
            NWbutcher

            ODBC/JDBC is not enabled.

            PHP was enabled, but I just turned it off to see how that effects the memory.

            • 3. Re: FMSA 11 is taking all available memory

              NWbutcher:

              Thanks for posting!

              Do you have any plugins enabled at all? If you do, would testing with these disabled be a possibility?

              Let's try checking the files for corruption. To do this, bring a copy of the files (you can use a back up) to a local machine. Open FileMaker Pro and navigate to File -> Recover. Then, select your file and choose "Check Consistency". Repeat this process for each database and let us know the results.

              We could also try and separate the back ups into groups and monitor the memory usage. Depending on the results of each group, we can separate the back up schedules further to narrow down the culprit. So if you have 20 files, create 4 separate schedules with 5 different files each. If we see a spike in memory without releasing for one group and not the others, we can narrow our focus to those files and create individual schedules for each to pinpoint the problem file, if there is one.

              TSBear

              FileMaker, Inc.

              • 4. Re: FMSA 11 is taking all available memory
                NWbutcher

                No Plugins enabled.  It's actually a rather simple instance of the server.

                I pulled copies of each of the databases and ran the consistency check on each one.  No errors were found.  I then stared testing each database on a test schedule for backups.  All of them used quite a bit of memory, and all but one would release the memory, once the backup was completed.  So I took a copy of that database from the backup, and did a full recovery on it.  It said it recovered it, with no errors.  I then replaced the file that was active.

                Unofortunately, it didn't fix the problem.  At least the file in question is an old file used for archival purposes, and we don't need to have it active all of them time.  Now, how do I fix it?

                Ideas?

                • 5. Re: FMSA 11 is taking all available memory

                  NWbutcher:

                  Would you be able to take that file down completely for a period to see if the issue is reproducible without it? If it isn't, we can focus our attention on that database a bit more confidently.

                  TSBear

                  FileMaker, Inc.

                  • 6. Re: FMSA 11 is taking all available memory
                    NWbutcher

                    The suspected offending file has been removed.  I will setup some backups for this evening and see if we can break it without the file loaded, or being backed up.

                    Thanks TSBear

                    • 7. Re: FMSA 11 is taking all available memory
                      NWbutcher

                      OK, so I haven't check the server in a few days.  The file hasn't been loaded, or backed up for 5 days.  Have had a couple back ups and such happen without it.  When I logged on it was back to full memory utilization, so the file in questions wan't the only issue, if it was at all.  At this point I will turn off all backups for a week (acceptable risk for me) and see if the problem disappears.  Then I will slowly add files to the backup schedule to see if the problem is solved.

                      Another thing I haven't mentioned before is the clients connection.  I still have a large number of FMP 8 clients that connect.  Do you think that could have anything to do with it?

                      • 8. Re: FMSA 11 is taking all available memory

                        NWbutcher:

                        That sounds like a good next step. Would we also be able to disable the Symantec Endpoint Protection service?

                        After re-reading your initial post, I should also mention that we do have a v2 update for FileMaker Server 11 Advanced. The release notes don't indicate a fix for anything similar to what you're experiencing but it's worth updating regardless. You can download this update here:

                        http://help.filemaker.com/app/answers/detail/a_id/7724

                        Regarding the FileMaker Pro 8 clients, I don't believe they could play a role in this issue but it's certainly something we'll keep in mind.

                        TSBear

                        FileMaker, Inc.

                        • 9. Re: FMSA 11 is taking all available memory
                          NWbutcher

                          TSBear

                          I updated the server to the newest update late last week (11.0v3).  I noticed the first database point in the more info says addresses issue where database(s) are near or larger than available memory.  I was hoping that this would help the situation, but it hasn't.  Just to reiterate I have a year old Windows server 2008 r2 64-bit (no SP update), dual quad core Xeon with 12 GB or ram.

                          The Server will grap 98+% or available memory any time it backs up or verifies any of my larger databases.  I currently have  a 17 GB and a 15.5 GB databases.  I also have a 7 GB db, but haven't tested it.  The server will not release the memory once the back up or verification is done.  The only way to get it to release the memory is to stop the FM server, through the FM server console, or obviously restarting the machine.  I am reuluctant to disable the Symantec, since we have a very public network.  I will have to schedule some time to disconnect the server from the network (off hours) to do it though.

                          I can recreate the problem on demand, by doing an immediate backup of either of those databases through the console.

                          NWButcher

                          • 10. Re: FMSA 11 is taking all available memory

                            NWbutcher:

                            Can you test with clones of those two databases? Also, how large are the clones of your files? If they're a reasonable size, would we be able to bring them in house for further testing?

                            TSBear
                            FileMaker, Inc. 

                            • 11. Re: FMSA 11 is taking all available memory
                              NWbutcher

                              TSBear

                              I need to correct myself.  The update to 11.0v3 has fixed most of the issue.  After backups, it is no longer holding all available memory, but....  A scheduled verification that includes a db that is larger than the installed memory on the server still fails to release the memory once the schedule is done.

                              I took a copy of my main database (16 GB) with most of it due to pdf's and word docs in container fields, and scheduled a verification only.  Memory usage climbed to the point of consuming all available memory during the verification, and failed to go down once the schedule was done, (waiting over an hour after completion of the schedule).

                              While memory usage was still max'd out (99.8%), I created a schedule to back up the same file, which included the verification option.  Memory usage dipped in about the middle of the process, but quickly max'd out again.  I'm assuming it was during the switch from backup to verification.  Once the schedule was completed, it released all of the memory, returning to a nomiinal usage of between 4 and 10%.

                              So, I took my test database and deleted about half of the records with container field occupied, bringing the overall size of the database down to approx 8 GB (server has 12GB of memory installed), and re-ran the two schedules to see what happens.

                              During the verification only schedule, memory usage increased by 8GB, and it was not released once the schedule was complete.  So, even though the file is not in excess of the total or available memory on the server, it still does not release it once the verification schedule is complete.

                              And with the backup including verification once again saw a dip in the middle of the process, with the release of the memory being used, once the schedule was completed.

                              So, I am operating normally, as long as I disable the verification only schedules.

                              Clones of these databases are tiny, (1-2MB).  I have not problem sending you clones.

                              NWButcher

                              • 12. Re: FMSA 11 is taking all available memory
                                TSGal

                                NWbutcher:

                                Yes, I would like to get clones of the files.  Please check your Inbox at the top of this page for instructions where to send the files.

                                TSGal
                                FileMaker, Inc.

                                • 13. Re: FMSA 11 is taking all available memory
                                  harlowtech

                                  I am able to replicate this problem with all the memory being taken on the machine after running a scheduled verification on our servers running 11v3 on Windows 2008 R2 SP1 and called it in to Tech Support yesterday. The problem is most apparent when you are running large (>2GB) databases.

                                  We actually found it during our QA process which for a patch invovles crashing the server on purpose and seeing if just a Windows admin can reboot the server and have it come back in a usable state, which as after the crash the files are verified so all the memory is taken and the system is essentially unusble.

                                  Daniel