5 Replies Latest reply on Mar 11, 2016 8:20 PM by osensnolf

    Memory ballooning on server

    johnnyb

      The memory in use on one of our FileMaker servers balloons to about 92% of the available RAM after running for a day or two, and I'd like to determine why, because I think it's affecting performance for the clients.

       

      The server runs FileMaker Server 13 (v5) on Windows Server 2008 R2 SP1. I've got the cache set to an ample 4GB now, although the value of this setting seems to have no material effect; whatever is soaking up the extra RAM simply soaks up whatever is available after I adjust the cache size.

       

      We have two FileMaker Server installations that are virtually identical but for the files they host, and a third as a WebDirect secondary machine. This is the only server exhibiting this behavior. In other words, on the other servers, usage hovers between 4 and 6 GB out of 16 regardless of the load.

       

      When the ballooning starts, memory usage increases steadily a few dozen megabytes at a time. It doesn't seem related to any scheduled scripts; we only have one, and the ballooning doesn't coincide with its execution. It also doesn't seem to correlate with the number of users or what they're doing, although I don't feel like I've ruled out an errant script or query someone might be running in some edge case I don't know about.

       

      I've been trying to examine the memory usage using things like perfmon, but the individual processes running on the machine all indicate normal usage, and nothing that adds up to 14-15 GB of allocated memory. The allocated memory doesn't seem to be associated with any particular process.

       

      So I can't figure out what's allocating all that memory or what to do about it, but I'd like to get it under control so that I can feel like FileMaker Server and the OS both have enough RAM to function without swapping.

       

      Any advice on analyzing this situation on the server?

       

      Any way I might try to narrow this down to a particular application or script?

       

      Edit: Forgot to link some (older) threads that seem to describe a similar situation:

      https://fmdev.filemaker.com/message/44672#44672

      https://fmdev.filemaker.com/message/101319#101319

      https://fmdev.filemaker.com/message/48098#48098

       

      Message was edited by: johnnyb

        • 1. Re: Memory ballooning on server
          wimdecorte

          johnnyb wrote:

           

           

          I've been trying to examine the memory usage using things like perfmon, but the individual processes running on the machine all indicate normal usage, and nothing that adds up to 14-15 GB of allocated memory. The allocated memory doesn't seem to be associated with any particular process.

           

           

          Something has to be consuming that memory so it has to be attached to some process.  I've never seen memory not linked to a process.  When you did perfmon, over how long a period did you collect stats?  Did  you include all processes on the server in the perfmon?

          • 2. Re: Memory ballooning on server
            johnnyb

            If it's linked to a process, I can't tell which. Working set, Private, Commit size, none of it adds up to the memory reported being in use.

             

            I've collected enough over 4 days just to see the paging file size and the amount of memory in use, so that I can watch it tick up over time.

             

            Let me set up a collector for the individual processes. What might be the best to capture? Working set? Private? Paged?

            • 3. Re: Memory ballooning on server
              NickLightbody

              Is it possible to swop the hosted files between the machines in order to identify whether the balloning is associated with the files or the machine?

               

              Cheers, Nick

              • 4. Re: Memory ballooning on server
                johnnyb

                Apparently, memory in use (and by this I mean physical RAM) doesn't have to be attached to any process, because the kernel's memory manager handles RAM as needed to satisfy virtual memory requirements — not just running process working set requirements.

                 

                Everything, it turns out, is allocated using virtual memory addresses. The size of the virtual memory table can and does far exceed the capacity of physical RAM. The memory manager, as is commonly known, will swap pages of memory out of RAM and onto disk when RAM becomes tight. But what's happening in my case is something of the opposite; unused RAM is being filled with pages swapped in from disk, particularly from commonly-accessed files. Which files? Files using what is called memory-mapped file access instead of stdio-style fopen and fclose.

                 

                The database server apparently uses memory-mapped file access to map the on-disk blocks of database files to blocks of virtual memory. This has the effect of treating the database file the same as a virtual memory page file (the backing store), just as if the file had been read into RAM and then swapped out to disk. With no further optimizations, this would bring gains in performance, as the server process need not perform costly I/O instructions, but can instead just access the file as if it were already in RAM, leaving the kernel's memory management to handle the I/O. But there are further optimizations. Specifically, the kernel caches frequently-used bits of virtual memory in physical RAM as needed, filling up as much as it can do so practically. The effect is the same as if FileMaker Server had read the entire database file into RAM. Except it doesn't have to, and it doesn't even know or care if that's been done, because it's all up to the kernel. FileMaker Server just says, "treat this file as if it were virtual memory swapped to disk," and the kernel says "Okay, here's a range of virtual memory addresses you can use to access it; I'll take it from here."

                 

                In my case, it looks like a large database file was being mapped into memory. The file was almost 12GB in size, more than enough to back-fill available RAM. As the file was used over time, each block the server needed to access would be "paged in" by the kernel from the disk. And then, because it was being used repeatedly, the kernel would decide just to keep it there.

                 

                No thrashing or anything was involved because it wasn't a low-RAM situation; the kernel could page the file back out to disk at any time with no appreciable impact to performance.

                 

                And this is all independent of FileMaker Server's RAM cache, which appears to be optimized for queries, summaries, and sorting.

                 

                So tracking down a container field which another developer had switched from external storage to internal storage cracked this case. By transferring all that data to secure external storage, the size of the database file plunged from 12GB or so down to under 800MB.

                 

                It doesn't seem to have dramatically improved performance, except perhaps for what could be attributable to the improved efficiency of accessing the data without the container field data padding everything up. But it has definitely solved the mystery of the bloat.

                 

                Further reading:

                 

                MSDN, "Managing Memory-Mapped Files"

                http://msdn.microsoft.com/en-us/library/ms810613.aspx

                 

                MSDN, "File Mapping"

                http://msdn.microsoft.com/en-us/library/windows/desktop/aa366556(v=vs.85).aspx

                 

                Ayende @ Rahien, "On Memory Mapped Files:

                http://ayende.com/blog/162791/on-memory-mapped-files

                 

                RAMMap (tool for examining memory-mapped files):

                http://technet.microsoft.com/en-us/sysinternals/ff700229.aspx

                • 5. Re: Memory ballooning on server
                  osensnolf

                  I know this is fairly old, but it came up when I searched for "filemaker using all memory".

                   

                  FileMaker is causing my Win7Pro machine use 15.5GB of my 16GB RAM.  This is a fresh Windows install with only 16 tasks running in the Process tab.  CPU usage is 0%.

                   

                  I do not remember this happening prior to upgrading to Windows7 as the machine I had been running Filemaker on was Windows Home Server with 8GB RAM.

                   

                  I do not feel that the performance is suffering but it is odd to see nearly 100% of my RAM used.