1 2 Previous Next 23 Replies Latest reply on Jun 5, 2012 11:45 PM by CarstenLevin

    FM11 Server Advanced Crashing -- any advice?

    svend

      Our FileMaker server has crashed three times so far, and it's not obvious what our next step should be.

       

      We're on a Windows Server 2008 R2 64-bit virtual machine, and the crash log says:

       

      Faulting application name: fmserver.exe, version: 11.0.1.86, time stamp: 0x4b6b3007

      Faulting module name: ntdll.dll, version: 6.1.7600.16915, time stamp: 0x4ec49d10

      Exception code: 0xc0000005

      Fault offset: 0x00066c9e

      Faulting process id: 0x1bc

      Faulting application start time: 0x01cd27840dc880fa

      Faulting application path: C:Program Files (x86)FileMakerFileMaker ServerDatabase Server mserver.exe

      Faulting module path: C:WindowsSysWOW64 tdll.dll

      Report Id: 64e26cb5-999a-11e1-9667-005056a10056

       

      It crashed Monday and Tuesday last week at around midday (when it would have seen a fair amount of activity), and tonight (Wednesday) at around quarter to six.

       

      In the two cases we've been able to look at close to crash time, the process monitor seems to indicate that there's a large spike in CPU activity around the time of the crash.

       

      The admin console seemed to be unaffected, and in the most recent crash, we were able to start the server process again from the console. Last Monday and this Wednesday, all the databases appeared to recover fully; during the crash last Tuesday, one frequently written-to file would not open, and we had to revert to an earlier version. (Because it is a file that basically keeps state/settings, rather than one that collects data, very little was lost.)

       

      The virtual machine is simulating a 4-processor 2.53GHz box, with 12 GB ram and a 100GB disk (which has 27.3 GB free). There are 85 db hosted, and the total size of the Databases folder is 6.69GB. We backup at 2am, and keep four days worth of backups locally (just in case the process that copies the backups to a filer fails over a long weekend; that's handled separately, at about 3am).

       

      We've turned off web publishing on this machine. We limit the number of connections to 75, and typically have 40 to 50 users during the day; we disconnect idle users after 3 hours, and allow external server accounts. Many of the databases have shadow tables connecting to tables on our MySQL servers.

       

      On the off-chance that it would help, last Tuesday we rebooted the machine, installing all outstanding Windows updates. We also rebuilt the 28 most active databases -- or rather, we cloned them, and copied their data into the clones -- in case there was some sort of weird corrupt file thing going on.

       

      Um -- is there anything else useful that I could say that would help diagnose the problem? Or somewhere else that I should be asking this question?

        • 1. Re: FM11 Server Advanced Crashing -- any advice?
          Stephen Huston

          You indicate that this is running on a virtual machine. At least twice in your log the "ntdll.dll" file is noted as an involved system file.

           

          Can you replace that DLL from another installation? DLL versions can cause failures when code from a faulty DLL is used, which is not necessarily very often.

           

          Any Windows gurus know anything about the ntdll.dll DLL file's interactivity with FM Server?

          • 2. Re: FM11 Server Advanced Crashing -- any advice?
            svend

            I'm not a Windows admin, but I'm told by our system team that this is a core Windows DLL, and replacing it would definitely involve a restart, and might break things unexpectedly.

             

            We're going to bump up the amount of memory the server has (which will also involve a restart, but is much less likely to randomly break stuff); and if that doesn't work, we'll go for your suggestion in a slightly more radical way, and create an entirely new virtual server from scratch, reinstalling both Windows and FileMaker Server.

             

            But... if anyone can suggest other places to look/what might be going wrong, so we can get an idea of whether all this work is likely to help, we'd very much appreciate it.

            • 3. Re: FM11 Server Advanced Crashing -- any advice?
              albi2103

              Hi,

               

              We experiment the same problem and I'm pretty sure that it is not coming from the dll.

               

              I send a forum question about that: Faulting application w3wp.exe

               

              If the Filemaker staff can be involved it will be nice!

              I talk about it to NH from Filemaker when he came to Montreal last week.

               

              Thank you

              Alain

              • 4. Re: FM11 Server Advanced Crashing -- any advice?
                svend

                Just to update this -- we upped the memory available to the virtual machine from 12GB to 14GB; we haven't had another crash so far, but it is early days yet.

                 

                (We also had a good look around for scripts that are run automatically, to see if we could reduce their impact -- for example, we keep a cache of a slow MySQL VIEW inside FileMaker, and we were able to reduce the number of records imported from that VIEW (by ignoring changes to the underlying tables that don't affect the data we care about).  Whether these sorts of changes had any affect is very difficult to say.)

                • 5. Re: FM11 Server Advanced Crashing -- any advice?
                  wimdecorte

                  svend wrote:

                   

                  Exception code: 0xc0000005

                   

                  While that's pretty generic, quite often it points to a memory problem.   Upgrading the RAM and reducing the impact on the memory by ways of the view could have solved it in a roundabout way.  I've been involved in a lot of FMS deployments and almost never see a crash like this.  Which leads me to believe that there is something with the underlying hardware

                   

                  Can you also verify that no OS updates or driver updates were installed prior to the crashing?

                  1 of 1 people found this helpful
                  • 6. Re: FM11 Server Advanced Crashing -- any advice?
                    svend

                    I can confirm that there were no OS updates prior to the first two crashes -- we've set up that Windows server so that you've got to actively ask for updates, rather than installing them automatically, and there were about ten updates pending when I rebooted.  I do not believe that there were any driver updates (unless they were part of the general Windows updates).

                    • 7. Re: FM11 Server Advanced Crashing -- any advice?
                      oditech

                      I know this sounds like a stretch but, if you are running vmware, did you add cpu's to your vm? This can cause random errors in your operating system. Just a thought.

                      • 8. Re: FM11 Server Advanced Crashing -- any advice?
                        svend

                        We do use VMWare, but there was no change to the hardware setup until we recently added RAM.  Unless you were talking about the other virtual servers on the same box?  I'm not sure about them...

                        • 9. Re: FM11 Server Advanced Crashing -- any advice?
                          ch0c0halic

                          svend may have RAMed a little too hard,

                           

                          Your mention of RAM upgrade reminded me of our problem.

                           

                          We had a bad time when we upgraded RAM on our VMware boxes too. The chips turned out to be flaky. Symptoms included: intermittent restart of the computer - with no OS log entries, FMP/S dying - with no OS/FMP/FMS log entry. We removed the RAM and the systems started behaving.

                          • 10. Re: FM11 Server Advanced Crashing -- any advice?
                            svend

                            Unfortunately, nope -- just confirmed with the Systems guys that we haven't changed the underlying physical hardware for a while.  They also point out that we run around 50 virtual machines on that hardware, and this is the only box that's had these issues.

                            • 11. Re: FM11 Server Advanced Crashing -- any advice?
                              oditech

                              Changing the number of CPUs on other VMs should not have any effect on your VM. Did they add physical ram to the server or just up your VMs ram? It could still be bad ram if it was just physically added and you could be addressing your VM on that space in ram.

                              • 12. Re: FM11 Server Advanced Crashing -- any advice?
                                svend

                                Nope, no physical change -- just upped the amount of ram that this VM grabs and holds onto.

                                • 13. Re: FM11 Server Advanced Crashing -- any advice?
                                  svend

                                  Okay -- it's been a while, and we haven't had a crash since we added the 2GB of memory to the virtual machine.  So... unless it was one of the users doing something odd that they've since stopped doing, this may indicate that insufficient memory was the problem.

                                   

                                  (Whether crapping out without warning is an acceptable response to insufficient memory is a question that I'll leave to one side.)

                                  • 14. Re: FM11 Server Advanced Crashing -- any advice?
                                    svend

                                    Except we had a crash on Monday.  While that's longer between crashes, this one actually corrupted a block in one of the actively used database, and we had to revert to a backup of the file (pulling in the new records from a recovered version of the crashed file). :/

                                     

                                    So -- we're going to see if something has gone wrong with the install.  We've spun up a completely new Windows VM, installed FileMaker Server Advanced 11, upgraded it to v11.03, and configured it identically.  We haven't switched over to this install yet, because other stuff has gotten bumped up my queue; but we'll see if it helps.

                                    1 2 Previous Next