10 Replies Latest reply on Jan 15, 2016 9:46 AM by TSGal

    FM14 WebDirect repeated crashing of web server

    thornburg

      Product and version FileMaker Server 14 (all versions to date)

      OS and version Windows Server 2012 R2

      Browser and version Google Chrome latest version, Safari (iPad only), latest version


      Description

      I've already got a call into support on this, but it has (quite frustratingly) gone silent.

       

      Is anyone else still experiencing crashes of the web server in a two server configuration utilizing Web Direct?

       

      The FileMaker service on the web server will just stop, with a very simple error in the log (saying the service stopped).

       

      Our web server for this project is dedicated to this purpose, and does not run anything else except FileMaker Server.

       

      Restarting the service will bring it back for some time.  Restarting server seems to give more time, but I haven't gathered enough data to be sure (because I've just stuck with rebooting the whole server after deciding it seemed to work better).

       

      We've been installing updates about two weeks after they come out, except that this problem has gotten bad enough I'm now only waiting 24-48 hours.

       

      To be clear, the FileMaker Database Server keeps running, and shows nothing of use in any logs.  Access by the FileMaker Pro application still works fine.

       

      Web Direct connections get the message "Take note of any unsaved data. Then close the web browser window and contact the FileMaker Server administrator".


      How to replicate

      Run a two-server FMS 14 Web Direct deployment in an active environment

       

      Message was edited by: Robert Thornburg

        • 1. Re: FM14 WebDirect repeated crashing of web server
          TSGal

          thornburg:

           

          Thank you for your post.

           

          I found your case into Technical Support, and I have forwarded the case to my manager.  You will receive a response from a Support Technician.

           

          In the case, I see the crash log with a 701 error, which is generally caused by out of memory.  Did you ever have this running on a single-machine configuration?

           

          I noticed in the web Event log file, everything was working normally until September 29 when the Web Publishing Engine stopped, and then every few days after that it would stop.  I know it has been a couple of months, but do you recall if any updates to the system were made around that time?

           

          TSGal

          FileMaker, Inc.

          • 2. Re: FM14 WebDirect repeated crashing of web server
            thornburg

            We never tried this particular solution as a single server.  We have approximately 200 users (not all simultaneous, but usage can be heavy at times).

             

            We never saw any high memory utilization issues reported by Windows.  We increased the web server RAM to 12GB at the time the issue started to become a real problem.

             

            It didn't actually start on Sept 29... it just became much more frequent.  Previously, it happened approximately once every few months, and the error message was different (we no longer get that old error message).

             

            I checked with our Server admin back when I opened the ticket, and he said no changes were made in the few days before the problem intensified.

             

            I don't know if this is actually related or not, but I just noticed that our FM13 split server deployment is also stopped, but our usage of that is seasonal, and it's not active right now, so I have no idea how long it's been stopped, or if it has been a recurring problem.

            • 3. Re: FM14 WebDirect repeated crashing of web server
              TSGal

              thornburg:

               

              Thanks for the additional information.

               

              The web event log started in mid-June, so I did not see any error until beginning with Sept 29.

               

              Of the "approximately 200 users", has there been an increase in activity in the last few months?

               

              Are the "approximately 200 users" accessing the server 24 hours/day?  To be more specific, since nothing appears to have changed, I'm wondering if FileMaker Server somehow became damaged, as this would require an uninstall and reinstall of FileMaker Server.  In other words, is there a time when nobody is accessing the server?

               

              Since the issue is occurring with FileMaker Server 14, the FileMaker Server 13 deployment would be irrelevant since it is installed on its own server(s).  Check the FileMaker Server 13 logs to see how the service was stopped.

               

              TSGal

              FileMaker, Inc.

              • 4. Re: FM14 WebDirect repeated crashing of web server
                thornburg

                The prior crash was not logged (at all, that I could find -- neither system/application logs, nor FileMaker logs).  The prior crash did not cause the actual service to stop, it was still "running", just not working.

                 

                I believe the change in behavior was related to an FMS update, but I'm not 100% certain.

                 

                There has been a stepwise increase in usage since July, as we've been phasing in the system with various groups.

                 

                The largest increase in usage should have been at the beginning of September (technically the last few days of August).

                 

                We do have usage nearly 24 hours/day, but we can arrange a maintenance window.

                I already tried to launch a whole new VM, but I was having trouble getting it to talk to the DB server, and ran out of time in my maintenance window, and had to switch back.

                Is there a secret to switching Web servers?  I initially tried to switch it "live", and when that didn't work, I switched the DB server back to single server setup, and then switched it back to two server setup.  It still wouldn't talk to the new web server.

                 

                I restarted the old server, and it began communicating immediately.

                 

                Also, you're not asking for both servers to reinstalled, are you?  If so, is there a way to export/import the config?  If not, I fear the process would take quite a while.

                • 5. Re: FM14 WebDirect repeated crashing of web server
                  TSGal

                  thornburg:

                   

                  Thank you for the additional information.

                   

                  Sorry for any confusion.  Only reinstall FileMaker Server 14.0.4 on the relevant server since that is where the problems occurred.

                   

                  I have just been informed that one of our senior Support Technicians is now reassigned to your original case, and he has recently contacted you.  He will now work with you directly.

                   

                  TSGal

                  FileMaker, Inc.

                  • 6. Re: FM14 WebDirect repeated crashing of web server
                    thornburg

                    Thank you.  I'll work with the Tech and see if we get anywhere.

                    • 7. Re: FM14 WebDirect repeated crashing of web server
                      techt

                      Have you had any resolution to this issue? We recently updated and started using a two-server deployment, and the web services in particular stop for no reason and a 701 error is all we get. Both machines are dedicated to FMS, nothing else, are just a few months old with 16 GB of RAM each. Stats never show an out of memory condition, so I'm not sure where else to turn.

                       

                      Restarting the servers resolves the issue for a while, but not long enough. I did notice that depending on how the connection is closed might have some input to the crash, but I'm still looking for some time to further test that.

                       

                      Did not have any of these problems with our two server FMS13 deployment.


                      TT

                      • 8. Re: FM14 WebDirect repeated crashing of web server
                        thornburg

                        In short, no.

                         

                        The tech I'm working with has asked me to reinstall the server software (both halves), but I used our winter holiday maintenance window for something else, and haven't made another one long enough yet.

                         

                        Your symptoms sound almost exactly like ours.  We aren't running with quite as much memory... but they're both VMs, and we've never seen it go anywhere near full, so it didn't seem like there was any need to increase it.  I can bump it if someone tells me it's going to solve this problem.

                         

                        I've found myself wondering if it's not a *system* memory limit, but a FMS memory limit, and that there's no obvious way to increase the setting on Web Direct front end.

                         

                        It seems like either that or some kind of allocation bug.

                        • 9. Re: FM14 WebDirect repeated crashing of web server
                          techt

                          Thanks for the update. To clarify, we're running two Mac minis, these are not virtual machines. So, since we appear to experiencing the same issue on different hardware and setups, it seems that this is a much deeper issue beyond simple configuration.

                          • 10. Re: FM14 WebDirect repeated crashing of web server
                            TSGal

                            thornburg:

                             

                            I have sent a message to the Support Technician that was last handling your case.  If you don't hear anything by Monday, please let me know.

                             

                            TSGal

                            FileMaker, Inc.