6 Replies Latest reply on May 14, 2013 11:48 PM by iancook

    Web Publishing Engine Crashing

    woodstock_1

      Summary

      Web Publishing Engine Crashing

      Product

      FileMaker Server

      Version

      FileMaker Server Advanced 12.0.2.232 64-bit

      Operating system version

      Windows Server 2008r2 64-bit

      Description of the issue

      Have a FileMaker server with 70 FileMaker users and another 50 or so web (cwp/php) users (though usually less than 5 at one time).  Every couple weeks to a month FileMaker Web Publishing engine crashes and cannot restart unless I completely restart the entire server (have tried resetting IIS/then trying to turning on web publishing engine however that didn’t work).

      What I've looked at:
      -  Windows event viewer is not showing the crash or any errors at the time of the crash. There are no events going on at the time of the crash with FileMaker Server (or any other process)
      -  IIS log is showing normal activity. There are only a few php users in the system, not 20+ like screenshot below.

      Steps to reproduce the problem

      Just normal FileMaker activity.

      Expected result

      Not crashing

      Actual result

      Crashing

      Exact text of any error message(s) that appear

      After trying to startup Web publishing again: A dialog in the FileMaker Server Admin Console: "The Web Publishing Engine failed to start" (screenshot below)

      Configuration information

      Server Info: Dedicated (not virtual) Windows Server 2008r2 (IIS 7) 64 bit sp1, 32 GB Ram,  Raid 1+0 (and a separate set of drives for backups), Intel Xeon 2 processors
      FileMaker Server Advanced 12.0.2.232 64 bit (yes am a couple behind, haven’t seen anything in release notes about fixes to cwp/php), Single Machine Deployment. IWP/XML publishing is turned Off. PHP Publishing is enabled.

      Workaround

      Complete restart of server (not ideal)

      fullimage.png

        • 1. Re: Web Publishing Engine Crashing
          woodstock_1

               Any ideas? Plan for now is to put the web publishing engine on a seperate server.

          • 2. Re: Web Publishing Engine Crashing
            Fred(CH)

                 Hi woodstock,

                 Hard topic... From my own experience, unfortunately, CWP specialists are rather rare... I think you didn't have any answer shortly because to answer you surely, a person must have encountered the same problem than you. Until this person find you post, it will take time... 

                 At contrary, i suppose you also tried to search other posts over internet and, as me, you not found a lot of result... crying

                 I don't use CWP but IWP and generally, it is IWP that suffer of crashing problems wink so i empathize with you. I can also TRY to help by suggesting these :

                 • Verifing log activity, in particular the Web Publishing Engine (WEP). More info available here :

                  

                 • I know well it may be not relevant but if i were you, i began by installing last updates. Release note are not exhaustive (i hope).

                 • Try to inquire by the fact that the problem occurs almost regularly. You said two weeks. What's hapened ? Your config seem to be really strong so it is not a memory problem like generally when a problem occurs regularly. Just to see, what is your C: free disk space ? I know my question is strange but... Because i often encoutered on Windows servers the same curiosity : C:/ free disk space shortage. And even FileMaker Server is installed on a D: or on a E: volume, it use temp files on C: system disk. After days it could be relevant. Don't shout …blush

                 Bye, Fred (who tried)

            • 3. Re: Web Publishing Engine Crashing
              woodstock_1

                    

                   Thanks for your response Fred.

                   Had already looked at many of those things. The C volume has 50gb available, D (where database are)/E (where backups are)  have even more. The log files show that right before a crash the number of php connections goes way up however the IIS log shows that there isn’t much activity. Was thinking it could be a long running php/filemaker search but I am not finding any.

                   However... It looks like using the command line to reset the Web Publishing Engine will work where the server admin console will not.  This is nice in that I don’t need to kick everyone out, but need to test this more:

                   fmsadmin.exe RESTART WPE -u Admin -p adminpassword - y

                   My best guess for now is that there is a memory leak in WPE that occurs after a while of use. So have a script schedule resetting the WPE each night using the above comand.

                   Upgrading is an option but would take some time as would also have to upgrade super container, deal with any java issues and reconfigure settings. Am holding off on this for now.

                   Regards,

                   woodstock

              • 4. Re: Web Publishing Engine Crashing
                iancook

                     Hi Woodstock,

                     I'm getting WPE crashes too but I am not getting the phantom web users. It only happens to me during peak periods. It started since upgrading to FMS12 - all was fine on FMS11. To get things going again I use the same command line solution as you, though this does not always see me though the day. There is no one thing in the log that is a common thread causing the crashes.

                     For the record the machine was set up using a clean install of FMS 12.0.4.405 on a mac runing 10.8.3 - 2.8GHz Quad-Core Intel Xeon 8GB Ram Apache 2.2.22 and PHP 5.3.23

                     I'm not using IWP either, just CWP

                     Not sure whether to post a separate item for this one or whether it's a related issue?

                     Ian

                • 5. Re: Web Publishing Engine Crashing
                  woodstock_1

                       Hey iancook, 

                       That sounds pretty similar. Crashing is occuring at fairly busy times for me as well, certianly during business hours at the peak time of the day. Also had no problems with 11 server.   Curious how many web users it is showing in your Statistics log right before a crash.

                       Restarting the WPE at midnight each night doesn't seem to have helped. Crashed yesterday twice:(  The command line is able to fix a crash however, so far..

                       Hoping FileMaker Inc. chimes in on this thread. Concerns me that you are on the latest version and command line restarting is not fixing your issue!

                       woodstock

                  • 6. Re: Web Publishing Engine Crashing
                    iancook

                         I did a restart yesterday morning at 6am and by 11:06am it had crashed. Crashed again at 14:25.

                         I turned on Activity Monitor to watch the 9 fmserver processes. fmscwpc usually seems to run at about 250MB 'Real Memory' with 15 or 16 Threads. But when WPE crashed twice this shot up to 1.36GB with 29 threads and 1.37GB with 23 threads.

                         There are two java processes too. One with 54 Threads which usually runs around 200MB and another with about 30/40 threads that uses 150MB - when the crash happens the latter of these climbs to about 320MB 

                         All the other fmserver processes such as FM Web Publishing seem to be stable even when the WPE crash occurs.

                         The server has 8GB RAM - so I reduced the database server allocation in the admin console to 1GB. Hopefully this will allow fmscwpc to use more RAM - But if fmscwpc is leaking then it's just going to max out anyway?

                         Also after the second crash I pressed the Quit Process button on the Activity Monitor for fmscwpc before going to terminal to restart WPE. The Command line does restart the WPE OK - it just does not guarantee 24 hours of no crashes. 

                         Currently 7am here - the quietest time for our users and fmscwpc only using 64.2MB with 13 Threads. Yesterday after the restart it was at 215MB with 12 threads.

                         Yesterday we had max of 27 FM users at any one time.

                         I'm not sure how many web users were online at the time of the crash. In Filemaker stats we only get the one php user as web users do not have sperate web logons via Filemaker. But to give you some idea we have about 70+ staff and a large precentage of them use the intrannet + all the clients use it extranet too.

                         Initially I did reduce the number of crashes by disabling a logging script that added records monitoring web users. I need to re-introduce this at some point as it is a critical part of our system.

                         Not sure if any of this info is useful but hopefully it could allow someone with the same problem some comparisons. And I'm desperate and need to get to the bottom of this as I don't want to have to revert back to FM11. We did testing on 12 before the upgrade and 2 of my other clients had already upgraded so I thought we were OK but we did not simulate high web usage before committing.

                         I'm not sure how many people are sucessfully running CWP solutions on 12 with a decent amount of web users?