This is a problem that has plagued IWP for years. We are one and a half weeks ito an annual (salary review) process that runs for 3-4 weeks and has hundreds of managers interact with data in a very complex database, via IWP. And this issue has re-appeared with version 12.
Under load, we have found IWP fails up to six times a day in each of the the last 2-3 versions of FileMaker Server. Unfortunately, the issue remains in FMSA12.
However, we have just this week discovered rougly where the probem is - we just don't know how to fix it.
The problem is with the Web Publishing Engine - specifically fmswpe.exe *32 (a 32 bit Java component the WPE). It appears that there is some sort of a memory leak that eventuates in the Instant Web Publishing process terminating "abnormally" (Error 701), often followed by a FMS WARNING wpe1 that says "Some problems were detected in the WPE COMPONENT: COMPONENT IS NOT RESPONDING." Web Publishing then needs to be manually restarted through the FileMaker Server Admin Console.
When the Server (hardware) is entirely restarted, the application fmswpe.exe *32 [for some reason a 32bit application on a 64 bit machine] starts using around 68,000k. Over time, with successive users logging in and out of the system it grows (beyond 2.6GB in our case, on a 4GB machine) until such time as there is no more available or free memory - at which time the service crashes. With less RAM allocated to the machine, crashes are more frequent.
After a lot of googling (see for example, http://www.mulesoft.com/tomcat-memory), it would seem to have something to configuration of Tomcat. If it is not Tomcat itself, it is FileMakers software that interacts with Tomcat, not tidying up after itself - not cleaning out the garbage.
So the problem is related to aggregated usage (rather than load at any one time), and poor memory management.
We are desperate for a fix, as it is inconveniencing hundreds of our users and taking an inordinate amount of time for us to narrow down the issue.
For the time being, we are partially managing this problem by:
- Allocating more memory to the server - so it takes longer before it runs out.
- Logging onto the Server after hours, taking the service down and rebooting, to clear the memory entirely.
- Stopped incremental backs, as we suspect a sudden memory spike (requirement) by the processes associated with incremental back-ups tips the server over the edge when memory is very thin.
I hope this helps. More than anything, I hope we get a fix - this has been an issue for 5+ year now.
Thank you for the input, PGageler,
We'll follow your clues and see what happens!
One interesting fact is that although we had very similar problems last March with FMSA11, including the crashing and non-availability of the console, something that my colleague did, working on it for days, with reinstalls of both IIS and FMSA11, and tweaks in the OS, and tweaks in the FM solution that I did, caused the server to run seamlessly without any problems from March to now, when we upgraded to FMSA12. We are not able to identify what specifically, of all that was done, caused the server to run perfectly for four and a half months. The re-appearance of the problem immediately upon spending thousands of dollars to upgrade to 12 is regrettable to say the least and we are back on the path of last March...
By all accounts it is a common issue and it is surprising that FM Inc., has not addressed it in a more meaningful way, other than suggesting reinstalling everything all the time.
Let's hope we get a comment to this thread (and a fix, finally) from Filemaker Inc.
i've same problem whit OSX lion SVR 10.7.4 and macmini server i7 8 gb ram and FMSA 12
the iwp terminated abnormaly periodicaly...
whith FMSA 11 don't have any problem same hardware, i've try to re-install all , sys and fmsa but the problem is not resolved...
As it is now after hours for us in Australia, I have just stopped and restarted the FileMaker Server Service. Memory usage immediately dropped, such that fmswpe.exe *32 consumes just over 76,000k. Watching Windows Task Manager for the last hour though, memory usage is going up for fmswpe.exe *32 (by 3mb over 60 minutes), even when there have been no IWP users in that time. CWP is not in use (and disbled through the FM Server Console).
Yes Thinkworks - we have through the extremely painful reinstall routine, but it has never resolved any of our problems. Our FMSA12 install was on a new server build, and not over the top of any previous versions.
Our solution is used across 9 time zones, and during weekdays I can only work on it my 5AM-8AM... except when IWP crashes 6-8 times a workday and I'm there to bring it back up in seconds...
I am actually REALLY happy someone found out why IWP crashes with the 701 error (when there are no tabbed IWP layouts - the known issue).
I have identified the process as "FM Web Publishing" in MacOSX. It increases in real memory use indefinitely (or until it crashes) as an IWP client navigates through layouts/performs finds/enters data. It does not increase memory usage noticeably if there are not users logged in.
Can we get a Filemaker rep to acknowledge this issue and send it to the dev team? I'm sure many IWP developers would appreciate an update for this being in the works!
We spent the last three days monitoring (per advice of PGageler, above) and established that our machine does NOT behave in the same way. Memory usage does not seem to be creeping up.
We fully uninstalled and reinstalled FMSA12 (...very carefully) and the errors 701, where IWP abnormally quits, dropped from 8-a-day to 2-a-day on identical usage. The difference that a reinstall makes, plus the fact that the problem occurs randomly and under different conditions and manifestations on each machine (according to the forums posts I;ve seen), points to the supposition that the FMSA code is simply not robustly securing its control over usage of OS resources. Something that a re-issue, or a patch, after establishing the real problem in the code, may be able to fix.
I emailed tech support of Filemaker on this site with a copy of my first post in this thread. 24 hours later, Filemaker support replied with a polite email where they ask me to contact Filemaker support by phone. The only phones on this site are US toll-free numbers (as far as I can tell at the support page) and I cannot call them from overseas where I am most of the time (I, and the solution we are running, and the machine, are based in the US).
It seems that Error 701 is a random malfunction, a "disconnect" of resources usage in the functioning of IWP, that follows no discernible pattern, a flaw, of the Filemaker Inc., code in the FMSA program and nobody seems to have an answer. It also appears that several users are suffering from it.
We invite a tech to respond to this thread, please. Again.
I can make no promises that you'll ge that response, but I have sent a private message to one of the TS staff at FileMaker Inc. to look at this thread. It's Saturday here in the Pacfic Time zone and it won't be til Monday that someone at FM Inc. could see that message, but usually a pm from me get's a fairly quick response unless the recipient is out of the office for some reason.
I've never understood why the techs don't make it a practice to closely monitor this part of the forum. Issues here represent questions form individuals and organizations that have spent a lot of $$ on FileMaker products and the type of issues posted here often exceed what peer support can resolve.
Here's hoping someone will take the time to respond...
Thanks for doing this!
To those who have the memory leak problem with the Web Publishing process and are on a mac, I'm using an Applescript to restart the IWP at 2am everyday. Since my WPE usually went out a couple times a week, this should temporarily workaround the problem. Users with more frequent shut-downs can run it multiple times. Hopefully by the time I get more IWP users, Filemaker will have patched this.
First, open Applescript and copy/paste:
setcurrentTabtodo script "fmsadmin STOP WPE"
do script "y" in currentTab
do script "fmsadmin START WPE" incurrentTab
do script "exit" in currentTab
Save as a script to some directory.
Open Terminal. Terminal>Preferences>Settings>Shell and change "When the Shell exits" to "Close the window". Quit Terminal.
Open iCal. Create event at whatever time you'd like to restart the server (I do 2am or so). Set reapeating to everyday. Remember, users logged on will be kicked off and will have to log in again. There uncomitted data will also be lost - but it's not like it's not happening with this leak anyway ;)
Set alert on your iCal event to "Run a Script." Point to your new Applescript.
I don't know the Windows equivalent but the command line commands are the same. I also signed up for a free account at www.pingdom.com and set up a monitor pointed to the database directory page (404s when the WPE is not running). It checks it once a minute and upon 404, immediately emails firstname.lastname@example.org which texts me. I then use my iPhone to remote login and hit that button to restart. It's not perfect, but it's better than your users having to tell you about an outage. You can also tell them to wait 5 minutes and it should be back on ;)
This is how I'm dealing with this. Even if you don't have the memory leak problem or are not using Mac, you can still use pingdom.com to let you know within a minute that you need to restart your WPE.
Thanks Daniel Swett. Can you confirm that memory leaks seems to be the problem for you too?
I also noticed that in the topic (Just upgraded. Webpublishing engine keeps shutting down) you said that you had re-installed FMSA. Did that end up helping at all?
Yes, the process continues to use more and more real memory. Reinstalling FMSA12 seemed to fix the problem temporarily. I believe it lasted 3 weeks until it went back to "normal" - crashing every few days. I was using the solution with FMSA11 previously and did not have the IWP crashing problem, though.
Thank you Daniel Swett,
In fact, we did have a ping tool in place to let us know. The problem is that, across our 9 time zones we cannot guarantee that someone will be up every day all the time to bring it back up. In the last 8 days of operation it dropped 17 times and on 16 instances we brought it back up within 3 minutes, but once it remained down for 4 hours.
PG- We still have no evidence of memory leaks per your description, but that problem may yet appear too. I'll keep you posted.
We reinstalled again. So far no crashes, but it's only been a few hour.
This pattern seems to match our issue.
We are running FMSA 12 on Windows Server 2003 R2 Standard.
Filemaker please make an urgent fix, this issue is occuring on productions servers affecting countless users.
From our standpoint it is making us look like fools to our work providers, as they cannot access our system!