We are currently doing a post mortem analysis of an issue that, while seeming to be (hopefully) resolved..leaves us NOT understanding what happened and why...
Here is what we know...
- FileMaker Pro clients running 14.latest
- FileMaker Server running 14.latest on 10.10.5 on an Xserve 3,1 with fast drives and plenty of RAM
- Everything running fast and stable for 1.75 years...
- 109 files, 21 GBs
2018-05-18 (Friday) users reported that things had been running slow for “about a week”.
The Help Requests from the users seemed to indicated that they might be seeing the issue reported here...
Upgraded from FM12 to FM16, Find in Progress... Delays
At the same time our FileMaker Server uptime monitoring systems reported that the backups were taking longer that expected.
We reached out to the customer and suggested a FileMaker Server restart.
2018-05-19 (Saturday) Over the weekend we had a look at the FileMaker Server, (but did not restart it). Verified that there was plenty of space on the drives and did not see anything obviously wrong.
2018-05-21 (Monday) We spoke to the customer on Monday and together decided to restart the FileMaker Server that night.
The FileMaker Server crashed that afternoon before we could restart it that night.
We then “warm” restarted the Xserve, started the FileMaker Server software, and then re-hosted files.
2018-05-22 (Tuesday) The server crashed again at approximately 4 p.m. (second time in two days)
2018-05-22 evening (after the users had gone home), I ran the following 3 tests (out of the 4 tests that we are aware of)...
(see link below)
1. Checked that the FMS 150 emails for the Backup Schedules with Verify were NOT reporting issues on the block level. OK there.
2. Ran a “diagnostic recovery” on the 109 files to check internals of the blocks
Having looked at many of diagnostic recovery logs over the years, I did not see anything that looked like a crashing issue.
(there where they usual false positives for some Custom Functions and one error message for every single Theme in every single file)
Recovering: theme 'com.filemaker.theme.classic' (1)
Recovering: theme 'com.filemaker.theme.custom.28FDBB55<obscured>' (2)
3. Ran an http://fmdiff.com/
No issues reported
4. We did NOT export DDR at this time (ran out of time). We did that test the next day. (see below)
2018-05-23 (Wednesday) Another FileMaker Server crash (3rd in 3 days) around 10 AM.
We suspected one particular file where the users were seeing beach balls on load...let us call it Suspect.fmp12
Here what I did next:
- Pulled a backup of Suspect.fmp12 from 2018-04-30 and Saved as a Clone
- Renamed the clone and hosted it with 0 records
- Opened the hosted Suspect.fmp12 file and verified that it was able to load (a good sign).
- Downloaded the most recent FileMaker backups and ran a recover on Suspect.fmp12
- Exported a .mer file from the recovered backup file
- Imported the .mer data into the hosted file
- Went to the hosted Suspect.fmp12 file and verified that we able to load fast and perform finds on it. Worked.
2018-03-23 Run a DDR on the 109 files (test 4/4) and validated that XML is well formed.
Here is where we are...
- Customer is up and running (which is good).
- The bad news is that none of the 4 tests performed on the files that were crashing the FileMaker Server reported an issue.
- This means that we are flying blind as to what caused the slow down and the daily crashes.
- Are there any other tests that one can perform that find issues with a file (other than the 4 mentioned above)?
- Might the issue have been with the data and not the structure? if I recall correctly (I looked unsuccessfully in my archive), someone had a utility that was said to check for bad data that might cause FileMaker to have issues. At that time there was a also debate on whether there was such a thing as “bad data” that could affect FileMaker.
- Is there a way to know if the content of a .mer file is “well formed”?
Feedback welcome, especially from FileMaker, Inc.
Thanks for reading this far!