it may sound obvious, but has the boot volume got plenty of space?
It may be running out of memory. Is the cache set correctly?
The server has 32 GB of RAM.
of which 10 GB assigned to FileMaker Server (the admin console says upto 29 GB are ok).
I reduced the cache to 4 GB (4096 MB)
The server says, lots (about 60%) of free RAM available to the windows system.
yes. the C drive has about 40GB left of space. 60GB are used.
Anything else in the logs that might lend a clue?
You might have to run some monitoring to see if there's a memory leak. How much memory is being used by the FMS process over time (i.e. after a restart)?