Have you tried running the latest version of the installer, 11.0v4+? There were later releases of 11 for lion compatibility that may have system files you need to work.
My honest advice is if you have to stick with FMS11, then you should stick to the hardware that is already working.
Alternatively, a windows server or virtual server from a host that is running windows server 2008 or 2012 may allow you the hardware upgrade you desire. I've found that IIS is much easier to modify than the OSX apache configs, and usually has less issues on install.
FMS11 was released in the 10.6-7 (snow leopard/lion) era and support was ending before yosemite even came out. It's technically not even supported for mavericks/10.9. v11 is now well over 5 years old, has had support discontinued with over a year's notice, and isn't even the current file format. The sooner you update, the better it will be for your long term continued use of FileMaker.
And as a final note, the official stance of filemaker starting with 10.9:
While these earlier versions may install and run on Mac OS X v10.9 Mavericks, you may encounter installation and stability issues for which there is no resolution.
I was confronted with the same decision and ended in installing 10.6 Server in a VMWare-Fusion virtual machine on the new Mac Mini. The VM can be run headless by shell commands and as FM Server 11 can't take advantage of all the possible RAM anyway, it runs fast and smooth. Advantage of this setup: you can install the new FMS version parallel to the host machine, when you're ready to upgrade.
Yep -- the installer I used was 11.05 with the fms11patch ...
I have serving FileMaker clients working just fine -- just the web server deployment that's the issue. No matter what I try, I don't seem to be able to get FileMaker to use it's own web server, OS X's, or even OS X Server.
Any ideas anyone before I just pitch using web serving?
You can try the virtual machine idea, you might be able to image snow leopard or an older OS inside of that to get compatibility with the old software.