Google "1008:9,-5000", you will find clues. This issue is not specific to FileMaker. Look into the installation log - can't tell if there is one.
You don't even have to go to Google. Just enter "1008:9,-5000" in this page's search box (upper right). Others before you (including me) have had this issue.
To shed some more light on this, I thought it was an issue with permissions... but even after repairing permissions (for which you have to disable csrutil) it turns out that you can't install FMS in El Capitan without disabling csrutil.
After much messing around, I eventually disabled csrutil (reboot holding cmd-R then run terminal and type "csrutil disable") and installed the server.
I'm leaving it off to save messing around for any FMS updates. Since that's all this server is doing, it should be fine.
You could re-enable it by repeating the boot process above and running "csrutil enable".
Can I confirm what version installer you are currently running?
There is an issue if you are installing using 14.0.1, 14.0.2 or 14.0.3 as these builds were not compatible with El Capitan
You will need to run FileMaker Server 14.0.4 to be able to install on El Capitan 10.11
Please note if you are running 10.11.4 you will need to install the 14.0.4b patch
I have included a link bellow with detail on this
FileMaker Server 14.0.4b Software Patch required for OS X El Capitan version 10.11.4