Had the same problem and am not a Command Line guy to but managed to muddle through it by trial and error and referencing the "fms11_help.pdf". These are the procedures what worked for me:
1. Launch "Terminal" (usually in Applications : Utility folder)
2. Enter "fmsadmin close -y" to close all running databases.
3. You will be asked for your username and password that you set up for FMP Server. Then you will see lines indicating databases being stopped.
4. Next enter "sudo launchctl stop com.filemaker.fms" which will stop all components of all FMS.
5. Install downloaded update 11.0.3. Restart.
Did a number of trial and errors but believe this was the last round of steps that worked. No guarantees so proceed at your own risk but it seemed to work for me. The new Server Admin console now launches.
Yeah it actually did wok and all is well again... once I figured out that the numeric characters in my FM password can't be entered from the numeric keypad. Plus the "If you enter a sudo command line it might launch ICBM's so proceed at your own risk"! I felt like I was working on a BMW with a chainsaw.
Thanks so much for your help.