Thanks for posting.
Having a previous version of FileMaker Server installed will prevent FileMaker Server 11 from installing correctly. If the terminal command is not working, you can try stopping the services from the activity monitor.
Open the activity monitor from Macintosh HD/applications/utilities folder. Filter the list of processes by user. Select and quit all processes for the user fmserver. Once the processes are stopped, you can then uninstall FileMaker Server 9. Here are the steps you want to follow for migrating to FileMaker Server 11.
1. With the processes stopped, run the FileMaker Server 9 install disk or file. In the install prompt choose the option to uninstall.
2. On your previous FileMaker Server 9 machine, launch the admin console. Go to the configuration pane and note down the settings for backup, script, and message schedules.
3. On your old server running FileMaker Server 9, back up the following folders. These folders contain you databases, shell scripts, and plug-ins. You will need to move these folders to the new server.
\Program Files\FileMaker\FileMaker Server\Data\Databases\ This folder contains the database files.
\Program Files\FileMaker\FileMaker Server\Data\Scripts\ This folder contains shell scripts.
\Program Files\FileMaker\FileMaker Server\Extensions\ This folder contains plug-ins.
4. On the new server, start the FileMaker Server 11 install from the CD or download. Once FileMaker Server 11 is installed, you can move the folders from FileMaker Server 9 to the corresponding folders in FileMaker Server 11, which are listed below.
/Library/FileMaker Server/Database Server/Extensions/
5. You can now start the Admin Console and configure your FileMaker Server deployment using the settings you noted earlier.