Make sure you downloaded a new Admin Console for FileMaker Server 10 since your previous shortcut will no longer work. In addition, delete the Java Cache (via Java Preferences) as the Java cache may be retaining the FileMaker Server 9 configuration.
- Mac OS X 10.4.xxx: Clear the cache in Applications > Utilities > Java > Java Preferences >General Tab > Delete Files Button.
- Mac OS X 10.5.xxx: Clear the cache in Applications > Utilities > Java > Java Preferences >Network Tab > Delete Files Button.
- Windows: Click Start > Control Panel > Java. Click Settings under Temporary Internet Files and click Delete Files and select Applications and Applets and click OK.
Note: This deletes all the Downloaded Applications and Applets from the cache.
This happened to me and this fixed it.