Would have been nice if the server would give better error response and not just a message there are unsaved changes.
This is a bit confusing, what more would you expect? Or do you mean that the message is not visible enough.
If you can make a "mockup" drawing of your idea, please post it to Product Ideas so it can be voted on.
Otherwise, you followed what I normally recommend to repair an installation; specifically you need to remove or rename this folder:
Macintosh HDD/Library/FileMaker Server
when you are reinstalling, so that the old files are not causing issues.