You must first close a file before you can remove it as this needs to physically move the file so the first message displayed would be expected. Not sure why you were unable to close the database and then remove it. You did wait until its status changed from "closing" to "closed"?
Once you see the file is closed, leave the admin console and just use your machine's operating system to try to move the file out of that directory. can you move it?
I'm having the same problem with the same OS, except the Remove button is grey for all the databases, whether closed or not. Where are the database files? Spotlight can't find them (I haven't disabled Spotlight yet). FM seems to be hiding everything.
There's a setting in the admin console where the location of the folder containing the files is specified.
In Server 10, you click the database servers link then select the default folders tab to see the location where the files are located. Presumably, Server 11 is very similar.