I am having similar problems to gordonwege from my desktop computer to a remote server (FMPA and FMS both v16, both Mac OS 10.12.6).
The problem occurs regardless of whether I use a Local Host or a manually defined host. I have credentials saved in my keychain for the server and these show up in the "Upload to Server" dialogue, but they are greyed out. I cannot get into the fields.
If I choose Open Remote, I am however able to see and open databases hosted on that server using these same Host definitions, so clearly the Host definitions are valid. Creating a new Host definition does not help.
I transferred the database file the to the server machine and then tried using a copy of FMPA 16 installed on the server to do the same operation and it worked successfully.
I could not find any settings on FMS that would affect this behaviour.
I had no difficulty uploading the same file to a client's FMS 14 (Windows Server 2008) so it seems more likely to be a server issue.
My Mac test server is just using the FileMaker default SSL Certificate. It does prompt and I allow, but could that be an issue?