After creating an account for the database (not on active directory) the XML service works. I'm not sure if it is supposed to be able to use AD or not.
I had the identical problem but the remedy was to disable authentication for FMWebSite in IIS
The Filemaker user and password was OK but IIS was expecting to authenticate also. To turn this off:
On the server hosting IIS websites, disable IIS authentication:
1. In Control Panel, choose System and Security > Administrative Tools > Internet Information Services (IIS) Manager.
2. In Internet Information Services (IIS) Manager, select the FMWebSite site. You may have to expand some of the nodes to see the websites.
3. In the center pane, double-click Authentication.
4. In the Authentication pane, do the following:
Ensure that Anonymous Authentication is enabled
Disable all other authentication methods
5. Exit the IIS manager and Admin run CMD and run iisrest to restart the webserver
Thanks for posting this, I was having an issue on FMS13 where Web Direct was working fine but direct access using XML was always prompting for a password and username -- and these instructions corrected my issue.
I'm not completely sure why there is an apparent difference in logging into Web Direct (which gave me no browser prompt, but only the login form I would expect), while the XML method (using: /fmi/xml/fmresultset.xml?) was always prompting until I followed your instructions.