What happens when you go to a browser and type:
I have had a problem with FileMaker Server not allowing a pre configured port to communicate on Mac's.
I'm guessing that if you create a Favourite (and often if you don't bother to specify the files to view, but just the IP address) it will open correctly, yes?
After verifying that Bonjour was installed correctly, the most common cause I've seen is AV software on either machine. Try turning them off completely as a test; then if that exposes the Server in 'Open remote' disable the firewall component, or add an exception (although I often find this fails also).
You may also want to reinstall the software with the AV turned off, then re-enable it.
Always start from the networking basics. You've mentioned DNS but not sure whether you're connecting to the server via the FQDN or server IP address. I'll assume you are using Windows as you've updated Bonjour, which is built into the Mac.
Enable Telnet from Control Panel/Program and Features and enable in Windows Features.
From the command line run Telnet <Server IP Address> 5003 and you should get a 'Connected to <Server IP address.' if not then you have a firewall issue.
If the above works and you are using the FQDN, repeat the above replacing the server address with the FQDN, if the above works and this doesn't you've a DNS problem.
Make sure extended privileges allow for access