Thank you for your post.
From your description, it appears the FileMaker Server certificate was not installed properly. I would uninstall and reinstall FileMaker Server. Then, turn on "Requests secure connections" again. Make sure ports 443 and 80 are open as well as the other ports referenced in Knowledge Base Article #12120:
You should then be able to connect via Web Direct.