There is no information that Sierra is not supported. The page you referenced just shows some known issues of which none is relevant to my problem...
That is the most official information you can get from FileMaker Inc, that FileMaker Pro is not yet supported for Mac OS Sierra. Hopefully we will have a new version of FMP15v3 soon, but until then
FM checks certificate of FMS. The error above is not related to the certificate, it is an error coming from your email service provider. Verify your smtp setting, login information and password. Support for sending mail via an SMTP server vs. an email client | FileMaker
gmail also has a App password that can be used. Sign in using App Passwords - Gmail Help
Thank you all for your Help,
i was able to track down the Error:
I was using the Script Step "Send Mail" in which i filled all required Data for the SMTP Server via Fields of a related record. The script was set up in a way that your could select the "Sender" from a dropbox Menu and all required Data for the "Send Mail" SMTP Settings would be populated. Somehow this doesn't work properly for the User credentials and the Password. If i enter it manually it works just fine.