You can't set the parameters directly, but you could define several dialogues with different settings and branch according to your fields.
How would I go about doing that?
Define the dialog with all settings, duplicate it and change the settings for encryption and authentication, make another duplicate, adjust again, until you have covered all (desired) permutations of authentication and encryption.
In your script branch like…
If [ myTable::myAuthSetting = "Plain" and myTable::myEncryptSetting = "TLS" ]
Send Mail [ with one set of settings ]
Else If [ different field contents ]
Send Mail [ with another set of settings ]
Else If [ … ]
If Send Mail is part of a larger script, it may be a good idea to put this branching logic into its own script, and call it from the main script.
It's a somewhat clumsy, but time-honored technique (think Print dialogs …).