Problem

When configuring an SMTP mail server in Jira, if Protocol is set to SECURE_SMTP then Test Connection fails with the error Unsupported or unrecognized SSL message.

Diagnosis

Jira logs an error similar to the following in the atlassian-jira.log file:

2023-01-01 00:00:00,000+0000 https-jsse-nio-8443-exec-1 ERROR admin 123x456789x0 abcdef 10.0.0.1 /secure/admin/VerifySmtpServerConnection!update.jspa [c.a.j.p.mail.webwork.VerifyMailServer] Unable to connect to the server at host.domain.com due to the following exception: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message

Cause

This error occurs because there are two ways in which an SMTP mail server can support SSL/TLS encryption:

1. The SMTP connection begins as unencrypted, then Jira uses the STARTTLS SMTP extension to upgrade the connection to use SSL/TLS. (This method is often used because it allows an SMTP server to support both encrypted and unencrypted connections on a single port.)
2. The SMTP connection is always encrypted with SSL/TLS.

In the first scenario, Protocol should be set to SMTP, with the TLS checkbox enabled.

In the second scenario, Protocol should be set to SECURE_SMTP.

The error is caused by setting Protocol to SECURE_SMTP when the first scenario applies. In this case, Jira immediately tries to establish an SSL/TLS connection, but the mail server is expecting to begin with an unencrypted connection since it hasn't yet received the STARTTLS command. This results in Jira logging an SSL error, since the mail server responded with an unencrypted banner message instead of the expected SSL/TLS negotiation.

Resolution

Set Protocol to SMTP with the TLS checkbox enabled.