Symptoms

The following error is logged in Confluence or JIRA logs:

An error has occurred with sending the test email:
com.atlassian.mail.MailException: javax.mail.MessagingException: 501 Syntax: HELO hostname

Some variations of the same error:

Unable to connect to the server at exchange.company.com due to the following exception: javax.mail.MessagingException: 501 Syntax: HELO hostname

Cause
Server's hostname is not defined, so Confluence or JIRA sends the server's IP address instead of the hostname on the HELO SMTP command.

Resolution

Edit /etc/hosts file (or C:\Windows\system32\drivers\etc\hosts) adding an entry to convert your computer's IP to its hostname. It will be something like:

192.168.132.27  exchange.company.com

This can be fixed as well on the DNS server. Indeed, make sure JIRA is using the correct DNS server.