Documentation for JIRA 4.2. Documentation for other versions of JIRA is available too.

When using named instances you will need to specify the URL slightly differently in the connection properties.

First off, try:

url="jdbc:jtds:<server_type>://<server>[:<port>][/<database>];instance=<instance_name>"

This is specified at the JTDS FAQ

If this doesn't work, try dropping the instance name, and changing the port to the port used by the named instance:

url="jdbc:jtds:<server_type>://<server>[:<instance_port>][/<database>]"

Note. This port is different to the normal SQL Server port as each instance listens on a different port.