When using named instances you will need to specify the URL slightly differently in the connection properties.
First off, try:
This is specified at the JTDS FAQ
If this doesn't work, try dropping the instance name, and changing the port to the port used by the named instance:
Note. This port is different to the normal SQL Server port as each instance listens on a different port.