Problem

When starting up a Jira node in cluster, and it is not coming up successfully, there is a FATAL error in the log: 

2020-07-28 16:34:37,899 Jira-Bootstrap FATAL      [c.a.jira.startup.JiraStartupLogger] Startup check failed. Jira will be locked.

• this means that the startup process failed to complete all the startup checks

• there are checks on db driver, database access and cluster access

• shared home is an important part of cluster access

The following appears in the catalina.out log:

28-Jul-2020 16:34:37.589 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
28-Jul-2020 16:34:37.596 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
28-Jul-2020 16:34:37.611 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 4293 ms
2020-07-28 16:34:37,723 Jira-Bootstrap INFO      [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /appl/jira/home/dbconfig.xml
2020-07-28 16:34:37,899 Jira-Bootstrap INFO      [c.a.jira.startup.JiraStartupLogger] Running Jira startup checks.
2020-07-28 16:34:37,899 Jira-Bootstrap FATAL      [c.a.jira.startup.JiraStartupLogger] Startup check failed. Jira will be locked.
2020-07-28 16:34:37,960 Jira-Bootstrap INFO      [c.a.jira.startup.LauncherContextListener] Memory Usage:

Diagnosis

Environment

• Database driver is present and there are no problems connecting to the database.

  •  If the character set or collation on the database are incorrect or unsupported, errors will not be thrown with default INFO level logging.

Diagnostic Steps

• A related symptom is that the application logs are not being written to

• the Jira process user already has full permissions to local node installation and Jira home directory

• You can go into your log4j.properties file to increase the logging for the log4j.logger.com.atlassian.jira.startup and log4j.logger.com.atlassian.jira.config.database packages from INFO to DEBUG for more verbose logging during these operations.
  • This will require a restart of the Jira instance.

Happy flow

In case of all checks are successful, there will be logging similar to this:

2020-01-13 17:08:12,779+0100 localhost-startStop-1 INFO      [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/app/jira/application-data/jira_8.5' is validated and locked for exclusive use by this instance.

Validation code 

Checks that are executed as part of validateJiraHome

 // Validate the jira.home, and create the directory if required
 // Note that we only save jiraHomeDirectory if everything is valid.
    private Either<JiraHomeStartupCheckFailure, File> validateJiraHome(final String jiraHome) {
        final File proposedJiraHome = new File(jiraHome);
        return checkProposedHomeAbsolute(proposedJiraHome, jiraHome)
                .flatMap(this::checkWebappPathCollision)
                .flatMap(this::checkJiraHomeCreationIsPossible)
                .flatMap(this::createJiraHomeAndSharedHome)
                .flatMap(this::lockJiraHome);
    }

Cause #1

There is a problem with the shared home access. Shared home in Jira Data Center needs to be accessed read/write by the Jira process. Additionally, if the shared home is too slow (latency) or intermittently unavailable, the same problem may be observed.

Subdirectories that checked/created as part of createSharedHomeDirectories

private Either<JiraHomeStartupCheckFailure, File> createSharedHomeDirectories(final File sharedJiraHome, final File proposedJiraHome) {
        final Set<String> subdirs = CollectionBuilder.<String>newBuilder()
                .add(AttachmentPathManager.ATTACHMENTS_DIR)
                .add(PluginPath.PLUGINS_DIRECTORY)
                .add(PluginPath.BUNDLED_SUBDIR)
                .add(PluginPath.OSGI_SUBDIR)
                .add(FileService.MAIL_DIR)
                .addAll(JiraHome.sharedsubdirs)
                .asMutableSortedSet();

Cause #2

This problem can be observed when the collation or character set of the database are unsupported. This can occur when a database failover occurs to a database with a different charset/collation than the primary database. 

Resolution

To fix the problem on startup:

• Verify that the database driver and access are valid on the node

• Ensure that all Jira folders are owned and writable by the Jira process user

• Validate and fix any problem with shared home read/write access 

• If a database migration or failover occurred, ensure that the charset and collation for the new database are correct and supported