Jira Data Center node locking on startup

Still need help?

The Atlassian Community is here for you.

Ask the community


Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.

Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

When starting up a Jira node in cluster, and it is not coming up successfully, there is a FATAL error in the log: 

2020-07-28 16:34:37,899 Jira-Bootstrap FATAL      [c.a.jira.startup.JiraStartupLogger] Startup check failed. Jira will be locked.
  • this means that the startup process failed to complete all the startup checks

  • there are checks on db driver, database access and cluster access

  • shared home is an important part of cluster access

The following appears in the catalina.out log:

28-Jul-2020 16:34:37.589 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
28-Jul-2020 16:34:37.596 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
28-Jul-2020 16:34:37.611 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 4293 ms
2020-07-28 16:34:37,723 Jira-Bootstrap INFO      [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /appl/jira/home/dbconfig.xml
2020-07-28 16:34:37,899 Jira-Bootstrap INFO      [c.a.jira.startup.JiraStartupLogger] Running Jira startup checks.
2020-07-28 16:34:37,899 Jira-Bootstrap FATAL      [c.a.jira.startup.JiraStartupLogger] Startup check failed. Jira will be locked.
2020-07-28 16:34:37,960 Jira-Bootstrap INFO      [c.a.jira.startup.LauncherContextListener] Memory Usage:

Diagnosis

Environment

  • Database driver is present and there are no problems connecting to the database.

    • (warning) If the character set or collation on the database are incorrect or unsupported, errors will not be thrown with default INFO level logging.

Diagnostic Steps

  • A related symptom is that the application logs are not being written to

  • the Jira process user already has full permissions to local node installation and Jira home directory

  • You can go into your log4j.properties file to increase the logging for the log4j.logger.com.atlassian.jira.startup and log4j.logger.com.atlassian.jira.config.database packages from INFO to DEBUG for more verbose logging during these operations.
    • This will require a restart of the Jira instance.

Happy flow

In case of all checks are successful, there will be logging similar to this:

2020-01-13 17:08:12,779+0100 localhost-startStop-1 INFO      [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/app/jira/application-data/jira_8.5' is validated and locked for exclusive use by this instance.


Validation code 

Checks that are executed as part of validateJiraHome

 // Validate the jira.home, and create the directory if required
 // Note that we only save jiraHomeDirectory if everything is valid.
    private Either<JiraHomeStartupCheckFailure, File> validateJiraHome(final String jiraHome) {
        final File proposedJiraHome = new File(jiraHome);
        return checkProposedHomeAbsolute(proposedJiraHome, jiraHome)
                .flatMap(this::checkWebappPathCollision)
                .flatMap(this::checkJiraHomeCreationIsPossible)
                .flatMap(this::createJiraHomeAndSharedHome)
                .flatMap(this::lockJiraHome);
    }


Cause #1

There is a problem with the shared home access. Shared home in Jira Data Center needs to be accessed read/write by the Jira process. Additionally, if the shared home is too slow (latency) or intermittently unavailable, the same problem may be observed.

Subdirectories that checked/created as part of createSharedHomeDirectories

private Either<JiraHomeStartupCheckFailure, File> createSharedHomeDirectories(final File sharedJiraHome, final File proposedJiraHome) {
        final Set<String> subdirs = CollectionBuilder.<String>newBuilder()
                .add(AttachmentPathManager.ATTACHMENTS_DIR)
                .add(PluginPath.PLUGINS_DIRECTORY)
                .add(PluginPath.BUNDLED_SUBDIR)
                .add(PluginPath.OSGI_SUBDIR)
                .add(FileService.MAIL_DIR)
                .addAll(JiraHome.sharedsubdirs)
                .asMutableSortedSet();

Cause #2

This problem can be observed when the collation or character set of the database are unsupported. This can occur when a database failover occurs to a database with a different charset/collation than the primary database. 

Resolution

To fix the problem on startup:

  • Verify that the database driver and access are valid on the node

  • Ensure that all Jira folders are owned and writable by the Jira process user

  • Validate and fix any problem with shared home read/write access 

  • If a database migration or failover occurred, ensure that the charset and collation for the new database are correct and supported


DescriptionJira DC startup problem
ProductJira Data Center

Last modified on Mar 9, 2022

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.