Storing attachments in Amazon S3

Configuring Amazon S3 object storage

On this page

Still need help?

The Atlassian Community is here for you.

Ask the community

This feature is currently available behind the feature flag: com.atlassian.jira.attachments.storage.configurable. When enabled, it introduces breaking changes to the API and might also break some Jira apps. Check what’s changed in the API

Make sure that you properly test the feature before enabling it in your production environment.

Learn how to enable dark features in Jira

By default, all attachments in Jira are stored locally in the file system. You can also configure a custom storage method — Amazon S3 (Simple Storage Service). We recommend using Amazon S3 if your team has large or increasing data needs and requires the ability to scale efficiently.

Before you begin

If you're considering using Amazon S3 for storing attachments, first read through the configuration requirements and current limitations to make sure this storage method is suitable for you. Learn more about Amazon S3 configuration

How S3 attachment storage is set up in Jira

Attachment storage in Amazon S3 is configured in the filestore-config.xml file that should be located in Jira <localhome>.

To use S3 as a target location for attachment data, the filestore attribute in the filestore-config.xml must match the s3-filestore ID.

Example filestore-config.xml file configured to use S3
<?xml version="1.1" ?>
<filestore-config>
  <filestores>
    <s3-filestore id="attachmentBucket">
      <config>
        <bucket-name>jira-bucket</bucket-name>
        <region>us-east-1</region>
      </config>
    </s3-filestore>
  </filestores>
  <associations>
    <association target="attachments" file-store="attachmentBucket" />
  </associations>
</filestore-config>

If you’re already storing avatars in S3, you can configure the filestore-config.xml file to store attachments there as well. To do this, add a new association element with the target attachments.

If you want to store attachments in the same bucket as avatars, the filestore attribute should point to the same bucket where avatars are located.

Example filestore-config.xml for storing both avatars and attachments in the same bucket
<?xml version="1.1" ?>
<filestore-config>
  <filestores>
    <s3-filestore id="jiraBucket">
      <config>
        <bucket-name>jira-bucket</bucket-name>
        <region>us-east-1</region>
      </config>
    </s3-filestore>
  </filestores>
  <associations>
    <association target="avatars" file-store="jiraBucket" />
    <association target="attachments" file-store="jiraBucket" />
  </associations>
</filestore-config>

To use separate buckets for attachments and avatars, define multiple <s3-filestore> elements and reference each of them in the respective association targets.

Example filestore-config.xml for storing avatars and attachments in separate buckets
<?xml version="1.1" ?>
<filestore-config>
  <filestores>
    <s3-filestore id="avatarBucket">
      <config>
        <bucket-name>jira-avatar-bucket</bucket-name>
        <region>us-east-1</region>
      </config>
    </s3-filestore>
    <s3-filestore id="attachmentBucket">
      <config>
        <bucket-name>jira-attachment-bucket</bucket-name>
        <region>us-east-1</region>
      </config>
    </s3-filestore>
  </filestores>
  <associations>
    <association target="avatars" file-store="avatarBucket" />
    <association target="attachments" file-store="attachmentBucket" />
  </associations>
</filestore-config>

If you want to use a Jira <sharedhome> directory to store attachments but don’t want to delete the configuration file, just remove the association element with a target of attachments. Jira will default to storing attachments locally if S3 isn't configured in filestore-config.xml or if this file is missing from the <localhome> directory.

Example filestore-config.xml file configured to use shared home
<?xml version="1.1" ?>
<filestore-config>
  <filestores />
  <associations />
</filestore-config>

Connecting to Amazon S3 dual-stack endpoints

If your Jira installation is on an IPv6-only network, you need to connect to Amazon S3 dual-stack endpoints. Learn more about Amazon S3 dual-stack endpoints

To set up the connection, you need to override the default endpoint with the dual-stack endpoint like this:

<?xml version="1.1" ?>
<filestore-config>
  <filestores>
    <s3-filestore id="attachmentBucket">
      <config>
        <bucket-name>dualstack-bucket</bucket-name>
        <region>us-east-1</region>
        <endpoint-override>https://s3.dualstack.us-east-1.amazonaws.com</endpoint-override>
      </config>
    </s3-filestore>
  </filestores>
  <associations>
    <association target="attachments" file-store="attachmentBucket" />
  </associations>
</filestore-config>

You can also use this configuration option to store attachments in a third-party object store that exposes an S3-compatible API. However, we’re not providing direct support for attachments that are stored in an object store other than Amazon S3.

Migrate your attachment data to Amazon S3

If you have existing attachments in the file system and want to use Amazon S3, you need to migrate all attachments to an S3 bucket for Jira to consume.

If you are using the Assets app for Jira Service Management, it will store its data in the <sharedhome>/data/attachments/insight/ directory. These files need to remain in the shared home. Otherwise, the Assets app will not function correctly.
Other Jira apps can also install app-specific data to the <sharedhome>/data/attachments directory. Moving this data to S3 might also break their functionality.

To migrate your attachments:

  1. Check your Jira version and make sure that you’re on Jira 9.11 or later.

  2. Create and set up a new Amazon S3 bucket for Jira. Learn how to configure Amazon S3 as your data storage method

  3. Migrate your attachment data from its physical location in <sharedhome>/data/attachments/ to the root prefix attachments/ in the S3 bucket.

    The physical location of attachments depends on your environment. For example, clustered environments typically host this data in a network file system (NFS) as a shared mount.

    You need to consider your setup and how much data you need to migrate. In general, we recommend using Amazon DataSync for migration. 

    Learn how to migrate the data with Amazon DataSync


  4. Wait for the migration to complete.

  5. Configure your Jira nodes one by one by creating a filestore-config.xml file with valid S3 bucket information in the <localhome> of each node in your Jira cluster. After you provide the relevant configuration, each node will require a restart. 
    During this process, if attachment files are created on nodes that have yet to be configured for S3, attachments won't be available to already configured nodes. Likewise, those nodes will not have access to attachments created on nodes that are already configured to use S3.

  6. Ensure the feature flag is enabled.

  7. Verify that Jira is using S3 object storage:

    1. In the upper-right corner of the screen, select Administration  > System.

    2. Under Advanced (in the left-side panel), select Attachments. To access the page directly, use the gg shortcut or press the full stop . and search for “Attachments”.

    3. Next to the Attachments location, you should see Amazon S3, as well as the region and bucket you specified in filestore-config.xml.
      Attachment settings showing attachments stored in s3

  8. Make sure all the nodes have been configured and all attachments are migrated, and then re-run the original DataSync job to perform a final sync.

  9. All attachments should now be read and written from Amazon S3.

DataSync doesn’t change or remove the source file system data. If you no longer need attachments located in the file system, you need to remove this data manually.

Switching back to local attachment storage

As the source file system data is not changed or removed by DataSync, Jira can be reverted back to reading and writing attachment data from the file system. To do this, remove the filestore-config.xml files from your <localhome> directories and restart Jira. You can also delete the <association> element targeting attachments.

If you are reverting back to the original file system, any data written to S3 must be synced back to the file system manually by the Jira administrator.

Configure Amazon S3 to store attachments

If you’re ready to proceed with Amazon S3 configuration, follow the instructions from the Configuring Amazon S3 as your data storage method.

Connect your S3 bucket with Jira

After you configure S3 object storage, you need to connect the created S3 bucket with your Jira instance:

  1. In the Jira application home directory of one of your Jira installation nodes, create a filestore-config.xml file. The Jira application home directory should be set to the value of the JIRA_HOME environment variable. Learn about the contents of the Jira application home directory

    In the filestore-config.xml file, define which S3 bucket will be used by Jira to store attachments and avatars. Learn more about storing avatars in S3

    Sample filestore-config.xml file:

    <?xml version="1.1" ?>
    <filestore-config>
      <filestores>
        <s3-filestore id="attachmentBucket">
          <config>
            <bucket-name>example-co-jira-attachment-bucket</bucket-name>
            <region>ap-southeast-4</region>
          </config>
        </s3-filestore>
      </filestores>
      <associations>
        <association target="attachments" file-store="attachmentBucket" />
      </associations>
    </filestore-config>
    1. If you’re running a clustered installation, copy the filestore-config.xml file to the Jira application home directory of the other nodes.

    2. Start or restart all Jira nodes.

    When Jira starts up, it will check your bucket configurations, such as bucket connectivity, name and region validity, and bucket permissions. Learn about potential errors and how to fix them

To verify that Jira is using Amazon S3 object storage:

  1. In the upper-right corner of the screen, select Administration  > System.

  2. Under Advanced settings (the left-side panel), select Attachments.

  3. Next to the Attachments storage location, you'll see that attachments are stored in S3. Here you can also check the S3 bucket name and region.Attachment settings showing attachments stored in s3

Troubleshooting S3 attachment storage

When Jira starts up, it runs a series of checks to make sure there are no issues with the filestore-config.xml file. If there are any errors during the file parsing, Jira won't start and will display an error message.

If there are problems connecting to or performing operations on S3, Jira will also detect them and flag the Attachment Filestore instance health check as failing.

The following sections list the problems that can happen during S3 configuration along with the resolution steps. The issues are mainly related to improper S3 configuration, permissions, or authentication.

Jira startup failures

You can also find more details about the problems by checking Jira logs at <localhome>/log/atlassian-jira.log. Learn how to access Jira logs

Problem

Root cause & resolution

Couldn’t parse filestore-config.xml

The filestore-config.xml file couldn’t be parsed.

Ensure that the file consists of valid XML.

Invalid <s3-filestore> value

The <s3-filestore> element doesn't contain the <config>, <bucket-name>, or <region> elements.

If you provide an <s3-filestore> element in your filestore-config.xml, it must contain all preceding elements with non-empty values.

Check out the sample filestore-config.xml file 

Missing <s3-filestore> value

An <association> was defined targeting a non-existent <s3-filestore> element.
Ensure that the <s3-filestore> element is defined in the <filestores> collection. If the element exists, make sure that the file-store attribute of the <association> matches the id attribute of the <s3-filestore> element.

<filestore-config>
  <filestores>
    <s3-filestore id="THESE MUST MATCH">
      ...
    </s3-filestore>
  </filestores>
  <associations>
    <association target="attachments" file-store="THESE MUST MATCH" />
  </associations>
</filestore-config>

Health check failures

Learn more about the instance health checks and how they are performed. Learn more about instance health checks

Problem

Root cause & resolution

Unable to list the contents of attachment storage

This health check can fail due to the following reasons.

  1. Jira can’t connect to Amazon S3.

  2. The <bucket-name> element has an invalid value:

    • Double-check that your bucket name follows Amazon S3 bucket naming rules. Check bucket naming rules

    • After correcting the bucket name, restart Jira.

  3. The ListBucket permission is not present:

Details on why the check failed are provided in the health check result. If you need further help diagnosing the issue, check the logs in <localhome>/log/atlassian-jira.log for a stack trace.

Unable to read data from attachment storage

The GetObject permission is not present:

  1. check that the correct bucket permissions are in place

  2. apply the permissions as necessary, and then restart Jira

Unable to write to attachment storage

The PutObject permission is not present:

  1. check that the correct bucket permissions are in place

  2. apply the permissions as necessary, and then restart Jira

Unable to delete files from attachment storage

The DeleteObject permission is not present:

  1. check that the correct bucket permissions are in place

  2. apply the permissions as necessary, and then restart Jira

Last modified on Aug 28, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.