Purpose

The purpose of this document is to provide instructions for resolving issues related to broken external integrations caused by entity ID mismatches following cloud migration. If Jira entities such as Project, Issue, Board, Filter, and Custom Field are referenced by their integer IDs on the server or Data Center, their continued accessibility may be compromised after migrating to the cloud, as these integer IDs are subject to change. Consequently, any programmatic or directed integration of these products post-migration may fail if the access patterns rely on integer IDs to retrieve the entities.

The problem is highlighted below.

Consider a Server To Cloud migration of the Atlassian Jira Project

Project Name: Sample Project
Project Key: SMP
Project ID: 10000

Issues(Key, Id)
SMP-1, 10000
SMP-2, 10001
IssueTypes(Key, Id)

Project Name: Sample Project
Project Key: SMP
Project ID: 10041

Issues(Key, Id)
SMP-1, 10003
SMP-2, 10004

Access Pattern that works and breaks on the cloud after migration

https://cloud.url/rest/api/2/project/SMP

https://cloud.url/rest/api/2/project/10000

curl --request POST \ --url 'http://{baseurl}/rest/api/2/issue' \ --user 'email@example.com:<api_token>' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' RequestBody { "fields": { "project": { "key": "SMP" }, "summary": "Epic created through REST", "description": "Creating of an issue using project keys and issue type names using the REST API"} }

curl --request POST \ --url 'http://{baseurl}/rest/api/2/issue' \ --user 'email@example.com:<api_token>' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' RequestBody { "fields": { "project": { "id": 10000 }, "summary": "Epic created through REST", "description": "Creating of an issue using project keys and issue type names using the REST API", "issuetype": { "id": 10000 }, "customfield_10106": "Rest API Epic Name" } }

https://cloud.url/rest/api/2/search=jql=issue=SMP-1

https://cloud.url/rest/api/2/search=jql=issue=10003

Solution

A solution currently exists to address external integration by retrieving mappings of these IDs (serverId, cloudId) utilising the aforementioned approach, manually correcting or replacing all integer IDs. However, this method is restrictive and lacks scalability, as it is labor-intensive to rectify all integrations manually through ID replacement.

Cloud Transition Tools (CTT) is a containerized and deployable application that can be implemented on the Containers as a Service (CaaS) platform of preference. CTT is developed upon the existing solution for retrieving mapping information and offers API access for enhanced functionality. CTT can either be directly integrated with customer application/ code or can access the REST APIs to fix integrations. Detailed information regarding the APIs can be found in the CTT APIs section. The APIs encompass fundamental translation capabilities to retrieve cloudID from (serverID, entityType), as well as high-level APIs aimed at addressing various use cases, including URL, API, and JQL, among others.

Complicated and proprietary integrations can be addressed by incorporating code into the external application with the appropriate calls to CTT Rest APIs for ID mapping and translation. For instance, to resolve a third-party integration that has cached thousands of Jira issues on their end, modifications to the application database would be necessary. This task falls outside the scope of CTT. A custom plugin can be developed for the third-party application to iterate through all entities and update their IDs accordingly. The CTT translation APIs can then be utilized by the third-party application to implement the integration solution effectively.

A few of the 3P apps for which this solution can be used are(but not limited to):

1. Rich Filters App

2. Jenkins

3. Github

4. Jenkins

5. Gitlab

6. Salesforce

7. Clarity

8. Jama

9. Snowflake

10. Scriptrunner

Solution Showcase and Demo

Please refer to the following videos on setup and usage of CTT

https://www.loom.com/share/e176c938e0124446a0251694ae33c66f?sid=0bbe506a-d5af-4e53-ba42-2dc884199685

https://www.loom.com/share/3d72cb1be49047a8b46afaa794a8190c?sid=55ceeb73-25e4-47ac-9042-7d3adcd92d22

CTT Spring Boot Application

CTT Application is scoped for a cloud. This means multiple migrations to the same cloud can be handled by a single instance of CTT Application. If you have multiple cloud destination endpoints, you would need to have multiple CTT Apps for each cloud. You need to specify the cloudUrl in application.properties  file as detailed in the README file.

App Configuration and Initialisation

This is a config-first-driven application, and you need to specify the parameter values in the configuration before building the application. These values are read and injected into the Spring Boot-based application to start up the same. Remember, you can always skip starting the Spring app for REST API endpoints and can directly use the classes and APIs, meaning you don’t need to configure Spring to use the tools/APIs but can call the methods directly.

Here is how the application file looks like.

# Application name
spring.application.name=cloud-transition-tools
server.servlet.context-path=/ctt

# Cloud URL - Ensure no trailing slashes
ctt.cloudURL = <SPECIFY YOUR CLOUR URL>

# Data loader types.
# only jcma loader is supported for now
ctt.data.loader = jcma

# Data store types.
ctt.data.store = memory

# Database access
# Demonstrated using H2 but can use any database with jdbc driver
spring.datasource.driver-class-name=org.h2.Driver
spring.datasource.url=jdbc:h2:file:./data/ctt
spring.datasource.username=<USERNAME>
spring.datasource.password=<PASSWORD>
spring.sql.init.schema-locations=classpath:mapping.sql
spring.sql.init.mode=always

Building CTT

CTT Spring boot application can be built by referring to the README.

CTT API Details

The following table summarises all available REST APIs offered by CTT.

curl --location --request POST 
'<ctt.url>/ctt/rest/v1/load? \
serverBaseURL=<server_url>& \
reload=false' \
--header 'Authorization: <auth>'

curl --location '<ctt.url>/ctt/rest/v1/server-to-cloud? \
entityType=jira%2Fclassic%3AissueType& \
serverId=10001& \
serverBaseURL='<server.base.url>'

{
    "serverUrl": "server.url",
    "entityType": "jira/classic:issueType",
    "serverId": 10001,
    "cloudId": 10004,
    "id": null
}

{
    "serverUrl": "server.url",
    "entityType": "jira/classic:issueType",
    "serverId": 29999,
    "cloudId": 0,
    "id": null
}

curl --location '<ctt.url>/ctt/rest/v1/cloud-to-server? \
entityType=jira%2Fclassic%3AissueType& \
serverId=10004& \
serverBaseURL='<server.base.url>'

{
    "serverUrl": "server.url",
    "entityType": "jira/classic:issueType",
    "serverId": 10001,
    "cloudId": 10004,
    "id": null
}

curl --location '<server.url>/ctt/rest/v1/url-sanitise? \
url=serverurl%2Frest%2Fapi%2F2%2Fproject%2F10200'

cloudurl/rest/api/2/project/10432

curl --location --request GET 
'ctt.url/ctt/rest/v1/api-sanitise? \
url=server.url%2Frest%2Fapi%2F2%2Fboard%3FprojectKeyOrId%3D10200' \
--header 'Content-Type: application/json' \
--data '{
        
        "fields": {
           "project":
           {
              "id": 10200
           },
           "summary": "Epic created through REST",
           "description": "Creating of an issue using project keys and issue type names using the REST API",
           "issuetype": {
              "id": 2
           },
           "customfield_10200": "Rest API Epic Name"
       }
    }'

{
    "url": "cloud.url/rest/api/2/board?projectKeyOrId=10037",
    "body": {
        "fields": {
            "project": {
                "id": 10037
            },
            "summary": "Epic created through REST",
            "description": "Creating of an issue using project keys and issue type names using the REST API",
            "issuetype": {
                "id": 10046
            },
            "customfield_10097": "Rest API Epic Name"
        }
    }
}

curl --location --request GET 'ctt.url/ctt/rest/v1/jql-sanitise? \
serverBaseURL=server.url.com' \
--header 'Content-Type: application/json' \
--data 'issue = 25887 AND issueType = 10001'

issue = 18341 AND issueType = 10004

CTT Application Modules

Migration Mapping Loader

Loads migration mapping information from Atlassian Cloud. We support only Atlassian Jira entities migrated from server to cloud in the first version. Typically this section does not require any changes for adding new integrations. Supported loaders are only  JCMA   and this can be specified in the application file. Data Loaders load the mapping file using API and populate a data store. Other data loaders say, for example, FileDataLoader to load from a file, can be implemented by extending the interface by implementing the load function. Also downloading the mapping information by other means can be added.

JCMA Loader

Loads the mapping information from the Jira cloud migration assistant(JCMA) present in the server. CTT App calls the JCMA REST API endpoint and downloads the CSV file containing all mapping, streams it, and populates the data store. The load function accepts a data store  to store this information. Further APIs access this data store for finding out the id-mapping. You are required to provide the server credentials as an authorisation header when instantiating the JCMA loader. This information will be directly transmitted to the function that executes the API call. The load function is implemented as a Kotlin suspend   function, which allows it to be executed as a coroutine. This design choice enables the main application thread to remain unblocked while loading and populating the mapping efficiently. All APIs implemented at the CTTService  will return an HTTP Error 503: Service Unavailable during the data loading process. You can see more about the usage by looking at loader tests.

Migration Data Store

Migration mapping information downloaded by the loader is populated in a data store. The mapping information is nothing but  (serverURL, entityType, serverID, cloudID) . The load function of the loader takes datastore as a parameter. DataStore stores the data as per requirement. That is, it can be stored in memory for nonpersistent use-case and quick translation, or it can be stored persistently in a database. Remember to use proper data stores for your use case. Storing a large amount of data in memory can result in memory exhaustion.

Memory Migration Store

This is an in-memory datastore that stores the mapping information in an in-memory hash map. You can specify memory as the value in the application.properties  to use this store. Remember that each load will reload the data and it will be time consuming

Persistent Migration Store

This stores the mapping information persistently in a database. You can specify the database details in application.properties   file. You can use any of the available JDBC class names as per your need, including MySQL, PostgreSQL, Oracle, MariaDB, and also any of the production databases with a valid JDBC driver. Refer to https://docs.spring.io/spring-boot/docs/1.3.0.M2/reference/html/boot-features-sql.html for more details. Say, for example, if you want to use Amazon RDS, you can connect through MySQL using the JDBC URL. Refer more https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/java-rds.html

CTTService and APIs

This is the main CTT service that exposes basic lookup APIs for ID mapping translation. That is given a server URL, entityType, and serverID. What is the cloud?  The Spring boot application automatically instantiates and injects this component using inversion of control. If you want to directly use the class, you need to provide the cloudURL  and migrationStore with data populated. So first, use a loader, load and populate the data store, and then instantiate CTT Service to access the APIs. You can refer to the available unit tests to see sample usage.

Following are the basic APIs available:

Load - Loads migration mapping data into the data store. You need to pass a completely instantiated loader to pull the migration data from. You have to call the load API before calling any other APIs. Once loaded, subsequent load calls with reload  false are a no-op.

translateServerIdToCloudId - Translates server ID to cloud ID. This is the basic lookup API that is used by fixing all kinds of integrations like URL, API, JQL, and 3P.

translateCloudIdToServerId - Translates cloud ID to server ID. This is the basic lookup API that is used by fixing all kinds of integrations like URL, API, JQL, and 3P.

Integration Services

Custom integration services are written on top of CTT Services. For example, URL Sanitisation, API Sanitisation, and JQL Sanitisation.

URL Sanitisation Service

This sanitises the Server URL to the cloud URL, meaning the translation of IDs from the URL is performed and outputs a cloud URL that can be directly used to access the data. All the integration services require basic CTTService for lookup and hence accept this as a parameter during construction. Refer to the unit tests for more use cases.

API Sanitisation Service

This sanitises the API to cater to cloud requests. This translates the API request URL as well as the request body.

JQL Sanitisation Service

Sanitises JQL query string by identifying and replacing all integer entity IDs. This uses an external library shipped as a Java executable(jar) in the libs folder.

Spring Boot Application and Controller

This is the RestController  for Spring App. It provides Rest APIs over all basic and integration APIs.

Integration Examples

Custom integrations can be built to solve specific use cases by leveraging the CTT APIs.

Fixing GithHub Jira Integration

GitHub integrates with Jira using Jira Automation rules over webhooks.

Configuration on Jira

• Create an Automation Rule with Incoming webhooks to accept connections from Github https://confluence.atlassian.com/automation/jira-automation-triggers-993924804.html

• Add required trigger, action, and condition to make changes to your Jira issues

• Export automation rules into a JSON file https://confluence.atlassian.com/automation/import-and-export-jira-automation-rules-1141480606.html

Configuration on Github

• On your GitHub repository, Create a webhook https://docs.github.com/en/webhooks

• Add the Jira webhook URL in the payload and add your credentials

• Select the required conditions for the trigger

After the configuration has been completed, on relevant events on GitHub, the Jira webhook will be triggered, and issues will be updated. After a cloud migration, these webhooks can be directly imported into your cloud instance. However, if the webhook uses a JQL query with integer identifiers, this can break. Let’s see how this can be fixed by using Cloud Transition Tools

Fixing Broken integration

• Deploy the CTT app on your local machine or any container. Refer https://bitbucket.org/atlassian/cloud-transition-tools/src/main/README.md for details on the setup

• Write a small script to modify the Webhook JSON files to make them work on your cloud instance.

# CTT App URL - Replace with your deployed instance URL
ctt_app_url  = "http://localhost:8080/ctt/rest/v1"

# Load mapping data
response = requests.post(ctt_app_url + "/load", params={
    "serverBaseURL": server_url,
    "reload" : False
}, auth=HTTPBasicAuth(username, password))

# Utility function to translate JQL
def translate_jql(jql):
    response = requests.get(ctt_app_url + "/jql-sanitise", 
    params={ "serverBaseURL": server_url,},
    json={ "jql": jql }, auth=HTTPBasicAuth(username, password))

    if response.status_code != 200:
        return None

    return response.json()['jql']

# Translate Server webhook to cloud
def translate_webhooks():
    with open('server_webhook.json') as f:
        json_data = json.loads(f.read())
        jql_value = json_data['rules'][0]['trigger']['value']['jql']

        # Translate the JQL
        cloud_jql = translate_jql(jql_value)

        # Dump the new JQL back to the file
        json_data['rules'][0]['trigger']['value']['jql'] = cloud_jql
        with open('cloud_webhook.json', 'w') as f:
            json.dump(json_data, f, indent=4)

Please refer to the video explaining how we can fix the integration using CTT: https://www.loom.com/share/3b4baf3b2fb74d2983a894867cbe98f2?sid=68b07a22-10b6-4e9e-9f58-ac9e0e8269bf

Contribution Guidelines

Refer: https://github.com/atlassian-labs/cloud-transition-tools/blob/main/CONTRIBUTING.md