Summary

Sometimes when submitting external imports into Assets you may run into 429 errors which relate to rate limiting as the problem. But how do you go about solving or working around this type of behavior? 

Assets Rate Limiting Scenario

All endpoints are rate-limited to 1,000 requests per minute per instance with a total limit of 10,000 requests per minute per instance. Any exceptions to this rule will be mentioned in the documentation of the individual endpoints. With this in mind, we essentially put the instance into "maintenance mode" which can return both a 429 and a 503. In this scenario, we are doing a deployment but it is not triggering the reindex and when the system receives the import outside of business hours (based on the instance’s time zone) it will reindex. This then causes the first request (and others that are made) to fail since it's outside of working hours to re-try the request. We do have a related Feature Requests for including "Retry-After" and "X-RateLimit" headers for Assets REST API as well:

• JSDCLOUD-11605 - Getting issue details... STATUS

Environment

You are using a Sandbox or a Product environment that does not have a lot of Objects or Asset activity in it already. You then import Asset Objects using the REST API to import many Objects

Cause

This is ultimately because the instance being used is internally set to small instead of large. The difference between the two is the number of objects that are in the instance. In small instances, they auto-scale to account for large imports, while large instances rarely do this as they can account for large Object imports.

Additionally, this happens when there are deployments of code updates at the same time an external import is being sent in, which will cause 429 errors in this scenario. That said, the initial job returns 429 because it is the first request after the restart and the instance has to reindex. Since the instance is small it's getting reshuffled, causing a restart and triggering a reindex when the import has started. 

Solution

The best (and first) solution in this scenario is to first reach out to Atlassian Support (https://support.atlassian.com/contact/#/) and raise a request if you see intermittent 429 errors with your External Imports. Our development team can then update the instance from a small to a large one, accounting for the larger imports that are being executed.

If this continues to happen, a workaround is to call another endpoint before import and retry until that returns 200, or go to the Assets main page from the browser to trigger the reindex, and then start the import as before. 

If the imports that are erroring are outside of business hours you could try to configure a try-close block in your import script, so that every 10 seconds, retry the external import until it 200s. Additionally, you would want to implement a limit into the script so it will stop retrying at like 20 retries, for example. Please see the example below as a reference.

Example Python script snippet to use as a try-close solution with Rate Limiting:

...
if request.status_code == 429 or count == 5:
    while request.status_code == 429:
        request_check = request.get(url, headers=headers)
        if request_check.status_code == 429:
            time.sleep(10)
            count += 1
        elif request_check.status_code == 200:
            break
...

Relevant Documentation

• Assets Import REST API