Rate Limiting restricting internal requests from the Confluence Data Center UI
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Rate limiting controls the number of external REST API requests that automation and users can make, as well as the frequency of those requests, ensuring that your Confluence Data Center instance remains stable.
Rate limiting targets only external REST API requests, which means that requests made within Confluence aren’t limited in any way. When users move around Confluence, creating pages, commenting, and completing other actions, they won’t be affected by rate limiting.
However, it has been observed that after enabling rate limiting, users are promptly rate-limited and receive HTTP 429 errors, as indicated by the HAR logs, when accessing certain screens, such as navigating spaces and pages, in Confluence.
Environment
Confluence Data Center 7.19.x
Diagnosis
After enabling rate limiting and capturing HAR logs during Confluence access, we observe HTTP 429 Too Many Requests errors when accessing various screens in Confluence.
Request URL:https://xxxxx/rest/menu/latest/appswitcher?_=1713963912409
Request Method:GET
Status Code:429 Too Many Requests
Remote Address:xx.xx.xx.xx:80
Request URL:https://xxxxx/rest/api/content/search?expand=container,metadata.currentuser.viewed,metadata.currentuser.lastcontributed&cqlcontext=%7B%22contentStatuses%22:%5B%22current%22,%22draft%22%5D%7D&cql=id%20in%20recentlyViewedContent(20,%200)&_=1713963912410
Request Method:GET
Status Code:429 Too Many Requests
Remote Address:xx.xx.xx.xx:80
Request URL:https://xxxxx/rest/experimental/search?cql=type=space%20and%20space.type=favourite%20order%20by%20favourite%20desc&expand=space.icon&limit=100&_=1713963912411
Request Method:GET
Status Code:429 Too Many Requests
Remote Address:xx.xx.xx.xx:80
Date:Wed, 24 Apr 2024 13:05:17 GMT
X-Ausername:xxxxxUpon reviewing the application logs after adding the additional class (com.atlassian.ratelimiting) with DEBUG level from Logging and Profiling, we observed that requests were immediately rate-limited as soon as rate limiting was activated.
2024-05-10 13:18:31,684 DEBUG [Caesium-1-1] [internal.cacheable.dao.CachingSystemRateLimitingSettingsProvider] tryReloadCache System settings reload not necessary (localVersion: 8, db version: Optional[8]), rate limiting mode is ON, fetch took 0 ms
2024-05-10 13:18:31,685 DEBUG [Caesium-1-1] [internal.cacheable.dao.CachingUserRateLimitingSettingsProvider] tryReloadCache User settings reload not necessary (local version: 35, db version: Optional[35]), fetch took 0 ms
2024-05-10 13:18:31,685 DEBUG [Caesium-1-1] [ratelimiting.internal.properties.SystemProperties] getOrDefaultValue System property 'com.atlassian.ratelimiting.whitelisted-url-patterns' not found
2024-05-10 13:18:31,686 DEBUG [Caesium-1-1] [ratelimiting.internal.properties.SystemProperties] getOrDefaultValue Defaulting system property 'com.atlassian.ratelimiting.whitelisted-url-patterns' to: '[]'
2024-05-10 13:18:31,686 DEBUG [Caesium-1-1] [ratelimiting.internal.properties.SystemProperties] getOrDefaultValue System property 'com.atlassian.ratelimiting.whitelisted-oauth-consumers' not found
2024-05-10 13:18:31,687 DEBUG [Caesium-1-1] [ratelimiting.internal.properties.SystemProperties] getOrDefaultValue Defaulting system property 'com.atlassian.ratelimiting.whitelisted-oauth-consumers' to: '[]'
2024-05-10 13:18:31,688 DEBUG [Caesium-1-1] [ratelimiting.internal.properties.AppLinkWhitelistedOAuthConsumers] getConsumers Whitelisting consumer key [jira:09e39103-188f-41a6-85bb-ba36cf04f7bf] for application link [JIRA] type [jira]
2024-05-10 13:18:31,688 DEBUG [Caesium-1-1] [ratelimiting.internal.properties.SystemProperties] getOrDefaultValue System property 'com.atlassian.ratelimiting.enable-pre-auth-filter' not found
2024-05-10 13:18:31,688 DEBUG [Caesium-1-1] [ratelimiting.internal.properties.SystemProperties] getOrDefaultValue Defaulting system property 'com.atlassian.ratelimiting.enable-pre-auth-filter' to: 'false'
.
.
2024-05-10 13:18:36,732 WARN [https-jsse-nio-8443-exec-16 url: /rest/menu/latest/appswitcher; user: xxxx] [ratelimiting.internal.filter.RateLimitFilter] lambda$userHasBeenRateLimited$0 User [8a91c8b97291a358017293c9cb5c0000] has been rate limited
-- url: /rest/menu/latest/appswitcher | traceId: 3d69518720da4fd2 | userName: WX10635
2024-05-10 13:18:36,733 DEBUG [https-jsse-nio-8443-exec-16 url: /rest/menu/latest/appswitcher; user: xxxx] [internal.requesthandler.logging.RateLimitedRequestLogger] logRateLimitedRequest User [8a91c8b97291a358017293c9cb5c0000] has been rate limited for URL [https://xxxxx/rest/menu/latest/appswitcher]Cause
Since rate limiting targets only external REST API requests, requests made within Confluence should not be limited. However, according to the existing implementation, a request is considered an internal request (UI request) if it contains at least two of the following:
- “Referer” header
- “Origin” header
- “JSESSIONID” cookie
- “SERAPH” cookie
If the request does not contain at least two of the specified headers or cookies, it will be classified as an external request and will be rate-limited according to the configuration set for rate limiting in the Confluence UI.
Solution
Carefully review the HAR logs to confirm whether the requests contain at least two of the specified headers or cookies, which are necessary for them to be classified as internal requests (UI requests).
In the below scenario, the requests only contained JSESSIONID cookie and missing other headers/cookie.
request": {
"method": "GET",
"url": "https://xxx.xx.xxx/rest/experimental/search?cql=type=space%20and%20space.type=favourite%20order%20by%20favourite%20desc&expand=space.icon&limit=100&_=1715340209690",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Accept",
"value": "application/json, text/javascript, */*; q=0.01"
},
.
.
.
{
"name": "Cookie",
"value": "_ga=GA1.2.1695956113.1710925714;
ajs_anonymous_id=%22894a712a-e684-43a9-86dc-28d270f74f30%22;
confluence.browse.space.cookie=space-templates;
_vid_t=tusuL0EJeZMZahiAlGgMRyaHAIozWfs+e3mavmdT2WLQWXBDYQooKE/lZ9As+keszSeFbhYvVfXSBQ==;
mywork.tab.tasks=false;
confluence.list.pages.cookie=list-content-tree;
_ga_VDT4TT7WFW=GS1.2.1714481248.9.0.1714481248.60.0.0;
JSESSIONID=15555C14E157FFC82DA767BD84EF7141"
}While reviewing the reverse proxy configuration, it was found that the absence of a “Referer” header in the requests is due to the following configuration in Nginx:
add_header Referrer-Policy "no-referrer" always; By commenting out the above configuration in Nginx, the request will include the JSESSIONID cookie and the Origin header, and therefore, it was not classified as an external request on enabling Rate Limiting.
Similarly, other headers and cookies can be checked to determine why certain requests are being rate-limited despite being an internal requests according to the existing logic.
In case of issues or concerns to follow this procedure, please reach out to the Atlassian Support team and share the information that's been gathered so far along with a fresh Support zip file.