This document describes how to investigate performance issues with mentionable and assignable functionality using JIRA stats logs - top mentionable & assignable users stats. Here you should be able to find if your permission scheme configuration is making calculating top mentionable and assignable expensive or not, what configuration changes can be done to make this functionality "lighter" on the system.

As with any other Jira Stats logs the usual steps during and investigation are:

• find last total (instance lifetime timeframe) stats on a given node to have a general overview on how a mentionable & assignable feature is performing
• in case of problems found in total stats, you narrow down the timeframe of the problem by searching the snapshot stats (5min timeframe)

How mentionable & assignable worked before 8.19.1?

Iterate through all users on instance matching the query (note the query could be empty and match all users) and only then on each user apply permission checks. With large number of users and non-trivial permission scheme configuration it could happen that the check had to be run on every user. Most of the time the permission check referred to data loaded present in caches, but it could also happen that the permission check was triggering a DB call for each user. This could result in very slow performance affecting mentionable & assignable functionality or even instance stability as even a single user could trigger (for every keystroke) a request (http thread) trying to process the permission check for thousands of users matching the query. 

How mentionable & assignable is working since 8.19.1?

With the 8.19.1 fix, we are always going to the DB to get top N users for given permission scheme configuration, i.e. the returned user list is already filter on the DB level with permissions. Now depending on your permission scheme configuration this can be achieved with 1 or more queries - but always limited/controlled number DB requests.

Let's look at mentionable & assignable users stats and what it can tell us:

{
	"legacy": {
		"findTopMentionableUsersInIssueLegacyMode": ...,
		"findTopAssignableUsersLegacyMode": ...,
	},
  	"optimised": {
    	"findTopMentionableUsers": ...,
		"findTopAssignableUsers": ...,
		"findTopUsersInternal": ...
	}
}		

The legacy.* are stats for the pre 8.19.1 behaviour and this should be empty in default configuration  >= 8.19.1. We can jump straight to the optimised.* section:

• findTopMentionableUsers - are stats for the method retrieving the top mentionable users
• findTopAssignableUsers - are stats for the method retrieving the top assignable users

Since both methods delegate to a common methods lets jump into findTopUsersInternal:

findTopUsersInternal": {
	"timeInMillis": ...,
	"unsupportedPermissionTypes": [],
	"resultSize": ...,
	"getPermissionSchemeIdFor": ...,
	"getSchemePermissions": ...,
	"findTopUsers": ...,
	"permissionSchemeLogic": ...,
	"findInIssueDefinedUsers": ...,
	"findTopUsersWithPermissionInIssue": ...,
	"findTopUsersWithPermissionInIssueFromGroups": ...
}

Let's see what are the times (findTopUsersInternal.timeInMillis)  spend in getting this list:

"findTopUsersInternal": {
      "timeInMillis": {
        "count": 6903,
        "min": 5,
        "max": 202,
        "sum": 63566,
        "avg": 9,
        "distributionCounter": {
          "5": 204,
          "10": 5610,
          "50": 1020,
          "100": 56,
          "500": 13,
          "1000": 0,
          "5000": 0,
          "10000": 0,
          "30000": 0
        }
      }
...
}

With the example above:

• method was called 6903 times
• on average it took 9 milliseconds
• 13 times out of 6903 it took between 100 and 500 milliseconds

... this means that on average getting the top list of mentionable users for a given query takes 9 milliseconds and only 0.2% is above 500milliseconds . 

Jira supports following permission types:

• group
• anyone
• user
• project role
• reporter
• project lead
• assignee
• user custom field
• group custom field
• application role

All those permission resolves at the end to: user names and/or group names.

In the stats you can find what type of permission types are being used on this instance:

"permissionSchemeLogic": {
        "anyUserType": 6038,
        "anyGroupType": 486,
        "allAllowed": 6832,
        "isGroupAny": 48,
        "isApplicationRoleAnyLoggedInUser": 6808,
        "nobodyAllowed": 0,

        "isIssueUserTypeReporter": 0,
        "isIssueUserTypeAssignee": 0,
        "isIssueUserTypeProjectLead": 0,

        "isUserTypeUser": 5774,
        "isUserTypeProjectRole": 486,
        "isUserTypeCustomField": 0,

        "isGroupTypeGroup": 23,
        "isGroupTypeProjectRole": 486,
        "isGroupTypeCustomField": 0,
        "isGroupTypeApplicationRole": 24
      }

• anyUserType - are permission types resolving to users
• anyGroupType - are permission types resolving to users via groups
• isGroupAny = "Group - Anyone on the web"
• isApplicationRoleAnyLoggedInUser = "Application access - Any logged in user"
• allAllowed = isGroupAny + isApplicationRoleAnyLoggedInUser
• isIssueUser* - users retrieved from issue object
• isUser* - specific user type permissions
• isGroup - specific group type permissions

Based on the data above you may want to investigate deeper user or group permission type queries:

• findTopUsersWithPermissionInIssue - user type permissions
• findTopUsersWithPermissionInIssueInGroups - group type permissions

Let's looks at the findTopUsersWithPermissionInIssueInGroups stats:

 "findTopMentionableUsersInIssueFromGroups": {
        "timeInMillis": ...,
        "resultSize": ...,
        "findTopUsersWithNameInGroups": {
          "timeInMillis": ...,
          ...
          "QueryDslJiraCrowdDAOSettings": {
            "batchSize": 1000,
            "maxTop": 100,
            "maxGroupNestedLevel": 20
          }
        },
        "findNestedGroupsOf": ...,
        "findGroupsFrom": {
          "timeInMillis": ...,
          "sqlCount": ...,
          "resultSize": ...,
          "findGroupsFromTypeGroup": ...,
          "findGroupsFromTypeProjectRoleGroup": ...,
          "findGroupsFromGroupCF": ...,
          "findGroupsFromSelectCF": ...,
          "findGroupsFromApplicationRole": ...,
        }
        "isNestedGroupsEnabledForAnyDirectory": ...
      }

It is possible to define a very expensive permission scheme configuration, a specially group based (check sqlCount - number of SQLs executions required to resolve a single findTopMentionableUsersInIssueFromGroups).

Let's assume we 15 levels of nested groups with hundreds of groups on each level and configure a permission scheme on the top level group with any directory in the system supporting nested groups.

This will results in the following queries being run for every mentionable search:

• get top level group (groups) from permission scheme configuration
• if there is at least one directory supporting nested groups search for all subgroups (currently stopping resolving groups when the nested level is > maxGroupNestedLevel - 20 by default); we are running an sql query for each nested level;
• we search for users in batched sets of groups (by default 1000) 
• by default we will only process up to numberOfBatches (default 1 batch = 1000) of groups

If your permission configuration scheme resolved to more then the maximum allowed number of groups, you should see the following waring message in the atlassian-jira.log:

WARN JiraCrowdDAO#findTopUsersWithNameInGroups: the results of the search might be incomplete due to too many groups.
     Number of groups: {}, batch size: {}, number of batches: {}. Allowed number of batches: {}.
     Learn more: https://confluence.atlassian.com/x/QBrSQ

What can I do with: WARN ...the results of the search might be incomplete due to too many groups...?

Please modify the configuration of the related permission scheme for: 

• mentionable: Permission Scheme → Browse Project
• assignable: Permission Scheme → Assignable

to resolve to a smaller (max: 1000 groups) number of groups.

For example: if this is a result of nested groups configuration you may want to point to a more specific group down the tree.