__Searching in JIRA
New Searching
The way a search is performed in JIRA has significantly changed. The introduction of advanced searching (JQL) necessitated a rewrite of the JIRA searching subsystem. In the process, the API for searching has also been changed (and improved) significantly. Unfortunately these changes will almost certainly mean that plugins that search will need to be updated for JIRA 4.0.
In JIRA 3.x and earlier, searching was achieved using a SearchRequest in combination with SearchParameters and SearchSorts. While the SearchRequest still continues to exist in JIRA 4.0, the SearchParameters have been replaced with the Query object.
/**
* The representation of a query.
*
*/
public interface Query
{
/**
* @return the main clause of the search which can be any number of nested clauses that will make up the full
* search query. Null indicates that no where clause is available and all issues should be returned.
*/
Clause getWhereClause();
/**
* @return the sorting portion of the search which can be any number of
* {@link com.atlassian.query.order.SearchSort}s that will make up the full order by clause. Null indicates that
* no order by clause has been entered and we will not sort the query, empty sorts will cause the default
* sorts to be used.
*/
OrderBy getOrderByClause();
/**
* @return the original query string that the user inputted into the system. If not provided, will return null.
*/
String getQueryString();
}
The Query is JIRA's internal representation of a JQL search. It contains the search condition (i.e. the "where" clause) and the search order (i.e. the "order by" clause). The Query object can be created using the JqlQueryBuilder. For example, to create a query "find all issues assigned to either Dylan or Tokes that are unresolved and due in the next week" you would call:
final JqlQueryBuilder builder = JqlQueryBuilder.newBuilder();
builder.where().assignee().in("Dylan", "Tokes").and().unresolved().and().due().lt().string("+1w");
builder.orderBy().dueDate(SortOrder.ASC);
Query query = builder.buildQuery();
Once the Query has been obtained, it can be used to execute a search. In JIRA 4.0 a new SearchService has been added to provide a central location for Query related operations. To run the search you can simply call SearchService.search() as documented on the SearchService. The SearchProvider is still available for those who need to control the finer details of searching.
The Query object is immutable; once it is created it cannot be changed. The JqlQueryBuilder represents the mutable version of a Query object. The JqlQueryBuilder can be primed with an already existing Query by calling JqlQueryBuilder.newBuilder(existingQuery).
In JIRA 3.x the SearchRequest was the object that was passed to the searching system to perform a search. The Query object has taken over this role in JIRA 4.0; the SearchProvider and SearchService now take in Query objects rather than SearchRequests. The SearchRequest object has been reworked in JIRA 4.0 to significantly reduce its responsibility. For instance, ordering information is now stored on the Query object rather than on the SearchRequest object. The SearchRequest really represents a saved search (aka. filter). You should only need to deal with SearchRequests if you are working with filters. Even in this case, all searching operations need to be performed on Query objects by calling SearchRequest.getQuery().
It is often necessary to get a URL for a particular Query. The SearchService provides the getQueryString(query) method for this. The method returns a parameter snippet of the form jqlQuery=<jqlUrlEncodedQuery>, which can be appended safely to an existing URL that points at the Issue Navigator. Note that the links that JIRA 4.0 generates are JQL based, so are incompatible with JIRA 3.x and before. Old valid JIRA 3.x URLs will still work with JIRA 4.0.
Given a Query object it is possible to retrieve its JQL representation by calling either getGeneratedJqlString(query) or getJqlString(query) on the SearchService. The service makes sure that any values in the Query that need to be escaped are handled correctly. Importantly, the Query.toString() method does not return valid JQL (on purpose).
The SearchService.parseQuery(jqlString) method can be used to turn a JQL string into its Query representation. The return from this method has details on any parse errors encountered.
A Query object, especially those parsed directly from the user, may not be valid. For example, the user may be trying to find issues in a status that does not exist. The SearchService.validateQuery(query) method can be used to see if a particular Query object is valid. Errors are returned with messages that can be displayed to the user. Executing an invalid Query will not result in any errors and in fact may return results. To run an invalid query, JIRA will just make the invalid conditions equate to false and run the query. For example, searching for status = "I don't Exist" or user = bbain will result in the query <false> or user = bbain actually being run.
There are some methods on the SearchService that we did not discuss here. Check out documentation on the SearchService for more information.
Examples
Here's a complete example how to obtain search results for the query "project is JRA and the reporter is the currently logged in user and custom field with id 10490 contains 'xss'":
String jqlQuery = "project = JRA and reporter = currentUser() and cf[10490] = xss";
final SearchService.ParseResult parseResult =
searchService.parseQuery(authenticationContext.getUser(), jqlQuery);
if (parseResult.isValid())
{
try
{
final SearchResults results = searchService.search(authenticationContext.getUser(),
parseResult.getQuery(), PagerFilter.getUnlimitedFilter());
final List<Issue> issues = results.getIssues();
}
catch (SearchException e)
{
log.error("Error running search", e);
}
}
else
{
log.warn("Error parsing jqlQuery: " + parseResult.getErrors());
}The preceding search could have also been written using the QueryBuilder:
final JqlQueryBuilder builder = JqlQueryBuilder.newBuilder();
builder.where().project("JRA").and().reporterIsCurrentUser().and().customField(10490L).eq("xss");
Query query = builder.buildQuery();
try
{
final SearchResults results = searchService.search(authenticationContext.getUser(),
query, PagerFilter.getUnlimitedFilter());
final List<Issue> issues = results.getIssues();
}
catch (SearchException e)
{
log.error("Error running search", e);
}Plugging into JQL and what happened to my Custom Field Searchers
The introduction of advanced searching (JQL) necessitated a rewrite of the JIRA searching subsystem. Unfortunately these changes will certainly mean that any CustomFieldSearchers will need to be updated to work in 4.0.
The most fundamental change is that all JIRA 4.0 searching is implemented using JQL. A JQL search consists of two components: firstly, a number of conditions, or Clauses, that must be matched for an issue to be returned; and secondly, a collection of search orderings that define the order in which the issues should be returned. The Query object is JIRA's internal representation of a search. It is now the responsibility of the CustomFieldSearcher to take a relevant Query, validate its correctness and generate a Lucene query to find issues that match it. By doing this your custom field becomes searchable using JQL.
The CustomFieldSearcher and/or the custom field is also responsible for ordering results if the order in the search includes the custom field. If your custom field ordered correctly in JIRA 3.x, then it will order correctly in JIRA 4.0. While the internal representation of an order has changed in JIRA 4.0, it still uses the same interfaces to order the search results. We will not address ordering again.
What is a JQL Clause?
A custom field must process the Clauses from a JQL search to integrate into JQL. Each Clause consists of a number of conditions (e.g. abc != 20) combined by the AND and OR logical operators (e.g. abc = 20 AND (jack < 20 OR jill > 34). In JIRA a condition is represented by a TerminalClause, the logical AND by an AndClause and a logical OR by an OrClause, all of which implement the Clause interface. Finally, the logical NOT operator can be used to negate any other Clause. It is represented by a NotClause that also implements Clause. These Clause objects are composed together to represent a complex conditions. For example, the condition abc = 20 AND NOT(jill > 34 OR NOT jack < 20) is represented by the following tree:
A Clause can be navigated by passing an instance of a ClauseVisitor to the accept method of a Clause. This follows the traditional visitor pattern.
The TerminalClause represents a Clause of the form "field operator value". Inside the TermincalClause the "operator" is one of the values from Operator enumeration while the "value" is represented as an Operand. An Operand can represent a single value (e.g. field = "single"), a list of values (e.g. field in ("one", 1235)), a function (e.g. field = function(arg1, arg2)) or EMPTY (e.g. field is EMPTY). In the end, all you want is the values from the Operand. These can be obtained as a list of QueryLiteral (see below) by calling JqlOperandResolver.getValues(). The JqlOperandResolver also has the isEmptyOperand, isFunctionOperand, isListOperand and isValidOperand methods that can be used to determine the type of the Operand.
A QueryLiteral represents either a String, Long or EMPTY value. These three represent JQL's distinguishable types. It is up to the CustomFieldSearcher to convert these values into something that makes sense to it. The type of a QueryLiteral can be determined by calling its isEmpty, getLongValue or getStringValue methods. The get methods will return null or false when the method and the QueryLiteral type do not match.
Integrating with JQL
In JIRA 3.x a CustomFieldSearcher was the way to provide customized searching functionality for custom fields. In JIRA 4.0 it is still the plugin point for searching; however, the CustomFieldSearcher interface has changed significantly to accommodate the introduction of JQL. One of the major changes is that the CustomFieldSearcher must return a CustomFieldSearcherClauseHandler in JIRA 4.0. This object is a composition of a ClauseValidator and a ClauseQueryFactory.
The ClauseValidator is used by JIRA to ensure that a JQL query is valid according to the CustomFieldSearcher.
/**
* Validates a clause and adds human readable i18n'ed messages if there is a problem.
*
* @since v4.0
*/
public interface ClauseValidator
{
/**
* Validates a clause and adds human readable i18n'ed messages if there is a problem.
*
* @param searcher the user who is executing the search.
* @param terminalClause the clause to validate.
*
* @return an MessageSet that will contain any messages relating to failed validation. An empty message set must
* be returned to indicate there were no errors. null can never be returned.
*/
@NotNull
MessageSet validate(User searcher, @NotNull TerminalClause terminalClause);
}
It is up to the validator to ensure that the operator and the value from the passed TerminalClause makes sense for the CustomFieldSearcher and its associated custom field. Any errors can be placed in the returned MessageSet. They should be internationalised with respect to the passed user.
The validate method must always return a MessageSet as its result. A null return is not allowed. A MessageSet is an object that contains all of the errors and warnings that occur during validation. All messages in the MessageSet need to be translated with respect to the passed searching user. An empty MessageSet indicates that no errors have occurred. A MessageSet with errors indicates that the JQL is invalid and should not be allowed to run. The returned messages will be displayed to the user so that any problems may be rectified. A MessageSet with warnings indicates that the JQL may have problems but that it can still be run. Any warning messages will be displayed above the results.
The ClauseValidator does not need to check if the passed TerminalClause is meant for the for it, JIRA makes sure that it only passes TerminalClauses that the ClauseValidator is meant to process. It does that by only passing TerminalClauses whose "field" matches one of the names the custom field must handle.
ClauseValidators need to respect JIRA security. A ClauseValidator should not leak information about JIRA objects that the searcher does not have permission to use. For example, a ClauseValidator should not differentiate between an object not existing and an object that the user has no permission to see. A ClauseValidator that behaves badly will not cause JQL to expose issues that the searcher is not allowed to see (since JQL does permission checks when it runs the filter), though it does open up an attack vector for information disclosure.
The ClauseValidator must be thread-safe and re-entrant to ensure correct behavior. JIRA will only create one instance of the ClauseValidator per custom field instance. This means that multiple threads may be calling the validator at the same time.
The ClauseQueryFactory is used by JIRA to generate the Lucene search for a JQL Clause.
public interface ClauseQueryFactory
{
/**
* Generates a lucene query for the passed {@link TerminalClause}....
*
* @param queryCreationContext the context of the query creation call; used to indicate that permissions should be
* ignored for "admin queries"
* @param terminalClause the clause for which this factory is generating a query.
* @return QueryFactoryResult contains the query that lucene can use to search and metadata about the query. Null
* cannot be returned.
*/
@NotNull
QueryFactoryResult getQuery(@NotNull QueryCreationContext queryCreationContext, @NotNull TerminalClause terminalClause);
}
It is the responsibility of the ClauseQueryFactory to create the Lucene search for the passed TerminalClause and QueryCreationContext. The generated Lucene search is returned in the QueryFactoryResult. The result contains the search (a Lucene Query object which is not related the the JQL Query object) and a flag to indicate whether or not the Lucene search should be negated. When set to true, JIRA will actually only match issues that do not match the returned Lucene search. For example, a ClauseQueryFactory may decide to implement a condition like field != value by returning a Lucene search that matches field = value and setting the flag to true. You can also implement this condition by returning a Lucene search that matches field != value and setting the flag to false.
The new argument here is the QueryCreationContext. This object contains the variables that may be necessary when creating the query. The QueryCreationContext.getUser method returns the user that is running the search and as such should be used to perform any security checks that may be necessary. The QueryCreationContext.isSecurityOverriden method indicates whether or not this function should actually perform security checks. When it returns true, the factory should assume that the searcher has permission to see everything in JIRA. When it returns false, the factory should perform regular security checks.
A ClauseQueryFactory should try to limit the queries so that issues that the user cannot see are excluded. Consider the query affectsVersion = "1.0". The ClauseQueryFactory might detect that there are two versions named "1.0", one from project1 and the other from project2. The factory might then notice that the user doing the search cannot see project1. The factory can then return a query that contains only the version from project2. This is mainly an efficiency concern as JIRA filters all search results to ensure users cannot see issues they are not allowed to.
The ClauseQueryFactory does not need to check if the passed ClauseQueryFactory is meant for it; JIRA makes sure that it only passes TerminalClauses that the ClauseQueryFactory is meant to process. It does that by only passing TerminalClauses whose "field" matches one of the JQL names the custom field must handle. Put simply, the ClauseQueryFactory must handle any passed TerminalClause.
The ClauseQueryFactory must also handle the situation when an invalid TerminalClause is passed to it. An invalid TerminalClause is one whose associated ClauseValidator would not validate. The ClauseQueryFactory must return an empty Lucene search if the passed TerminalClause is invalid. Most importantly, the ClauseQueryFactory must not throw an exception on an invalid TerminalClause.
A ClauseQueryFactory needs to be careful when implementing any of the negating operators (i.e. !=, !~, "not in"). These operators should not match what is considered empty by the custom field and CustomFieldSearcher. For example, the JQL query resolution is EMPTY will return all unresolved issues in JIRA. The query resolution != fixed will only return all resolved issues that have not been resolved as "fixed", that is, it will not return any unresolved issues. The user has to enter the query resolution != fixed or resolution is EMPTY to find all issues that are either unresolved or not resolved as "fixed".
A ClauseQueryFactory also needs to consider field visibility. A CustomFieldSearcher should not match any issues where its associated custom field is not visible. Importantly, asking for EMPTY should not match issues where the custom field is not visible. For example, the JQL query resolution is EMPTY will not return issues from a project whose resolution field has been hidden. A hidden field is assumed not to exist.
There are some extra interfaces that the CustomFieldSearcherClauseHandler may also implement to provide optional functionality to the searching subsystem:
- ValueGeneratingClauseHandler: Gives the
CustomFieldSearcherthe ability to suggest some values during JQL entry auto-complete. This is really only useful for custom fields whose values come from an allowable finite set. - CustomFieldClauseSanitiserHandler: Gives the
CustomFieldSearcherthe ability pre-process the query and remove sensitive information from the query before it is displayed to the passed user. - CustomFieldClauseContextHandler: Gives the
CustomFieldSearcherthe ability to customize JIRA's query context calculation. This interface is best left alone, unexplained and unimplemented.