How to test and validate Bitbucket Secret Scanning rules
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bitbucket Data Center has default rules to test against a secret that is mistakenly included in the files committed to Bitbucket.
Other than that, Bitbucket also allow adding new rule and give it a custom name and the new rule at the Secret Scanning feature at the project or repository level.
In this article, we share some guidance on how to test these secret scanning rules when there is a need for testing and validating whether the custom user-created rule or default Bitbucket secret scanning rule works.
Environment
Bitbucket Data Center 8.3+
Diagnosis
The user adds a secret such as below into the code but no notifications are sent even though there is a secret in the code line.
Example : $headers = @{'Authorization': 'Basic c3ZjLUlOTkgxNDUxMTA6QW9uT25lRGV2b3BzQDEyMw==';'x-atlassian-token'='no-check'}
Solution
Bitbucket uses RE2J for the regex rule for secret scanning.
Below are some useful external tools that you could use for testing:-
Please note that Atlassian does not provide support for the external regex tester tool. It is added for reference only
There is a difference between Bitbucket scanner and regex tool. Using the regex tool above, it tries to find the occurrence of a regex pattern within a given string. However, in certain version of Bitbucket it tries to match the whole string in a line against the custom/default pattern in a line which means the line should not contain any other pre or post-characters that wraps the secret in the line for it to pass.
Email notifications are sent out to everyone involved in the commit history of the secret: the authors, committers, and the developer who pushed or merged the code containing secrets into the repositories. Therefore, ensure that mail notifications feature is setup successfully for the user who is testing out secret scanning.
For example a sample below on Git Personal Access Token rule, matches the default Bitbucket rule \b(?i)glpat-[0-9a-z\-]{20}\b and email notifications will be sent out. If you would like to receive an email immediately, you will need to change the email notifications to be sent immediately in the user profile.
$gitLabAPIKey="glpat-XFZAzGLyz8Jjy1nw4XPB"
If we use the tool Regex tester for Golang, we can place the Test String as $gitLabAPIKey="glpat-XFZAzGLyz8Jjy1nw4XPB" and Regular Expression as \b(?i)glpat-[0-9a-z\-]{20}\b
The tool will show that test string and pattern matches and the same for Bitbucket, as the whole string matches the rule pattern.
Resolution to the diagnosis case
On detecting Basic Authentication secret in code, the default rule in Bitbucket is the pattern (\"|')?Basic [A-Za-z0-9\\+=].
It will not match below test string, because the test string contains a pre and post characters that wraps the secret.
Example : $headers = @{'Authorization': 'Basic c3ZjLUlOTkgxNDUxMTA6QW9uT25lRGV2b3BzQDEyMw==';'x-atlassian-token'='no-check'}
The above will work if the Line Pattern rule is tweaked to .*(\"|')?Basic ([A-Za-z0-9\\+=]{44}(\"|'))?.* by adding .* to the pre and post of the initial Line pattern so that it will match the test string above.
Changes in newer Bitbucket version
This ticket
-
BSERV-14161Getting issue details...
STATUS
addresses the above diagnosis and in newer version of Bitbucket, the secret scanning feature will match the secret found in any part of an entire code line.