How to enable Client IP Forwarding For SSH Sessions by setting up Proxy protocol for Bitbucket Data Center.
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
According to Install Bitbucket Data Center, for setting up a clustered version of Bitbucket Data Center, you need to establish a load balancer responsible for balancing requests between nodes using the same base URL.
The specificity of the load balancer is that it behaves exactly as a reverse proxy by replacing the headers of the incoming request with its own.
In the case of HTTPS, which operates on the 7th - Application layer of the OSI model, you can solve the issue with headers overwriting by using the option forwardfor or analog for other load balancers. It will make the load balancer send headers with both values, which Bitbucket can correctly write into access and audit logs.
For SSH, the load balancing is performed on the 4th - Transport layer, which doesn't have any headers so that the application node of Bitbucket can get only the IP address from which it received the request. In our context, such a server will be precisely the load balancer. As a result, all the entries for incoming requests will be written into access and audit logs of Bitbucket with the IP address of the load balancer. It makes it difficult to comprehensively analyze the incoming traffic for SSH.
The solution came with Proxy protocol, which adjusts the connection string from the load balancer to the application node in Bitbucket with additional information:
PROXY_STRING + single space + INET_PROTOCOL + single space + CLIENT_IP + single space + PROXY_IP + single space + CLIENT_PORT + single space + PROXY_PORT + "\r\n"
#The following is an example of the proxy protocol line for IPv4
PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n
You can find more details on the PROXY protocol implementation in the following sources:
Environment
Bitbucket Data Center 7.20.x and newer
HAProxy 1.5-dev3 and newer
AWS Network Load balancer
Solution
Bitbucket part:
Bitbucket Data Center 7.20 and newer have PROXY protocol enabled by default.
However, you can always set it explicitly in $bitbucket_home/shared/bitbucket.properties:
plugin.ssh.haproxy.proxy-enabled=true
That's the only requirement from the Bitbucket side. No further configuration is required. If that doesn't work, the issue originates from the load balancer itself.
Load balancer part - HAProxy scenario
To configure the PROXY protocol with the HAProxy 1.5-dev3 and newer, you need to enable the protocol support (send-proxy) for each cluster node in haproxy.cfg section for SSH backend:
backend bitbucket_ssh_backend
mode tcp
balance roundrobin
server bitbucket01 163.25.93.240:7999 check send-proxy port 7999
server bitbucket02 192.128.0.220:7999 check send-proxy port 7999
timeout server 15m
Restart HAProxy after the changes and try the clone once again.
After the changes are successfully applied, you will see double IP addresses in access logs:
CLIENT_IP,LOAD_BALANCER_IP | ssh | i@DX7BUKx631x44878x0 | username | 2022-05-25 10:31:38,622 | SSH - git-upload-pack '/demo/repo.git' | "SSH-2.0-OpenSSH_8.6" | - | - | - | - | - | 17i1kgv |
CLIENT_IP,LOAD_BALANCER_IP | ssh | o@DX7BUKx631x44878x0 | username | 2022-05-25 10:31:41,300 | SSH - git-upload-pack '/demo/repo.git' | "SSH-2.0-OpenSSH_8.6" | 0 | 727 | 22705297 | cache:hit, clone, protocol:2, ssh:user:id:2 | 2678 | 17i1kgv |
`
Load balancer part - AWS Network Load balancer scenario
In the case of AWS, only Network Load Balancer supports PROXY protocol version 2.
To enable proxy protocol v2 using the new console:
- Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
- On the navigation pane, under LOAD BALANCING, choose Target Groups.
- Choose the name of the target group to open its details page.
- On the Attributes tab, select Edit.
- On the Edit attributes page, select Proxy protocol v2
- Choose Save changes.
To enable proxy protocol v2 using the old console:
- Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
- On the navigation pane, under LOAD BALANCING, choose Target Groups
- Select the target group
- Choose Description, Edit attributes.
- For Proxy protocol v2, choose Enable.
- Choose Save.
For AWS-CLI please use modify-target-group-attributes withp proxy_protocol_v2.enabled
For more up-to-date information about proxy protocol support for AWS NLB please check https://docs.aws.amazon.com/elasticloadbalancing/latest/network/elb-ng.pdf
For Citrix Netscaler, please check the following guide https://support.citrix.com/article/CTX224265/how-to-configure-netscaler-to-send-proxy-protocol-to-backend-servers (Atlassian doesn't provide any assistance with Netscaler. Please reach official Netscaler support).