Bamboo Data Center ephemeral agent refuses to start with error message "pods is forbidden"
Platform Notice: Data Center - This article applies to Atlassian products on the Data Center platform.
Note that this knowledge base article was created for the Data Center version of the product. Data Center knowledge base articles for non-Data Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
The steps outlined on this article are provided AS-IS. This means we've had reports of them working for some customers — under certain circumstances — yet are not officially supported, nor can we guarantee they'll work for your specific scenario.
You may follow through and validate them on your own non-prod environments prior to production or fall back to supported alternatives if they don't work out.
We also invite you to reach out to our Community for matters that fall beyond Atlassian's scope of support!
Summary
Bamboo job failed to start on an ephemeral agent with below error message on cluster communication log.
Error from server (Forbidden): pods is forbiddenEnvironment
- This issue and the solution has been tested on Bamboo Data Center 9.4.1 and 9.3.2
- This happened on an Openshift Kubernetes cluster 4.12
Diagnosis
The cluster communication logs reveal an "Unauthorized" message when trying to create the agent pod. To access these logs, navigate to the Bamboo administration page. From there, go to the Ephemeral agents section and select Pods. On the Manage pods and ephemeral agents page, you can find the cluster communication log.
Jan 16, 2024, 2:03:26 PM Launching Ephemeral Agent pod eph-ephe-job1-6-flfhpdkq for EPHEMERAL - EPHE - Default Job #6 (EPH-EPHE-JOB1-6)
error: You must be logged in to the server (error when creating "/opt/atlassian/atlassian-bamboo-9.4.1/temp/pod1931317460300020553.yaml": Unauthorized)
Jan 16, 2024, 2:03:54 PM Launching Ephemeral Agent pod eph-ephe-job1-6-qgblqhra for EPHEMERAL - EPHE - Default Job #6 (EPH-EPHE-JOB1-6) [2. attempt of 3]
error: You must be logged in to the server (error when creating "/opt/atlassian/atlassian-bamboo-9.4.1/temp/pod14230423914922192937.yaml": Unauthorized)
Jan 16, 2024, 2:04:24 PM Launching Ephemeral Agent pod eph-ephe-job1-6-njxxvsbf for EPHEMERAL - EPHE - Default Job #6 (EPH-EPHE-JOB1-6) [3. attempt of 3]
error: You must be logged in to the server (error when creating "/opt/atlassian/atlassian-bamboo-9.4.1/temp/pod11931554224856638448.yaml": Unauthorized)
Jan 16, 2024, 2:04:24 PM The Ephemeral Agent's pod eph-ephe-job1-6-njxxvsbf for EPHEMERAL - EPHE - Default Job #6 (EPH-EPHE-JOB1-6) failed to start - dropping launch request
Jan 16, 2024, 2:15:42 PM Launching Ephemeral Agent pod eph-ephe-job1-7-alphnzop for EPHEMERAL - EPHE - Default Job #7 (EPH-EPHE-JOB1-7)The below error message is also seen on the <bamboo-home>/logs/atlassian-bamboo.log file. The service account "system:serviceaccount:default:bamboo" is forbidden from listing pods.
2024-01-16 14:19:06,156 WARN [http-nio-8085-exec-3] [KubernetesExecutor] Cannot perform KubernetesExecutor#getPods: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:bamboo" cannot list resource "pods" in API group "" in the namespace "default"
2024-01-16 14:19:16,293 WARN [http-nio-8085-exec-24] [KubernetesExecutor] Cannot perform KubernetesExecutor#getPods: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:bamboo" cannot list resource "pods" in API group "" in the namespace "default"
2024-01-16 14:19:26,432 WARN [http-nio-8085-exec-11] [KubernetesExecutor] Cannot perform KubernetesExecutor#getPods: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:bamboo" cannot list resource "pods" in API group "" in the namespace "default"
2024-01-16 14:19:36,592 WARN [http-nio-8085-exec-12] [KubernetesExecutor] Cannot perform KubernetesExecutor#getPods: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:bamboo" cannot list resource "pods" in API group "" in the namespace "default"
2024-01-16 14:19:46,729 WARN [http-nio-8085-exec-8] [KubernetesExecutor] Cannot perform KubernetesExecutor#getPods: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:bamboo" cannot list resource "pods" in API group "" in the namespace "default"
2024-01-16 14:19:56,902 WARN [http-nio-8085-exec-14] [KubernetesExecutor] Cannot perform KubernetesExecutor#getPods: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:bamboo" cannot list resource "pods" in API group "" in the namespace "default"
2024-01-16 14:20:07,043 WARN [http-nio-8085-exec-4] [KubernetesExecutor] Cannot perform KubernetesExecutor#getPods: Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:bamboo" cannot list resource "pods" in API group "" in the namespace "default"
Cause
The service account used here has not been provided with required privilege to create pods in the default namespace.
Please also note that namespace could have been modified in the Ephemeral agent template by adding namespace: bamboo-ephemeral to the metadata section.
metadata:
name: '{{NAME}}'
namespace: bamboo-ephemeral
labels:
'{{RESOURCE_LABEL}}': <value>
From <bamboo-home>/logs/atlassian-bamboo.log file, we can see this user, "system:serviceaccount:default:bamboo". This is a service account named bamboo in the default namespace.
Use the command below to find out the role assigned to the service account. In this case, there was no output which explains why the service account had no permission to create a pod in the default namespace.
oc get rolebindings -o json | jq -r '
.items[] |
select(
.subjects // [] | .[] |
[.kind,.namespace,.name] == ["ServiceAccount","default","bamboo"]
) |
.metadata.name'
Solution
Assign the edit role to the service account in question.
oc project default #Switch to the default namespace
oc get sa bamboo #Confirm the presence of the service account in the current namespace
oc adm policy add-role-to-user edit -z bamboo #assign the edit role to service account
Now you can check the role bindings assigned to the user using the below
oc get rolebindings -o json | jq -r '
.items[] |
select(
.subjects // [] | .[] |
[.kind,.namespace,.name] == ["ServiceAccount","default","bamboo"]
) |
.metadata.name'
edit
Now try running the build again using the ephemeral agent and it should work fine.