BuildKit on Kubernetes: agent reconnection + ephemeral storage eviction

[BeckerTim]

Hi! I’m evaluating the Blue Agent Helm chart for building container images on Kubernetes and I’ve hit two issues.

Environment

Chart version: 12.0.1
Image: ghcr.io/clemlesne/blue-agent: (bookworm flavor)

Kubernetes: Client Version: v1.29.1
Server Version: v1.27.10+rke2r1

StorageClass: csi-sc-cinder
KEDA: not activated

1) Azure Pipelines agent does not reconnect after first successful job

After the first job completes, the agent does not reconnect to the DevOps server. The default start.sh seems to exit after the initial run, so I mounted a custom script via ConfigMap to keep it running. With that change, the agent does not reconnect and stays alive.

Custom start.sh:
`

#!/bin/bash
set -e
export AGENT_ALLOW_RUNASROOT="1"

bash config.sh \
  --acceptTeeEula \
  --agent "$AZP_AGENT_NAME" \
  --auth PAT \
  --pool "$AZP_POOL" \
  --replace \
  --token "$AZP_TOKEN" \
  --unattended \
  --url "$AZP_URL" \
  --work "$AZP_WORK" & wait $!

bash run-docker.sh "$@" & wait $!

`

If the reconnection behavior is intentional, what could be my problem not being able to reconnect?

2) Pod evictions due to ephemeral storage, despite BuildKit PVC

I’m seeing evictions with:

ephemeral local storage usage exceeds the total limit of containers

Bumping the ephemeral limit helps temporarily but isn’t ideal since there is not much space on the nodes (50Gi). I tried moving BuildKit’s state to a volume per docs. (Side note: the docs link currently appears broken: link.)

`

pipelines:
  cache:
    volumeEnabled: true
    size: 50Gi
    type: csi-sc-cinder
  tmpdir:
    volumeEnabled: true
    size: 20Gi
    type: csi-sc-cinder

extraVolumes:
  - name: docker-config
    secret:
      secretName: harbor-registry
      items:
        - key: .dockerconfigjson
          path: config.json
  - name: start-sh
    configMap:
      name: start-sh
      items:
        - key: start-sh
          path: start.sh
          defaultMode: 0755
  - name: buildkitd
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          storageClassName: csi-sc-cinder
          resources:
            requests:
              storage: 100Gi
          # volumeMode omitted (defaults to Filesystem)

extraVolumeMounts:
  - name: docker-config
    mountPath: /app-root/.docker/
  - name: start-sh
    mountPath: /app-root/azp-agent/start.sh
    subPath: start.sh
  - name: buildkitd
    mountPath: /app-root/.local/share/buildkit

`

Pipeline step rootless buildkit:

`

set -eu

mkdir -p ~/.config/buildkit
echo 'root = "/app-root/.local/share/buildkit"' > ~/.config/buildkit/buildkitd.toml

rootlesskit buildkitd \
  --oci-worker-no-process-sandbox \
  --addr "$BUILDKIT_HOST" &
while ! buildctl debug workers; do sleep 1; done

`

I can confirm the PVC is mounted at /app-root/.local/share/buildkit (separate filesystem), but the pod still accrues significant ephemeral usage and eventually gets evicted.

Questions / Request for guidance

Mount paths: For the bookworm image running BuildKit rootless with HOME=/app-root, is /app-root/.local/share/buildkit the correct path for BuildKit state? (I believe yes.)

Additional mounts: Do I also need to mount a volume at /tmp (or symlink /tmp → /app-root/.local/tmp) to prevent other tools from writing large intermediates to the container writable layer? The chart mounts a PVC-backed tmpdir at /app-root/.local/tmp, but some tools ignore $TMPDIR and use /tmp.

Workspace: Should AZP_WORK=/app-root/azp-work be on a PVC by default with the chart’s pipelines.cache settings, or should I add an explicit mount there as well?

start.sh: Is there anything in the default start.sh that’s required for BuildKit that I might have removed by supplying my own script?

Recommended GC: Any recommended buildkitd.toml GC settings for rootless usage (e.g., maxUsedSpace, reservedSpace) that you expect to work well with this chart?

Happy to provide rendered manifests or logs if that helps. Thanks for any pointers, and for the chart!

[BeckerTim]

Update: Tracked down the source of the pod-ephemeral pressure and resolved the evictions.

While BuildKit’s rootless state was mounted at /app-root/.local/share/buildkit on a PVC, I noticed the writable layer was still growing under /var/lib. A quick check showed a rootful BuildKit daemon writing to /var/lib/buildkit:

du -xhd1 /var/lib 2>/dev/null | sort -h
4.0K /var/lib/PackageKit
4.0K /var/lib/git
4.0K /var/lib/misc
4.0K /var/lib/private
4.0K /var/lib/python
8.0K /var/lib/dbus
8.0K /var/lib/sudo
12K /var/lib/sgml-base
16K /var/lib/polkit-1
16K /var/lib/xml-core
28K /var/lib/pam
40K /var/lib/apt
300K /var/lib/systemd
11M /var/lib/dpkg
15G /var/lib
15G /var/lib/buildkit

Fix applied: I mounted a volume at /var/lib/buildkit (in addition to the existing PVC at /app-root/.local/share/buildkit). After that change, I’ve had zero evictions across 10 successful builds.

If helpful for others, two actionable follow-ups that would make this more robust by default:

Docs clarity: Call out that a rootful BuildKit daemon (e.g., via buildctl-daemonless.sh or default socket) will use /var/lib/buildkit, which counts toward pod-ephemeral storage—suggest mounting a volume there as a safety net, or ensuring only the rootless daemon is used.

Chart option: Consider exposing an optional volume mount for /var/lib/buildkit behind a values.yaml flag to prevent accidental rootful writes to the writable layer.

I’d really appreciate any suggestions or hints if I’ve misconfigured something or missed best practices—happy to adjust my setup based on your guidance.

Thanks!

[clemlesne]

Azure Pipelines agent does not reconnect after first successful job

After the first job completes, the agent does not reconnect to the DevOps server. The default start.sh seems to exit after the initial run, so I mounted a custom script via ConfigMap to keep it running. With that change, the agent does not reconnect and stays alive.

This is intentional. The security model is based on agents that are used once to ensure isolation and idempotency, see the doc. Once you have pending jobs in your DevOps Server, new pods (managed from KEDA) will be created to consume the queue.

start.sh: Is there anything in the default start.sh that’s required for BuildKit that I might have removed by supplying my own script?

The default CMD has been extensively tested with the capacity to handle cleanup and gentle shutdown transmitted to the Azure DevOps Agent. I definitely don't recommend anyone to surcharge it.

Pod evictions due to ephemeral storage, despite BuildKit PVC

I’m seeing evictions with:

ephemeral local storage usage exceeds the total limit of containers

Bumping the ephemeral limit helps temporarily but isn’t ideal since there is not much space on the nodes (50Gi). I tried moving BuildKit’s state to a volume per docs.

I can confirm the PVC is mounted at /app-root/.local/share/buildkit (separate filesystem), but the pod still accrues significant ephemeral usage and eventually gets evicted.

Mount paths: For the bookworm image running BuildKit rootless with HOME=/app-root, is /app-root/.local/share/buildkit the correct path for BuildKit state? (I believe yes.)

Interesting. So perhaps BuildKit is not using /app-root/.local/share/buildkit? But thier doc mentions explicitly this path. Don't know yet, to be investigated.

Update: Tracked down the source of the pod-ephemeral pressure and resolved the evictions.

While BuildKit’s rootless state was mounted at /app-root/.local/share/buildkit on a PVC, I noticed the writable layer was still growing under /var/lib. A quick check showed a rootful BuildKit daemon writing to /var/lib/buildkit:

du -xhd1 /var/lib 2>/dev/null | sort -h 4.0K /var/lib/PackageKit 4.0K /var/lib/git 4.0K /var/lib/misc 4.0K /var/lib/private 4.0K /var/lib/python 8.0K /var/lib/dbus 8.0K /var/lib/sudo 12K /var/lib/sgml-base 16K /var/lib/polkit-1 16K /var/lib/xml-core 28K /var/lib/pam 40K /var/lib/apt 300K /var/lib/systemd 11M /var/lib/dpkg 15G /var/lib 15G /var/lib/buildkit

Fix applied: I mounted a volume at /var/lib/buildkit (in addition to the existing PVC at /app-root/.local/share/buildkit). After that change, I’ve had zero evictions across 10 successful builds.

Indeed BuildKit mentions it in their source code, via sources and example config.

In that scenario, do you find any written file in /app-root/.local/share/buildkit? If yes, what is the content of that file?

Chart option: Consider exposing an optional volume mount for /var/lib/buildkit behind a values.yaml flag to prevent accidental rootful writes to the writable layer.

In that case why just replace the PVC path to /var/lib/buildkit? Will depends on if BuildKit is writing in /app-root/.local/share/buildkit.

Docs clarity: Call out that a rootful BuildKit daemon (e.g., via buildctl-daemonless.sh or default socket) will use /var/lib/buildkit, which counts toward pod-ephemeral storage—suggest mounting a volume there as a safety net, or ensuring only the rootless daemon is used.

I don't recommend at all to run BuildKit as root. It breaks Pod isolation and exposes your Kubernetes Node.

Side note: the docs link currently appears broken: link.

Thanks, fixed in 18ae924 (still in dev for now).

Additional mounts: Do I also need to mount a volume at /tmp (or symlink /tmp → /app-root/.local/tmp) to prevent other tools from writing large intermediates to the container writable layer? The chart mounts a PVC-backed tmpdir at /app-root/.local/tmp, but some tools ignore $TMPDIR and use /tmp.

Have you experienced a tool not using $TMPDIR? Which one?

Workspace: Should AZP_WORK=/app-root/azp-work be on a PVC by default with the chart’s pipelines.cache settings, or should I add an explicit mount there as well?

The folder azp-work is explicitly defined as a dedicated volume because it can be quite I/O intensive. Without that, you'll experience noisy neighbor issues.

See implementation:

blue-agent/src/helm/blue-agent/templates/_helpers.tpl

Lines 244 to 250 in bebac51

	# Separate volume for job working directory
	- name: azp-work
	{{- if .Values.image.isWindows }}
	mountPath: C:\\app-root\\azp-work
	{{- else }}
	mountPath: /app-root/azp-work
	{{- end }}

blue-agent/src/helm/blue-agent/templates/_helpers.tpl

Lines 274 to 290 in bebac51

	# Separate volume for job working directory
	- name: azp-work
	{{- if .Values.pipelines.cache.volumeEnabled }}
	# Avoid using the host tmpdir for size limitations, performance concerns and data leakage in multi-tenant environments
	ephemeral:
	volumeClaimTemplate:
	spec:
	accessModes: [ "ReadWriteOnce" ]
	storageClassName: {{ .Values.pipelines.cache.type | required "A value for .Values.pipelines.cache.type is required" }}
	resources:
	requests:
	storage: {{ .Values.pipelines.cache.size | required "A value for .Values.pipelines.cache.size is required" }}
	{{- else }}
	# Host tmpdir is used, be careful with performance and data leakage
	emptyDir:
	sizeLimit: {{ .Values.pipelines.cache.size | required "A value for .Values.pipelines.cache.size is required" }}
	{{- end }}

Recommended GC: Any recommended buildkitd.toml GC settings for rootless usage (e.g., maxUsedSpace, reservedSpace) that you expect to work well with this chart?

I don't know. Don't hesitate to transmit your feedback & PR! I'll be happy to publish a default config.

Happy to provide rendered manifests or logs if that helps. Thanks for any pointers, and for the chart!

<3

[BeckerTim]

This is intentional. The security model is based on agents that are used once to ensure isolation and idempotency, see the doc. Once you have pending jobs in your DevOps Server, new pods (managed from KEDA) will be created to consume the queue.

Since i am not using KEDA, no new pods will be scaled. Thats a problem. I read it in docs that is a default, but what about not using KEDA to scale with queue lenght?

In that scenario, do you find any written file in /app-root/.local/share/buildkit? If yes, what is the content of that file?

`

ls /app-root/.local/share/buildkit
lost+found

`
i have no idea what that is buts its very small
12K /app-root/.local/share

In that case why just replace the PVC path to /var/lib/buildkit? Will depends on if BuildKit is writing in /app-root/.local/share/buildkit.

Good idea - thanks

[clemlesne]

This is intentional. The security model is based on agents that are used once to ensure isolation and idempotency, see the doc. Once you have pending jobs in your DevOps Server, new pods (managed from KEDA) will be created to consume the queue.

There are two ways the pods initialize, either by being a template or not.

Without KEDA a Deployment is used, thus no auto-scaling but a fixed pool of agents running. The Deployment is defined here.

Then the var AZP_TEMPLATE_JOB is pushed to the container init, which performs differentiated startup.

Can you send your Pod logs? Normally, when a template agent is ran, a log Template job enabled, agent cannot be used for running jobs - see documentation for details then Agent will be stopped after 1 min should appear. If you don't find them, we can exclude this process from our bug cause.

Withion a Deployment, Pods are expected to restart, thus ensuring the build idempotnecy.

Can you confirm me that, without any customization from your end related to the startup script, deployment is done within a Deployment, Pods are running, waiting for your queue, and then retsarted between each run? This is fundamental for security.

Since i am not using KEDA, no new pods will be scaled. Thats a problem. I read it in docs that is a default, but what about not using KEDA to scale with queue lenght?

KEDA is about running one Pod per agent queue. Your agents will autoscale depending on the length on the queue (10, 0, 10000, ...). Cheap and good developer experience.

In that scenario, do you find any written file in /app-root/.local/share/buildkit? If yes, what is the content of that file?

`

ls /app-root/.local/share/buildkit
lost+found

` i have no idea what that is buts its very small 12K /app-root/.local/share

In that case why just replace the PVC path to /var/lib/buildkit? Will depends on if BuildKit is writing in /app-root/.local/share/buildkit.

Good idea - thanks

Do you confirm BuildKit is started with rootlesskit buildkitd? Double-check before committing the change.

[BeckerTim]

KEDA is about running one Pod per agent queue. Your agents will autoscale depending on the length on the queue (10, 0, 10000, ...). Cheap and good developer experience.

I’m aware of KEDA, but we can’t create or manage scaledjobs on the target cluster, so its not an option as of now.

          set -eu

          mkdir -p ~/.config/buildkit
          echo 'root = "/app-root/.local/share/buildkit"' > ~/.config/buildkit/buildkitd.toml

          rootlesskit buildkitd \
            --oci-worker-no-process-sandbox \
            --addr "$BUILDKIT_HOST" &
          while ! buildctl debug workers; do sleep 1; done

This is my pipeline script for buildkit start. i can remove the config i guess. But yes i started BuildKit with rootlesskit but I noticed some meaningful logs:

time="2025-08-14T13:16:47Z" level=warning msg="[rootlesskit:parent] Running RootlessKit as the root user is unsupported."
error: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing: dial unix /run/user/0/buildkit/buildkitd.sock: connect: connection refused"
time="2025-08-14T13:16:48Z" level=info msg="auto snapshotter: using overlayfs"
time="2025-08-14T13:16:48Z" level=warning msg="NoProcessSandbox is enabled. Note that NoProcessSandbox allows build containers to kill (and potentially ptrace) an arbitrary process in the BuildKit host namespace. NoProcessSandbox should be enabled only when the BuildKit is running in a container as an unprivileged user."
time="2025-08-14T13:16:48Z" level=warning msg="CDI setup error /var/run/cdi: failed to monitor for changes: no such file or directory"
time="2025-08-14T13:16:48Z" level=warning msg="CDI setup error /etc/buildkit/cdi: failed to monitor for changes: no such file or directory"
time="2025-08-14T13:16:48Z" level=warning msg="CDI setup error /etc/cdi: failed to monitor for changes: no such file or directory"
time="2025-08-14T13:16:48Z" level=info msg="found worker \"ups1fnw8w8qp4dn1bw65llktp\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:agent-0 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:no-sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/386]"
time="2025-08-14T13:16:48Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2025-08-14T13:16:48Z" level=info msg="found 1 workers, default=\"ups1fnw8w8qp4dn1bw65llktp\""
time="2025-08-14T13:16:48Z" level=warning msg="currently, only the default worker can be used."
time="2025-08-14T13:16:48Z" level=info msg="running server on `/run/user/0/buildkit/buildkitd.sock"

# whoami
root

Did i missconfigured a thing?
Do i have to set a specific user like 1000?
The Dockerfile does some magic with user 0
I found something considering securityContext in the documentation:

 securityContext:
  seccompProfile:
    type: Unconfined
  allowPrivilegeEscalation: true
  capabilities:
    drop: []

But altering the user is not mentioned.

I pulled the chart once more to start from vanilla. Here is what happened:

1. Agent comes up and running.
2. Agent is picking up a build task from my pipeline.
3. Agent runs task. Meanwhile i am facing logs like

VS30063: You are not authorized to access <REDACTED_HOST>.
A job is still running, waiting 15 seconds before retrying the removal
Removing agent from the server
Error reported in diagnostic logs. Please examine the log for more details.

4. The task has finished; however, the sheer volume of error logs makes it difficult to determine if a template job executed.

[2025-08-14 15:03:33Z ERR  VisualStudioServices] GET request to <REDACTED_URL> is not authorized. Details: VS30063: You are not authorized to access <REDACTED_HOST>.
[2025-08-14 15:03:33Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2025-08-14 15:03:33Z INFO LocationServer] Unable to connect to <REDACTED_URL>.
[2025-08-14 15:03:33Z ERR  LocationServer] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: VS30063: You are not authorized to access <REDACTED_HOST>.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, Boolean originalCancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/LocationServer.cs:line 36
[2025-08-14 15:03:33Z INFO Terminal] WRITE LINE: Failed: Removing agent from the server
[2025-08-14 15:03:33Z ERR  Agent] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: VS30063: You are not authorized to access <REDACTED_HOST>.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, Boolean originalCancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.LocationServer.ConnectAsync(VssConnection jobConnection) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/LocationServer.cs:line 36
   at Microsoft.VisualStudio.Services.Agent.Util.ServerUtil.GetConnectionData(String serverUrl, VssCredentials credentials, ILocationServer locationServer, Boolean skipServerCertificateValidation) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/Util/ServerUtil.cs:line 94
   at Microsoft.VisualStudio.Services.Agent.Util.ServerUtil.DetermineDeploymentType(String serverUrl, VssCredentials credentials, ILocationServer locationServer, Boolean skipServerCertificateValidation) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/Util/ServerUtil.cs:line 85
   at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.CheckIsHostedServer(IConfigurationProvider agentProvider, AgentSettings agentSettings, ICredentialProvider credProvider, Boolean skipServerCertificateValidation) in /mnt/vss/_work/1/s/src/Agent.Listener/Configuration/ConfigurationManager.cs:line 1118
   at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.ConfigurationManager.UnconfigureAsync(CommandSettings command) in /mnt/vss/_work/1/s/src/Agent.Listener/Configuration/ConfigurationManager.cs:line 592
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.ExecuteCommand(CommandSettings command) in /mnt/vss/_work/1/s/src/Agent.Listener/Agent.cs:line 120
[2025-08-14 15:03:33Z ERR  Terminal] WRITE ERROR: VS30063: You are not authorized to access <REDACTED_HOST>.
➡️ Removing agent
Removing agent from the server
Error reported in diagnostic logs. Please examine the log for more details.
    - /app-root/azp-logs/Agent_20250814-150347-utc.log
Failed: Removing agent from the server
VS30063: You are not authorized to access <REDACTED_HOST>.
A job is still running, waiting 15 seconds before retrying the removal
Removing agent from the server

5. Agent remains offline. tasks where no longer picked up.

I double checked it with another PAT - both full access, same problem

[clemlesne]

I’m aware of KEDA, but we can’t create or manage scaledjobs on the target cluster, so its not an option as of now.

Ok

Do i have to set a specific user like 1000?
The Dockerfile does some magic with user 0
I found something considering securityContext in the documentation

Security policies are variable depending on your Kubernetes cluster. I chose not to force anything. It's up to you to start the container with the user you want. For security reasons, it is generally advised to run as non-root with a UID non 0.

But BuildKit requires to set allowPrivilegeEscalation=true. Anyway, you can still run as non-root.

Docker-in-docker is tricky. I'm open easing the configuration with a more explicative doc.

GET request to <REDACTED_URL> is not authorized.

Can you share the path of the URL? Of course exclude the domain.

Details: VS30063: You are not authorized to access <REDACTED_HOST>.

I see it twice, at startup and at auto-removal. This is an auth error raised by the server because the agent is not authorized. What permissions did you give on the personal access token? It should have agent pools (read & manage).

[BeckerTim]

I see it twice, at startup and at auto-removal. This is an auth error raised by the server because the agent is not authorized. What permissions did you give on the personal access token? It should have agent pools (read & manage).

Yes the PAT has Full Access. But maybe its on our hosted devops server solution. I will investigate further with some team mates.

Can you share the path of the URL? Of course exclude the domain.
it was basically the domain name "https://my.devops.server"

with running as non root (user 1000 example in helm chart ) with some extraEnv:

securityContext:
  capabilities:
    drop:
    - ALL
  readOnlyRootFilesystem: false <-- true resulted in error to touch .env 
  runAsNonRoot: true
  runAsUser: 1000
extraEnv:
  - name: XDG_RUNTIME_DIR
    value: /run/user/1000
  - name: BUILDKIT_HOST
    value: unix:///run/user/1000/buildkit/buildkitd.sock

starting buildkit resulted in following error:
compute uid/gid map: user: unknown userid 1000 [rootlesskit:child ] error: EOF error: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing: dial unix /run/user/1000/buildkit/buildkitd.sock: connect: no such file or directory"

Is it somehow possible that my containers are not aware of that user? The image sets up everything for root:

ENV USER=root
...
RUN mkdir -p /run/user/0 ${HOME}/.local/tmp ${HOME}/.local/share/buildkit \
    && chown -R ${USER} /run/user/0 ${HOME} \
    && echo ${USER}:100000:65536 | tee /etc/subuid | tee /etc/subgid
USER 0:0
ENV XDG_RUNTIME_DIR=/run/user/0
ENV TMPDIR=${HOME}/.local/tmp
ENV BUILDKIT_HOST=unix:///run/user/0/buildkit/buildkitd.sock

Is there some magic i missed to run buildkit with rootlesskit?