ECR Create-on-Push with EKS Pod Identity

[Diesel93]

Requirement

Context

This is a follow-up to issue #4107, which was closed as "not planned." However, AWS ECR now supports create-on-push (released December 2025), which resolves the original concern about pre-creating repositories. Despite this new feature working correctly with manual pushes, Jib authentication with EKS Pod Identity remains broken.

Environment

• Camel K Version: 2.8.0
• AWS ECR: Create-on-push enabled (December 2025 feature)
• Authentication: EKS Pod Identity (NOT IRSA)
• Build Strategy: pod (using Jib)

Problem Summary

AWS ECR's create-on-push feature works perfectly when manually pushing images using tools like skopeo or docker with AWS CLI authentication. However, Jib fails with 401 Unauthorized when attempting to push IntegrationKit images during builds, despite the build pod having valid Pod Identity credentials.

Error Message

Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.1:build (default-cli) on project camel-k-integration:
Build image failed, perhaps you should make sure your credentials for '123456789.dkr.ecr.eu-west-2.amazonaws.com/camel-integrations/camel-k/camel-k/camel-k-kit-xxx' are set up correctly.
See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help:
Unauthorized for 123456789.dkr.ecr.eu-west-2.amazonaws.com/camel-integrations/camel-k/camel-k/camel-k-kit-xxx: 401 Unauthorized

Root Cause Analysis

Jib doesn't integrate with the AWS SDK credential chain, so it cannot automatically discover and use Pod Identity credentials exposed via the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable.

Jib's supported authentication methods:

1. Docker config files (~/.docker/config.json)
2. Docker credential helpers (e.g., docker-credential-ecr-login)
3. Explicit authentication parameters

Why these don't work with Pod Identity:

• Credential helper: The docker-credential-ecr-login helper works with IRSA but is broken with EKS Pod Identity (as of v0.8.0+)
• Builder trait approach: Attempted to use Camel K's builder trait to create Docker config in init container, but custom task containers don't share filesystem/volumes with the main build containers (init writes to /root/.docker/config.json, but jib container has HOME=/builder/kit-xxx)

Verification - ECR Permissions Are Correct

Manual push with the same service account and Pod Identity succeeds:

# Test pod using camel-k-builder service account
kubectl exec -n camel-k ecr-test-pod -- sh -c '
  TOKEN=$(aws ecr get-login-password --region eu-west-2)
  echo $TOKEN | skopeo copy \
    --dest-creds AWS:$(cat -) \
    docker://alpine:latest \
    docker://123456789.dkr.ecr.eu-west-2.amazonaws.com/camel-integrations/camel-k/test:latest
'
# SUCCESS - Repository auto-created, image pushed

This proves:

• ECR create-on-push is working
• Pod Identity credentials are valid
• IAM permissions are correct
• The issue is specifically with Jib's credential discovery

Pod Identity Environment Variables (Available in Build Pod)

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
AWS_REGION=eu-west-2
AWS_DEFAULT_REGION=eu-west-2

Failed Workaround - Builder Trait Custom Task

Attempt:

spec:
  traits:
    builder:
      tasks:
        - 'ecr-login;amazon/aws-cli:latest;/bin/sh -c "TOKEN=$(aws ecr get-login-password) && mkdir -p ~/.docker && echo {\"auths\":{\"ECR_URL\":{\"username\":\"AWS\",\"password\":\"$TOKEN\"}}} > ~/.docker/config.json";0'
      tasksFilter: "ecr-login,builder,package,jib"

Why it failed:

• The ecr-login init container doesn't have the shared build volume (/builder/kit-xxx) mounted
• Docker config created in init container at /root/.docker/config.json is not accessible to the jib container
• The jib container has HOME=/builder/kit-xxx but init containers use different HOME paths

Request for Enhancement

Would it be possible to enhance Jib integration in Camel K to support AWS SDK credential providers (similar to how AWS CLI/skopeo work)? This would allow Jib to automatically discover credentials from:

• EKS Pod Identity (via AWS_CONTAINER_CREDENTIALS_FULL_URI)
• IRSA (via AWS_WEB_IDENTITY_TOKEN_FILE)
• EC2 instance metadata

This would enable seamless ECR integration without requiring:

• Manual credential management
• Kubernetes secrets with expiring ECR tokens
• Workarounds with credential helpers or custom init containers

Alternative Solution Considered

Using a Kubernetes Secret with ECR credentials requires:

• CronJob to refresh token every 11 hours (ECR tokens expire after 12 hours)
• Additional infrastructure complexity
• Not ideal for cloud-native environments where Pod Identity should "just work"

Related Issues

• Original issue: Introduce and document support for AWS Elastic Container Registry (ECR) #4107 - Closed as "not planned" due to ECR not supporting create-on-push. This is now resolved - AWS ECR added create-on-push in December 2025, eliminating the need to pre-create repositories. The remaining blocker is Jib authentication with Pod Identity.
• Upstream credential helper bug: Unable to Use amazon-ecr-credential-helper with EKS pod identity. awslabs/amazon-ecr-credential-helper#959 - docker-credential-ecr-login doesn't support EKS Pod Identity (only IRSA)

[squakez]

Why don't you try using a custom packaging task instead? After the build and package phase you have the Dockerfile generated, the jar dependencies and a variable which contain the name of the container eventually used by the container. If you use that container image you have used to login and push the image from there, I think you won't need anything else.