Skip to content

ACK controllers crashloop on namespace scope #2625

@aleixripoll

Description

@aleixripoll

Describe the bug
We are running ACK controllers with namespace scope in a cluster with restricted cluster-scoped permissions. When installing them we remove namespaces-cache ClusterRole and ClusterRoleBinding from the templates and also the "namespaces" RBAC from the helpers. So far this worked correctly in all controllers (example, IAM controller v1.3.8), we get some errors logged but they just work, example:

{"level":"error","ts":"2025-09-10T06:54:11.897Z","msg":"pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User \"system:serviceaccount:xxx:ack-dynamodb-controller\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:147\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:292\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:290\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72"}

Recently we updated all ACK controllers to the latest versions (example, IAM v1.5.0) and we noticed they are all crashing:

{"level":"error","ts":"2025-09-10T07:10:44.239Z","msg":"Unhandled Error","logger":"UnhandledError","error":"pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User \"system:serviceaccount:xxx:ack-dynamodb-controller\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72"}

The only difference we see is that the error is now of type "UnhandledError", the controller crashes right after that message.

Steps to reproduce
Remove cluster-scoped resources and RBAC in a namespace-scoped controller, add an additional namespace to the controller's "watchNamespace" list.

Expected outcome
Controller is able to to run in namespace scope with no cluster-scoped permissions.

Environment
All environments.

  • Kubernetes version
  • Using EKS (yes/no), if so version? yes, v1.32.7-eks-b707fbb
  • AWS service targeted (S3, RDS, etc.): All services

Metadata

Metadata

Labels

area/runtimeIssues or PRs as related to controller runtime, common reconciliation logic, etc

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions