feat(crossplane): PoC — Crossplane ObjectStore capability for pack infrastructure (AWS/EKS) - #502
Draft
oldsj wants to merge 13 commits into
Draft
feat(crossplane): PoC — Crossplane ObjectStore capability for pack infrastructure (AWS/EKS)#502oldsj wants to merge 13 commits into
oldsj wants to merge 13 commits into
Conversation
Add the foundational, out-of-cluster half of the Crossplane pack-infra PoC (ADR-0012). Clusters opt into named capabilities via `crossplane_capabilities` (bare keys "s3"/"rds", validated at config load); each enabled key provisions a scoped EKS Pod Identity role and least-privilege policy in crossplane-iam.tf so enabling a capability is an explicit, per-capability authorization rather than a blanket grant. - config: crossplane_capabilities field, validation against the supported set, and EnabledCrossplaneCapabilities() returning provider-qualified ids (aws-s3). - tofu: thread sorted capability keys into tfvars for deterministic output. - crossplane-iam.tf: one name-prefixed role/policy per capability, scoped to project-prefixed S3/RDS ARNs and guarded by RequestTag/ResourceTag on the crossplane-providerconfig tag; crossplane_iam_test.go locks those guardrails. - cluster.InfraSettings gains CrossplaneCapabilities (the shared, provider- agnostic contract) plus the CrossplaneEnabled() predicate consumed in commit 2.
…ties Add the in-cluster half of the Crossplane PoC (ADR-0012): the core chart and providers ArgoCD Applications, plus per-capability provider packages, provider configs, and post-install config Applications for aws-s3 and aws-rds. Crossplane is provider/profile-conditional foundational software (ADR-0012 §3): gate the entire tree on CrossplaneEnabled() so a cluster that opts into no capability receives none of it -- the core chart, providers Application, and every manifest are pruned from the gitops repo, not just skipped. When at least one capability is enabled, foundational manifests are written and the existing per-capability gate trims the un-opted-in layers. Drop the bundled provider-keycloak package: it is out of scope for this PoC.
Opting into the s3 Crossplane capability now also enables the narrowly
scoped aws-iam and aws-eks provider controllers needed to create
per-workload Pod Identity bindings for managed S3 object stores.
- EnabledCrossplaneCapabilities expands s3 -> {iam, eks} internal deps
- add aws-eks / aws-iam provider + provider-config manifests and ArgoCD
app templates, gated on the same opt-in capability set
- crossplane-iam.tf provisions one scoped Pod Identity role per provider
- register aws-iam / aws-eks in crossplaneCapabilities gating list
Verifying the ObjectStore Composition end-to-end (ADR-0012) surfaced four
issues in crossplane-iam.tf, all fixed here.
OpenTofu render (blocked any apply once S3 was enabled):
- Split the per-provider tag/deny statements from one mixed-shape 4-element
tuple into four single-element conditionals so they unify with the empty
branch.
- Replace crossplane_provider_boundary's {}/object ternary for_each with a
filtered comprehension.
IAM scoping (blocked the IAM/EKS providers creating workload roles):
- iam:GetRole on a not-yet-created role authorizes against role/<name> with no
path, so observe-before-create never matched the path-scoped ARN. Add a
name-only read ARN; writes stay path-, boundary-, and tag-scoped.
- CreatePodIdentityAssociation validates the target role via iam:GetRole, but
the EKS provider had only iam:PassRole. Add a read-only GetRole grant.
Show how an application pack (e.g. nebari-mlflow) consumes the platform XObjectStore API: an XObjectStore claim naming a ServiceAccount, and a probe pod that reads/writes the provisioned bucket via EKS Pod Identity (proving the access is scoped to just that bucket). The XObjectStore XRD + Composition it depends on are a platform capability, provided separately.
Persist the XObjectStore XRD + Composition + composition functions under
examples/crossplane/objectstore/platform as reference alongside the consumer
example. Values (account/region/cluster/prefix) are hardcoded to the phase-0
PoC and the Composition/README say so loudly.
Making this a portable, NIC-managed platform contract (consumed by packs like
nebari-mlflow across projects) needs EnvironmentConfig parameterization plus a
verbatim-copy path in the writer -- the Composition's {{ }} go-templating
collides with NIC's own text/template pass. Tracked as a follow-up; applied by
hand until then.
Replace the strict-separation IAM (per-capability provider-aws-s3/-iam/-eks roles, individually scoped policies, per-provider permissions boundaries) with the single broad account-local provider role ADR-0012 recommends. All provider controllers share one role via Pod Identity; the role uses a broad Allow with outer denies for IAM privilege-escalation containment (writes confined to the workload-role path, boundary required on create, scoped iam:PassRole) and a best-effort collision guard on the foundational EKS control plane. The workload permissions boundary and provisioning-vs-runtime identity separation are kept. Collapse the four ClusterProviderConfigs (aws-s3/iam/eks/rds) into a single foundational `aws` config and one config Application; provider packages stay gated per-capability. Update the composition providerConfigRefs, guardrail tests, writer gating, and docs/comments accordingly.
# Conflicts: # pkg/providers/cluster/aws/templates/variables.tf # pkg/providers/cluster/aws/tofu.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Proof-of-concept implementation of the Crossplane capability boundary from
ADR-0012 (#501),
addressing the research/PoC request in
#453.
It lets an authorized cluster provision the external cloud infrastructure a
software pack needs (the motivating case: an S3 bucket + keyless access for
MLflow artifacts) in-cluster via Crossplane, while OpenTofu keeps ownership of
the foundational cluster. Deliberately narrow per the ADR: one
ObjectStorecapability, one AWS/EKS implementation, one consumer — not a capability
marketplace.
Warning
PoC, not production. Opening as a draft. The
ObjectStoreComposition isstill hardcoded to the phase-0 PoC cluster (account/region/cluster/prefix), and
the ADR's validation gates
(functional, security, lifecycle/ops) are not yet all met. See Not in scope below.
What's included
crossplane-iam.tf: opt-in AWS permissions for theCrossplane provider controllers, provisioned only when a capability is enabled.
No static credentials — EKS Pod Identity throughout.
and the shared provider config install only when at least one capability is
opted into (ADR-0006 conditional-foundational path); provider packages are
gated per-capability.
crossplane_capabilities: [s3]on the AWS cluster;enabling
s3also pulls in the IAM and EKS providers needed for per-workloadPod Identity bindings.
ObjectStoreexample (examples/crossplane/objectstore/) — the platformXRD + Composition, a consumer claim, and a probe pod that proves scoped
read/write. Marked PoC-only/hardcoded.
IAM security model
Implements ADR-0012's recommended dedicated-account model: all provider
controllers share one broad account-local role (assumed via Pod Identity),
with outer denies for IAM privilege-escalation containment (writes confined to
the workload-role path, permissions boundary required on role creation, scoped
iam:PassRole) plus a best-effort collision guard on the foundational EKScontrol plane. Provisioning identity stays separate from runtime identity: each
ObjectStoregets its own bucket-scoped, boundary-constrained workload role.An earlier revision of this branch implemented the stricter per-capability-role
variant; it was refactored to the dedicated-account baseline the ADR proposes for
initial support (simpler OpenTofu→Crossplane handoff, single
awsProviderConfig).Not in scope (follow-ups)
writer change) so it's a portable NIC-managed contract.
ADR validation gates (Access Analyzer, isolation tests, destroy/recovery drills).
nic destroypre-destroy phase.Testing
go build ./...,go vet ./..., andgo test ./...all pass;terraform fmtclean on the templates. IAM guardrails covered by
pkg/providers/cluster/aws/crossplane_iam_test.go; gating byTestWriteAllToGit_CrossplaneCapabilities.Related