|
| 1 | +# spanner-autoscaler - AI Agent Guide |
| 2 | + |
| 3 | +## Project Structure |
| 4 | + |
| 5 | +**Single-group layout (default):** |
| 6 | +``` |
| 7 | +cmd/main.go Manager entry (registers controllers/webhooks) |
| 8 | +api/<version>/*_types.go CRD schemas (+kubebuilder markers) |
| 9 | +api/<version>/zz_generated.* Auto-generated (DO NOT EDIT) |
| 10 | +internal/controller/* Reconciliation logic |
| 11 | +internal/webhook/* Validation/defaulting (if present) |
| 12 | +config/crd/bases/* Generated CRDs (DO NOT EDIT) |
| 13 | +config/rbac/role.yaml Generated RBAC (DO NOT EDIT) |
| 14 | +config/samples/* Example CRs (edit these) |
| 15 | +Makefile Build/test/deploy commands |
| 16 | +PROJECT Kubebuilder metadata Auto-generated (DO NOT EDIT) |
| 17 | +``` |
| 18 | + |
| 19 | +**Multi-group layout** (for projects with multiple API groups): |
| 20 | +``` |
| 21 | +api/<group>/<version>/*_types.go CRD schemas by group |
| 22 | +internal/controller/<group>/* Controllers by group |
| 23 | +internal/webhook/<group>/<version>/* Webhooks by group and version (if present) |
| 24 | +``` |
| 25 | + |
| 26 | +Multi-group layout organizes APIs by group name (e.g., `batch`, `apps`). Check the `PROJECT` file for `multigroup: true`. |
| 27 | + |
| 28 | +**To convert to multi-group layout:** |
| 29 | +1. Run: `kubebuilder edit --multigroup=true` |
| 30 | +2. Move APIs: `mkdir -p api/<group> && mv api/<version> api/<group>/` |
| 31 | +3. Move controllers: `mkdir -p internal/controller/<group> && mv internal/controller/*.go internal/controller/<group>/` |
| 32 | +4. Move webhooks (if present): `mkdir -p internal/webhook/<group> && mv internal/webhook/<version> internal/webhook/<group>/` |
| 33 | +5. Update import paths in all files |
| 34 | +6. Fix `path` in `PROJECT` file for each resource |
| 35 | +7. Update test suite CRD paths (add one more `..` to relative paths) |
| 36 | + |
| 37 | +## Critical Rules |
| 38 | + |
| 39 | +### Never Edit These (Auto-Generated) |
| 40 | +- `config/crd/bases/*.yaml` - from `make manifests` |
| 41 | +- `config/rbac/role.yaml` - from `make manifests` |
| 42 | +- `config/webhook/manifests.yaml` - from `make manifests` |
| 43 | +- `**/zz_generated.*.go` - from `make generate` |
| 44 | +- `PROJECT` - from `kubebuilder [OPTIONS]` |
| 45 | + |
| 46 | +### Never Remove Scaffold Markers |
| 47 | +Do NOT delete `// +kubebuilder:scaffold:*` comments. CLI injects code at these markers. |
| 48 | + |
| 49 | +### Keep Project Structure |
| 50 | +Do not move files around. The CLI expects files in specific locations. |
| 51 | + |
| 52 | +### Always Use CLI Commands |
| 53 | +Always use `kubebuilder create api` and `kubebuilder create webhook` to scaffold. Do NOT create files manually. |
| 54 | + |
| 55 | +### E2E Tests Require an Isolated Kind Cluster |
| 56 | +The e2e tests are designed to validate the solution in an isolated environment (similar to GitHub Actions CI). |
| 57 | +Ensure you run them against a dedicated [Kind](https://kind.sigs.k8s.io/) cluster (not your “real” dev/prod cluster). |
| 58 | + |
| 59 | +## After Making Changes |
| 60 | + |
| 61 | +**After editing `*_types.go` or markers:** |
| 62 | +``` |
| 63 | +make manifests # Regenerate CRDs/RBAC from markers |
| 64 | +make generate # Regenerate DeepCopy methods |
| 65 | +``` |
| 66 | + |
| 67 | +**After editing `*.go` files:** |
| 68 | +``` |
| 69 | +make lint-fix # Auto-fix code style |
| 70 | +make test # Run unit tests |
| 71 | +``` |
| 72 | + |
| 73 | +## CLI Commands Cheat Sheet |
| 74 | + |
| 75 | +### Create API (your own types) |
| 76 | +```bash |
| 77 | +kubebuilder create api --group <group> --version <version> --kind <Kind> |
| 78 | +``` |
| 79 | + |
| 80 | +### Deploy Image Plugin (scaffold to deploy/manage ANY container image) |
| 81 | + |
| 82 | +Generate a controller that deploys and manages a container image (nginx, redis, memcached, your app, etc.): |
| 83 | + |
| 84 | +```bash |
| 85 | +# Example: deploying memcached |
| 86 | +kubebuilder create api --group example.com --version v1alpha1 --kind Memcached \ |
| 87 | + --image=memcached:alpine \ |
| 88 | + --plugins=deploy-image.go.kubebuilder.io/v1-alpha |
| 89 | +``` |
| 90 | + |
| 91 | +Scaffolds good-practice code: reconciliation logic, status conditions, finalizers, RBAC. Use as a reference implementation. |
| 92 | + |
| 93 | + |
| 94 | +### Create Webhooks |
| 95 | +```bash |
| 96 | +# Validation + defaulting |
| 97 | +kubebuilder create webhook --group <group> --version <version> --kind <Kind> \ |
| 98 | + --defaulting --programmatic-validation |
| 99 | + |
| 100 | +# Conversion webhook (for multi-version APIs) |
| 101 | +kubebuilder create webhook --group <group> --version v1 --kind <Kind> \ |
| 102 | + --conversion --spoke v2 |
| 103 | +``` |
| 104 | + |
| 105 | +### Controller for Core Kubernetes Types |
| 106 | +```bash |
| 107 | +# Watch Pods |
| 108 | +kubebuilder create api --group core --version v1 --kind Pod \ |
| 109 | + --controller=true --resource=false |
| 110 | + |
| 111 | +# Watch Deployments |
| 112 | +kubebuilder create api --group apps --version v1 --kind Deployment \ |
| 113 | + --controller=true --resource=false |
| 114 | +``` |
| 115 | + |
| 116 | +### Controller for External Types (e.g., from other operators) |
| 117 | + |
| 118 | +Watch resources from external APIs (cert-manager, Argo CD, Istio, etc.): |
| 119 | + |
| 120 | +```bash |
| 121 | +# Example: watching cert-manager Certificate resources |
| 122 | +kubebuilder create api \ |
| 123 | + --group cert-manager --version v1 --kind Certificate \ |
| 124 | + --controller=true --resource=false \ |
| 125 | + --external-api-path=github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1 \ |
| 126 | + --external-api-domain=io \ |
| 127 | + --external-api-module=github.com/cert-manager/cert-manager |
| 128 | +``` |
| 129 | + |
| 130 | +**Note:** Use `--external-api-module=<module>@<version>` only if you need a specific version. Otherwise, omit `@<version>` to use what's in go.mod. |
| 131 | + |
| 132 | +### Webhook for External Types |
| 133 | + |
| 134 | +```bash |
| 135 | +# Example: validating external resources |
| 136 | +kubebuilder create webhook \ |
| 137 | + --group cert-manager --version v1 --kind Issuer \ |
| 138 | + --defaulting \ |
| 139 | + --external-api-path=github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1 \ |
| 140 | + --external-api-domain=io \ |
| 141 | + --external-api-module=github.com/cert-manager/cert-manager |
| 142 | +``` |
| 143 | + |
| 144 | +## Testing & Development |
| 145 | + |
| 146 | +```bash |
| 147 | +make test # Run unit tests (uses envtest: real K8s API + etcd) |
| 148 | +make run # Run locally (uses current kubeconfig context) |
| 149 | +``` |
| 150 | + |
| 151 | +Tests use **Ginkgo + Gomega** (BDD style). Check `suite_test.go` for setup. |
| 152 | + |
| 153 | +## Deployment Workflow |
| 154 | + |
| 155 | +```bash |
| 156 | +# 1. Regenerate manifests |
| 157 | +make manifests generate |
| 158 | + |
| 159 | +# 2. Build & deploy |
| 160 | +export IMG=<registry>/<project>:tag |
| 161 | +make docker-build docker-push IMG=$IMG # Or: kind load docker-image $IMG --name <cluster> |
| 162 | +make deploy IMG=$IMG |
| 163 | + |
| 164 | +# 3. Test |
| 165 | +kubectl apply -k config/samples/ |
| 166 | + |
| 167 | +# 4. Debug |
| 168 | +kubectl logs -n <project>-system deployment/<project>-controller-manager -c manager -f |
| 169 | +``` |
| 170 | + |
| 171 | +### API Design |
| 172 | + |
| 173 | +**Key markers for** `api/<version>/*_types.go`: |
| 174 | + |
| 175 | +```go |
| 176 | +// +kubebuilder:object:root=true |
| 177 | +// +kubebuilder:subresource:status |
| 178 | +// +kubebuilder:resource:scope=Namespaced |
| 179 | +// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=".status.conditions[?(@.type=='Ready')].status" |
| 180 | + |
| 181 | +// On fields: |
| 182 | +// +kubebuilder:validation:Required |
| 183 | +// +kubebuilder:validation:Minimum=1 |
| 184 | +// +kubebuilder:validation:MaxLength=100 |
| 185 | +// +kubebuilder:validation:Pattern="^[a-z]+$" |
| 186 | +// +kubebuilder:default="value" |
| 187 | +``` |
| 188 | + |
| 189 | +- **Use** `metav1.Condition` for status (not custom string fields) |
| 190 | +- **Use predefined types**: `metav1.Time` instead of `string` for dates |
| 191 | +- **Follow K8s API conventions**: Standard field names (`spec`, `status`, `metadata`) |
| 192 | + |
| 193 | +### Controller Design |
| 194 | + |
| 195 | +**RBAC markers in** `internal/controller/*_controller.go`: |
| 196 | + |
| 197 | +```go |
| 198 | +// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds,verbs=get;list;watch;create;update;patch;delete |
| 199 | +// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds/status,verbs=get;update;patch |
| 200 | +// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds/finalizers,verbs=update |
| 201 | +// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch |
| 202 | +// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete |
| 203 | +``` |
| 204 | + |
| 205 | +**Implementation rules:** |
| 206 | +- **Idempotent reconciliation**: Safe to run multiple times |
| 207 | +- **Re-fetch before updates**: `r.Get(ctx, req.NamespacedName, obj)` before `r.Update` to avoid conflicts |
| 208 | +- **Structured logging**: `log := log.FromContext(ctx); log.Info("msg", "key", val)` |
| 209 | +- **Owner references**: Enable automatic garbage collection (`SetControllerReference`) |
| 210 | +- **Watch secondary resources**: Use `.Owns()` or `.Watches()`, not just `RequeueAfter` |
| 211 | +- **Finalizers**: Clean up external resources (buckets, VMs, DNS entries) |
| 212 | + |
| 213 | +### Logging |
| 214 | + |
| 215 | +**Follow Kubernetes logging message style guidelines:** |
| 216 | + |
| 217 | +- Start from a capital letter |
| 218 | +- Do not end the message with a period |
| 219 | +- Active voice: subject present (`"Deployment could not create Pod"`) or omitted (`"Could not create Pod"`) |
| 220 | +- Past tense: `"Could not delete Pod"` not `"Cannot delete Pod"` |
| 221 | +- Specify object type: `"Deleted Pod"` not `"Deleted"` |
| 222 | +- Balanced key-value pairs |
| 223 | + |
| 224 | +```go |
| 225 | +log.Info("Starting reconciliation") |
| 226 | +log.Info("Created Deployment", "name", deploy.Name) |
| 227 | +log.Error(err, "Failed to create Pod", "name", name) |
| 228 | +``` |
| 229 | + |
| 230 | +**Reference:** https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines |
| 231 | + |
| 232 | +### Webhooks |
| 233 | +- **Create all types together**: `--defaulting --programmatic-validation --conversion` |
| 234 | +- **When`--force`is used**: Backup custom logic first, then restore after scaffolding |
| 235 | +- **For multi-version APIs**: Use hub-and-spoke pattern (`--conversion --spoke v2`) |
| 236 | + - Hub version: Usually oldest stable version (v1) |
| 237 | + - Spoke versions: Newer versions that convert to/from hub (v2, v3) |
| 238 | + - Example: `--group crew --version v1 --kind Captain --conversion --spoke v2` (v1 is hub, v2 is spoke) |
| 239 | + |
| 240 | +### Learning from Examples |
| 241 | + |
| 242 | +The **deploy-image plugin** scaffolds a complete controller following good practices. Use it as a reference implementation: |
| 243 | + |
| 244 | +```bash |
| 245 | +kubebuilder create api --group example --version v1alpha1 --kind MyApp \ |
| 246 | + --image=<your-image> --plugins=deploy-image.go.kubebuilder.io/v1-alpha |
| 247 | +``` |
| 248 | + |
| 249 | +Generated code includes: status conditions (`metav1.Condition`), finalizers, owner references, events, idempotent reconciliation. |
| 250 | + |
| 251 | +## Distribution Options |
| 252 | + |
| 253 | +### Option 1: YAML Bundle (Kustomize) |
| 254 | + |
| 255 | +```bash |
| 256 | +# Generate dist/install.yaml from Kustomize manifests |
| 257 | +make build-installer IMG=<registry>/<project>:tag |
| 258 | +``` |
| 259 | + |
| 260 | +**Key points:** |
| 261 | +- The `dist/install.yaml` is generated from Kustomize manifests (CRDs, RBAC, Deployment) |
| 262 | +- Commit this file to your repository for easy distribution |
| 263 | +- Users only need `kubectl` to install (no additional tools required) |
| 264 | + |
| 265 | +**Example:** Users install with a single command: |
| 266 | +```bash |
| 267 | +kubectl apply -f https://raw.githubusercontent.com/<org>/<repo>/<tag>/dist/install.yaml |
| 268 | +``` |
| 269 | + |
| 270 | +### Option 2: Helm Chart |
| 271 | + |
| 272 | +```bash |
| 273 | +kubebuilder edit --plugins=helm/v2-alpha # Generates dist/chart/ (default) |
| 274 | +kubebuilder edit --plugins=helm/v2-alpha --output-dir=charts # Generates charts/chart/ |
| 275 | +``` |
| 276 | + |
| 277 | +**For development:** |
| 278 | +```bash |
| 279 | +make helm-deploy IMG=<registry>/<project>:<tag> # Deploy manager via Helm |
| 280 | +make helm-deploy IMG=$IMG HELM_EXTRA_ARGS="--set ..." # Deploy with custom values |
| 281 | +make helm-status # Show release status |
| 282 | +make helm-uninstall # Remove release |
| 283 | +make helm-history # View release history |
| 284 | +make helm-rollback # Rollback to previous version |
| 285 | +``` |
| 286 | + |
| 287 | +**For end users/production:** |
| 288 | +```bash |
| 289 | +helm install my-release ./<output-dir>/chart/ --namespace <ns> --create-namespace |
| 290 | +``` |
| 291 | + |
| 292 | +**Important:** If you add webhooks or modify manifests after initial chart generation: |
| 293 | +1. Backup any customizations in `<output-dir>/chart/values.yaml` and `<output-dir>/chart/manager/manager.yaml` |
| 294 | +2. Re-run: `kubebuilder edit --plugins=helm/v2-alpha --force` (use same `--output-dir` if customized) |
| 295 | +3. Manually restore your custom values from the backup |
| 296 | + |
| 297 | +### Publish Container Image |
| 298 | + |
| 299 | +```bash |
| 300 | +export IMG=<registry>/<project>:<version> |
| 301 | +make docker-build docker-push IMG=$IMG |
| 302 | +``` |
| 303 | + |
| 304 | +## References |
| 305 | + |
| 306 | +### Essential Reading |
| 307 | +- **Kubebuilder Book**: https://book.kubebuilder.io (comprehensive guide) |
| 308 | +- **controller-runtime FAQ**: https://github.com/kubernetes-sigs/controller-runtime/blob/main/FAQ.md (common patterns and questions) |
| 309 | +- **Good Practices**: https://book.kubebuilder.io/reference/good-practices.html (why reconciliation is idempotent, status conditions, etc.) |
| 310 | +- **Logging Conventions**: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines (message style, verbosity levels) |
| 311 | + |
| 312 | +### API Design & Implementation |
| 313 | +- **API Conventions**: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md |
| 314 | +- **Operator Pattern**: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ |
| 315 | +- **Markers Reference**: https://book.kubebuilder.io/reference/markers.html |
| 316 | + |
| 317 | +### Tools & Libraries |
| 318 | +- **controller-runtime**: https://github.com/kubernetes-sigs/controller-runtime |
| 319 | +- **controller-tools**: https://github.com/kubernetes-sigs/controller-tools |
| 320 | +- **Kubebuilder Repo**: https://github.com/kubernetes-sigs/kubebuilder |
0 commit comments