test - please ignore - #1411
Draft
silvestre wants to merge 23 commits into
Draft
test - please ignore#1411silvestre wants to merge 23 commits into
silvestre wants to merge 23 commits into
Conversation
silvestre
commented
Sep 1, 2026
silvestre
commented
Member
- docs(scale-to-zero): design for activator-based scale-to/from-zero
- feat(api): allow instance_min_count 0 for scale-to-zero
- feat(scalingengine): park app before scaling to zero
- feat(activator): scaffold scale-from-zero activator service
- feat(mta): deploy activator module and route-service UPSI
- Bump Go to 1.26.4
- chore(deps): add code.cloudfoundry.org/routing-api
- feat(activator): implement park/wake loops and engine wiring
- test(acceptance): add scale-to/from-zero acceptance test
Relax the v0.1 policy schema floor from 1 to 0 on instance_min_count and add a minimum of 1 on instance_max_count (an app with max 0 can never serve). Legacy schema is unchanged.
Add an Activator seam (Park/Unpark) that the engine invokes before removing an app's last instance, so routes are covered for scale-from-zero (bind -> confirm -> scale-0). Defaults to a no-op activator so the engine runs standalone; the real client is wired in with the activator service. Park failure fails the scale. See docs/design/scale-to-zero.md.
New CF-app microservice that will park apps behind a route service while scaled to zero and wake them on request. Boots with the standard startup.Bootstrap pattern and serves mTLS, health, and CF (route-service) endpoints. The route-service handler is the Loop A entry point and currently returns 503 + Retry-After; the Parker/ReadinessWatcher seams (Loop A/B) are defined as interfaces with no-op stubs. See docs/design/scale-to-zero.md.
Add the activator as an MTA go module + activator-config UPSI, with its public route and routing_api url wired through the extension file. MTA cannot declare a user-provided service with a route_service_url, so the autoscaler-activator-rs route-service UPSI is created imperatively in mta-deploy.sh (create-or-update), pointed at the activator's route. Route bindings to individual app routes are managed at runtime. See docs/design/scale-to-zero.md.
Add routing-api v0.12.0 for the activator's route-registration event stream (scale-from-zero readiness signal). Mirror routing-api's own ifrit pin via a replace directive so MVS does not pull in the digest that log.Fatalf's on http shutdown (tedsuo/ifrit#48).
Implement the runtime scale-to-zero data path end to end: - cf: wrapper methods to list app routes and create/delete service route bindings, plus resolve a user-provided service instance by name. - activator: CFParker binds/unbinds an app's routes to the route-service UPSI; ReadinessWatcher (Loop B) consumes the routing-api Upsert event stream and un-parks apps whose real endpoint reappears (from any cause - request, schedule, or manual cf scale); Handler (Loop A) holds a Gorouter-forwarded request, wakes the app via the scaling engine, waits on the readiness signal, then forwards to X-CF-Forwarded-Url. On timeout it returns 503 + Retry-After. An in-memory Registry tracks parked apps. - scalingengine: HTTP ActivatorClient implementing the Activator seam; wired via WithActivator when activator_url is configured (opt-in). - routes: activator park/unpark route table. - deploy: activator cf/scaling_engine/routing_api config in the extension file; routing-api UAA client secret via credhub; scalingengine gets the activator_url. Loop-B readiness uses the real routing-api SSE client. Waiter is registered before scale-up to avoid missing a fast Upsert. See docs/design/scale-to-zero.md.
Push an app with instance_min_count 0 and an idle-cpu scale-in rule, assert it scales to zero, then hit its route and assert it wakes back to one instance and serves 200. Filterable via --focus=scaletozero or --label-filter=scaletozero. Requires a deployed activator. See docs/design/scale-to-zero.md.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The deployed activator crash-looped: a routing-api subscribe failure returned from the watcher runner and tore down the whole ifrit group (exit 1). The watcher now reconnects with backoff and never crashes the process, so health/park/route-service stay up when the event stream is unavailable. Also fix the routing_api url: the routing-api client appends /routing/v1/events itself, so the configured URL must be the base (https://api.<domain>) not .../routing/v1 — the suffix produced a doubled path and 404 Not Found.
…achability Verified on cf-deployment: the routing-api events stream is reachable at https://api.<domain>/routing/v1/events (200 with a routing_api_client token that carries routing.routes.read). The api endpoint is a public IP already covered by the platform public_networks ASG, so the activator needs no custom security group — remove the placeholder one. Document the base-URL requirement and the verified reachability/credentials.
The deploy failed at 'cf create-user-provided-service autoscaler-activator-rs' with 'No org targeted': cf_login authenticates but does not target an org/space, and a UPSI lives in a space. Add cf_target before the UPSI step so it is created in the deployment's space.
The bind failed with 'instance_min_count must be >= 1' because the GenerateDynamicScaleOutAndInPolicy builder emits no schema-version, so the broker validated the policy against the legacy schema (which keeps the minimum of 1). Scale-to-zero is a v0.1-only feature for now, so the test now supplies a raw policy with schema-version "0.1" and instance_min_count 0.
The engine's PUT /park reached the activator via its public route, which
Gorouter sends to the CF server where only the route-service catch-all
was mounted -> 503 'no parked app'. Co-host the control API and the
route-service on the CF server: an XFCC-authed subrouter for
/v1/apps/{appid}/park registered before the unauthenticated route-service
catch-all (mux matches specific paths first; subrouter-scoped .Use keeps
the catch-all XFCC-free). The mTLS server is now a vestigial 404 stub.
Also: the acceptance test uses a v0.1 policy (schema-version 0.1) so
instance_min_count 0 is accepted by the broker.
Documented as PoC-only; production should split control onto a cf- route.
See docs/design/scale-to-zero.md 7.3.
…binding Park failed with CF 10008 'The service instance and the route are in different spaces': the route-service UPSI lives in the activator's space, but an app's routes live in the app's space, and CF forbids binding a route to a service instance in another space. On Park, resolve the app's space and share the UPSI into it (cf v3 ShareWithSpace) before binding. Adds cf wrapper methods GetAppSpaceGUID and ShareServiceInstanceWithSpace.
service_instance_sharing is disabled on the foundation, so sharing the activator's UPSI into an app's space failed (CF 330002). Instead, the activator now finds-or-creates a user-provided route-service instance (route_service_url = its own route) in the app's OWN space at park time, then binds the app's routes to it — no cross-space sharing, no feature flag, works regardless of foundation policy. - cf: replace GetUserProvidedServiceInstanceGUID + ShareServiceInstanceWithSpace with EnsureRouteServiceInstance(name, spaceGUID, routeServiceURL). - activator: parker ensures the per-space UPSI; new route_service_url config. - deploy: drop the deploy-time global UPSI creation; pass route_service_url via the extension file. See docs/design/scale-to-zero.md 3.
…an't wake from zero End-to-end validation on cf-deployment: scale-TO-zero works (bind v0.1 min_count=0 policy, park, scale web process to 0), but the wake request returns 404 — CF deregisters the route from Gorouter at 0 instances, so a bound route service never receives it. Keeping a zero-instance route routable requires the activator to be a live backend; every mechanism was evaluated: - route destination / route-service sharing: mTLS-correct but need route_sharing / service_instance_sharing flags (both disabled here); - routing-api HTTP register: plaintext-only (no tls backend fields) -> mTLS nogo; - NATS router.register direct: schema supports mTLS, but NATS is ASG-blocked and a CF app can only self-register its overlay IP, unreachable by Gorouter. Only mTLS-preserving path: CF route destinations, gated on the (disabled) route_sharing flag — a foundation-operator decision. Documented in 2.1.
…zero) Replace the route-service interception (which cannot wake a zero-instance app — CF deregisters the route at 0 instances, 404) with direct Gorouter route registration over NATS, the only mechanism that is mTLS-preserving, cross-space, and needs no disabled sharing feature flag. On Park, the activator publishes router.register for each of the app's route URIs advertising its OWN Gorouter-reachable mTLS backend tuple (host=CF_INSTANCE_IP, tls_port=external_tls_proxy from CF_INSTANCE_PORTS, server_cert_domain_san=CF_INSTANCE_GUID — derived from env, verified to match the activator's own gorouter registration). This keeps the parked route alive with the activator as a live mTLS backend. A refresh loop re-publishes under the 120s staleness threshold; Unpark publishes router.unregister. Loop A now identifies the app by Host (no route-service X-CF-Forwarded-Url) and forwards to the real app after Loop B deregisters the activator on the readiness Upsert. - activator: registrar.go (NATS publisher), instance.go (self tuple from env), parker.go rewritten to drive the registrar, handler.go Host-based, config Nats block, main wiring + refresh-loop runner. - cf: shrink CFClient to GetAppRoutes (drop route-service/UPSI methods). - deploy: nats-client UPSI (mTLS certs from credhub, materialized like syslog-client), activator ASG for NATS egress. Security note (deferred to a GitHub issue): the activator holds the NATS route-registration cert, i.e. the ability to register routes for any app on the foundation — Gorouter trusts NATS unconditionally. See docs/design/scale-to-zero.md.
|
|
||
| // forward proxies the held request to targetURL (the woken app's route). | ||
| func (h *Handler) forward(w http.ResponseWriter, r *http.Request, targetURL string, logger lager.Logger) { | ||
| outReq, err := http.NewRequestWithContext(r.Context(), r.Method, targetURL, r.Body) |
The activator crash-looped with 'nats: Authorization Violation': CF NATS authenticates with username 'nats' + password over TLS (the client cert only secures transport), but the registrar sent only the cert. Pass username/password (nats_password from credhub) in the activator-config nats block. Also make the NATS connection non-fatal at startup: use RetryOnFailedConnect so a NATS outage no longer crashes the activator (health/wake-trigger stay up), matching the readiness-watcher resilience.
…ss timeout Live validation reached the wake path: the NATS-registered activator DID receive the wake request (route stayed alive, Gorouter routed to the activator over mTLS) and called scale-up — but the engine ignored it with 'App in cooldown', because the scale-to-zero that parked the app moments earlier set a cooldown window. Add Trigger.BypassCooldown; the activator's scale-from-zero wake sets it so the engine scales up immediately despite the cooldown. Also raise the activator readiness timeout 30s->120s: a cold start (stage->start->healthy ->route Upsert) routinely exceeds 30s. See docs/design/scale-to-zero.md 9 (cooldown-vs-wake).
…n risks Note the unresolved design decision on how long the activator should hold a request while the app cold-starts: respect the app's/platform's configured startup+readiness health-check timeouts (so we don't 503 an app that is legitimately slow to start) vs. an upper cap so held requests and activator resources aren't tied up (and crash-looping apps don't pin them). Also refresh the open-questions list for the NATS-registrar design (cooldown -vs-wake now resolved via Trigger.BypassCooldown; route-service items replaced with NATS registration correctness + the NATS-cert security escalation).
Only file GitHub issues for findings in existing/merged code. Deferred items about the current in-progress branch / unmerged PR go in the design doc or PR description until the PR is review-ready — filing repo issues for not-yet-merged work is premature noise.
the routing-api event stream (loop B readiness signal) returns 401 in CI even with a valid routing_api_client token, so no Upsert arrives, the wake handler times out at 120s, and the held request never forwards. replace it with a NATS router.register subscriber on the registrar's existing connection: a registration for a parked URI from a backend that isn't the activator (filtered by instance guid / host+tls_port) is the readiness edge. confirmed from gorouter mbus/subscriber.go that route-emitter and the activator share one route bus. - new nats_route_events.go: subscriber + pure decode/self-filter - registrar exposes Conn() so the watcher reuses the mTLS connection - watcher unparks (deregisters self) before releasing the waiter to avoid load-balancing the forward back to the activator - remove routing_events.go, RoutingAPIConfig, routing_api_client uaa creds from the extension file; drop routing-api/go-sse/rata deps
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.