feat(ci): build nightly distribution images from source

[cdoern]

What

Adds a maintained nightly distribution-image build so users can pull a trustworthy "latest main" container, and retires the flaky test.pypi-based nightly docker build.

nightly-distro.yml (new)

• Builds distribution images from the current main source tree (INSTALL_MODE=editable) every night at 02:00 UTC. Building from source removes the test.pypi propagation wait that was the main flake source in the old nightly.
• Each architecture builds on its own native runner (ubuntu-24.04 + ubuntu-24.04-arm) — no QEMU at runtime — so each image is boot-smoke-tested on its real architecture before it's trusted.
• A push only happens if the image actually booted. Per-arch tags are then merged into a single multi-arch manifest via docker buildx imagetools create.
• Tags: :nightly, :<YYYYMMDD>, :<short-sha>. :latest is intentionally not touched — it stays owned by the release pipeline and means "latest stable release". Pull :nightly to track main.
• Per-PR gate: on PRs touching build-relevant paths, it builds + boot-smoke-tests the starter distro on amd64 only, no push — so a change that breaks server startup is caught before merge instead of at the next nightly. (providers-build.yml builds venv-only on PRs, so nothing booted a distro container per-PR before this.)

pypi.yml (changed)

• Removes the schedule trigger from the publish-docker-images job. Nightly images are now built from source by nightly-distro.yml. Release and workflow_dispatch image builds are unchanged, and the nightly test.pypi package publish is untouched.

scripts/smoke-test-distro.sh (new)

• Boots a built image with stubbed provider keys, polls /v1/health for OK, asserts /v1/models returns valid JSON, and dumps container logs on failure. Reusable locally and in CI.

Test plan

Static checks:

• actionlint on both workflows — clean
• shellcheck scripts/smoke-test-distro.sh — clean
• Matrix resolution simulated for pull_request / schedule / workflow_dispatch:
  • PR → [starter/amd64], push=false
  • schedule → [starter/amd64] [starter/arm64] [postgres-demo/amd64] [postgres-demo/arm64], push=true

End-to-end, locally (native arm64):

docker buildx build --load -f containers/Containerfile \
  --build-arg INSTALL_MODE=editable --build-arg DISTRO_NAME=starter \
  --tag ogxai/distribution-starter:smoke-local .
bash scripts/smoke-test-distro.sh ogxai/distribution-starter:smoke-local

Output:

Waiting for http://localhost:8321/v1/health (up to 150s)...
Server is healthy after 15s
Checking /v1/models endpoint...
Smoke test passed for ogxai/distribution-starter:smoke-local
EXIT CODE = 0

(The 401 provider model-refresh errors from the dummy keys are expected and non-fatal — the server still starts and serves.)

Notes / follow-ups

• Requires ubuntu-24.04-arm GitHub-hosted runners enabled for the org.
• Reuses existing DOCKERHUB_USERNAME / DOCKERHUB_TOKEN secrets — nothing new to provision.
• Curated nightly set is starter,postgres-demo; expand later as needed.

🤖 Generated with Claude Code