Skip to content

OPERATIONS.md

Latest commit

 

History

History
130 lines (107 loc) · 7.32 KB

OPERATIONS.md

File metadata and controls

130 lines (107 loc) · 7.32 KB
Raw

Operations Runbook — DevOps Observatory Web App

Deploy, configure, and operate the executive SPA (AWS Amplify Gen 2 + React) over the DevOps Agent hub backend. Satisfies Requirement 1.1: the app is reached through a single entry-point URL in a browser — no AWS console or scripts needed to use it.

Credential safety: the browser never holds AWS credentials. It holds only a Cognito session (JWT) and calls the API layer; every AWS call (S3, Neptune, Bedrock, Step Functions) runs server-side in a Lambda handler with a least-privilege IAM role. Never commit .env or amplify_outputs.json, and never put credentials in .env.example, amplify.yml, or Amplify env vars.

1. Prerequisites

  • Node.js 18+ / npm 9+.
  • AWS credentials for the hub account (123456789012) with permission to deploy the Amplify backend (Cognito, API Gateway, Lambda, Step Functions, ECS/Fargate, DynamoDB, IAM roles).
  • The hub backend already provisioned (S3 bucket, Neptune Analytics graph, Bedrock KB) via the repo scripts/.

2. Configuration values

Non-secret resource identifiers wired into the backend at deploy time. Set them as Amplify Hosting environment variables for the branch (App settings → Environment variables). Defaults live in amplify/backend.ts and mirror .env.example so synth/typecheck work unconfigured.

Env var (deploy) config.env key Purpose / value
HUB_BUCKET HUB_BUCKET S3 hub bucket — devops-agent-hub-123456789012-us-east-1
NEPTUNE_GRAPH_ENDPOINT (from NEPTUNE_GRAPH_NAME) Neptune Analytics endpoint; graph id derived from the host
KB_ID (resolved from KB_NAME) Bedrock knowledge base id used by /chat
KB_CHAT_MODEL_ARN KB_CHAT_MODEL_ARN Chat model / inference profile that synthesizes answers
APP_ORIGIN — Allowed browser origin(s) for CORS; comma-separated; * default
MGMT_ROLE_ARN — Required for refresh (Task 26). Role in the management account the refresh workers assume to list Organizations accounts + collect the mgmt account. Set to arn:aws:iam::210987654321:role/DevOpsAgentHubMgmtReadRole.
REFRESH_MAX_CONCURRENCY — Optional. Max accounts collected in parallel by the refresh Distributed Map (default 50).

MGMT_ROLE_ARN is required for the in-app refresh to work. Without it, the refresh state machine fails at the ListAccounts stage (organizations:ListAccounts is only callable from the management account or a delegated admin — not the hub account). The referenced role must (a) trust the hub account (123456789012) so the worker Lambdas can assume it, and (b) grant organizations:ListAccounts plus the aidevops:List*/Get*/Describe*/ValidateAwsAssociations read surface (so the management account's own agent spaces are collected too). The DevOpsAgentHubMgmtReadRole created for this deployment already satisfies both. Note: the DevOps Agent IAM action prefix is aidevops, not devops-agent.

  • Region is taken from the deploy region (the Amplify app's region), and is passed to handlers as HUB_REGION. Deploy in us-east-1 to co-locate with the hub backend.
  • The SPA reads the API base URL and chat stream URL from amplify_outputs.json (generated by the backend deploy). No API URLs are hardcoded. Local dev may override with VITE_API_BASE_URL / VITE_CHAT_STREAM_URL (non-secret URLs only — they are compiled into the browser bundle).

3. First-time hosting setup (once per app)

  1. In the Amplify console (hub account, us-east-1), create a new app and connect the git repository.
  2. Set the app root / build spec to the repo-root amplify.yml (it targets appRoot: webapp). Amplify supplies AWS_BRANCH and AWS_APP_ID.
  3. Add the environment variables from §2 for the branch (at minimum HUB_BUCKET, NEPTUNE_GRAPH_ENDPOINT, KB_ID, KB_CHAT_MODEL_ARN; set APP_ORIGIN after the first deploy once the app URL is known, then redeploy to tighten CORS).
  4. (Optional) For client-side deep links, add a hosting rewrite so unknown paths serve the SPA shell: rewrite /<*> → /index.html (200). The app switches views in-app from a single entry URL, so this is only needed if you later add URL-based routing.

4. Deploy (CI/CD)

Pushing to the connected branch triggers Amplify Hosting to run amplify.yml:

  1. Backend — npx ampx pipeline-deploy --branch $AWS_BRANCH --app-id $AWS_APP_ID deploys the CDK backend and writes amplify_outputs.json.
  2. Frontend — npm run build type-checks and builds the SPA to webapp/dist, baking in amplify_outputs.json.

The hosted SPA is served at the branch URL (https://<branch>.<app-id>.amplifyapp.com), the single entry-point URL for executives.

Manual / local deploy

cd webapp
npm ci
# Pipeline deploy (needs AWS creds + AWS_BRANCH/AWS_APP_ID):
AWS_BRANCH=main AWS_APP_ID=<app-id> npm run deploy
# Personal cloud sandbox for testing (writes amplify_outputs.json):
npm run sandbox          # tear down with: npm run sandbox:delete

5. Refresh operations

Data refresh is an in-app, Admin-only action — no scripts required (Requirement 10). The POST /refresh handler starts a Step Functions state machine that runs the pipeline as an ECS Fargate task in the order collect → transform → KB sync → graph reset-and-reload (webapp/amplify/refresh/, pipeline/Dockerfile).

  • Trigger: sign in as an Admin user → Refresh view → start refresh. Returns an acceptance within 5s; progress is polled via GET /refresh/status.
  • Single-execution lock: a DynamoDB conditional write rejects a second refresh while one is running (concurrent starts get "already running").
  • On success, Last_Sync_Date (manifest collectedAt) advances; on start or stage failure it is left unchanged and a stage-specific error is shown.
  • Executive users never see the refresh action, and the backend rejects any non-Admin POST /refresh.
  • Logs: the Fargate task streams to its CloudWatch log group (RefreshLogs); use it to find the failing stage on a failed run.

6. Rollback & troubleshooting

  • Rollback: redeploy a previous commit from the Amplify console (frontend + backend redeploy together via amplify.yml).
  • "API endpoint is not configured" in the SPA: the build did not produce amplify_outputs.json (backend phase failed) — check the backend build log.
  • CORS errors in the browser: set APP_ORIGIN to the exact SPA origin and redeploy; verify no trailing slash mismatch.
  • Chat/graph 5xx: confirm KB_ID / NEPTUNE_GRAPH_ENDPOINT env vars are set for the branch and point at the hub resources; check the handler's CloudWatch logs.
  • Backend deploy IAM errors: ensure the deploy credentials can create the backend resources listed in §1.