Skip to content

IMPLEMENTATION_PLAN.md

Latest commit

 

History

History
66 lines (51 loc) · 2.89 KB

IMPLEMENTATION_PLAN.md

File metadata and controls

66 lines (51 loc) · 2.89 KB
Raw

Implementation Plan

Phase 1 — Core Foundation (Complete)

Goal: Working end-to-end trace pipeline from SDK to storage.

Tasks

  • Implement Span dataclass with full lifecycle (start, end, error, to_dict)
  • Implement Tracer class with run and span management
  • Implement RunContext with context variable nesting
  • Implement JSONLExporter with file-based append
  • Build FastAPI server with /api/traces and /api/runs endpoints
  • SQLAlchemy models for Run, Span, Trace
  • Async database session with SQLite/PostgreSQL support
  • Basic run listing in dashboard

Deliverable: Record a run, export to JSONL, list runs via API.

Phase 2 — FinOps & Debugging (Complete)

Goal: Cost tracking, budget alerts, and rich debugging UI.

Tasks

  • trace_llm wrapper capturing model, tokens, cost, latency
  • trace_tool wrapper capturing tool name, input, output, errors
  • Cost analytics API (/api/costs/*) — summary, timeseries, breakdowns
  • Budget model with alert thresholds (/api/budgets)
  • /costs dashboard page with Recharts charts and budget status
  • /live SSE tail page for real-time span streaming
  • Waterfall timeline with cost overlay, error highlighting, nested spans
  • Run detail page with compare (diff) and replay tabs
  • Cost breakdown and token usage visualizations

Deliverable: Run an agent, see full trace with costs, budgets, and live tail.

Phase 3 — Integrations & Scale (Complete)

Goal: Provider-aware wrappers, hybrid client, batch ingestion.

Tasks

  • trace_openai() and trace_anthropic() provider-specific wrappers
  • HybridLLMClient (AGENTTRACE_LLM_MODE=sim|real)
  • LangChain callback handler with auto-instrumentation
  • Multi-agent demo with correlation_id
  • Batch ingestion endpoint (POST /api/traces/batch)
  • Buffered API exporter using batch endpoint
  • PostgreSQL support with Alembic migrations
  • Docker Compose with healthchecks, auto-migrate, auto-seed
  • Deploy guide (docs/DEPLOYMENT.md)
  • Benchmark script (scripts/benchmark.py)

Deliverable: Production-deployable system with provider wrappers and high-throughput batch export.

Phase 4 — Local engineering (Complete)

  • OTLP HTTP/JSON resource, scope, event, link, status, and timestamp support
  • Deterministic head/tail sampling with error/slow overrides and buffering
  • Pluggable in-memory realtime transport with optional Redis adapter
  • Grafana dashboard JSON artifact
  • Persisted cost/latency alerts with deduplication and acknowledgement
  • Single-tenant admin/ingestor/viewer RBAC
  • Redacted audit log and admin read endpoint
  • Reproducible offline portfolio evidence and wheel-install checks

Hosted/team workflows, external notifications/scheduling, OTLP protobuf/gRPC, and mandatory infrastructure services remain deferred product boundaries.