[LOW] Implement A/B Testing Framework for Model Deployment

[frankbria]

🎯 Overview

Add A/B testing capability to compare different model versions in production with traffic splitting.

✅ Acceptance Criteria

• Traffic splitting configuration (e.g., 90/10, 50/50)
• Multiple model variants in production
• Performance comparison dashboard
• Statistical significance testing
• Automatic winner selection
• Gradual rollout support
• Experiment management interface
• Results export and reporting

🏗️ Technical Requirements

• Backend: ABTestingService
• Traffic router for model selection
• Performance metrics aggregation by variant
• Statistical significance calculator
• Endpoints: POST /api/ab-tests, GET /api/ab-tests/{id}/results

🔗 Dependencies

#84 (API deployment), #85 (Deployment monitoring)

🏷️ Labels

`low-priority`, `backend`, `frontend`, `stage-8-deployment`, `experimentation`

⏱️ Estimated Effort

4-5 weeks

[traycerai]

Plan

Observations

The codebase already has a solid A/B testing foundation with backend routes (file:apps/backend/app/api/routes/ab_testing.py), database models (file:apps/backend/app/models/ab_test.py), service layer (file:apps/backend/app/services/ab_testing.py), and frontend service (file:apps/frontend/lib/services/abTesting.ts). The infrastructure includes experiment creation, variant assignment, metrics tracking, and statistical significance testing. However, key components are missing: production traffic routing integration, dashboard UI, automatic winner selection, gradual rollout capabilities, and results export functionality.

Approach

The implementation will extend the existing A/B testing infrastructure by integrating it with the production model serving pipeline, building a comprehensive dashboard UI, adding automation for experiment lifecycle management, and implementing gradual rollout capabilities. The approach focuses on minimal disruption to existing code by creating new middleware for traffic routing, new frontend components for the dashboard, background jobs for automation, and export utilities for reporting—all while leveraging the existing experiment management foundation.

Implementation Steps

1. Traffic Router Integration with Production API

Objective: Integrate A/B testing with the production model serving endpoint to enable automatic traffic splitting.

Backend Changes

Create Traffic Router Middleware (file:apps/backend/app/middleware/ab_traffic_router.py):

• Implement middleware that intercepts prediction requests to /production/v1/models/{model_id}/predict
• Check if the model is part of any active A/B test experiments
• Use ABTestingService.assign_variant() to determine which model variant to serve
• Extract user identifier from request headers or generate session-based identifier
• Route request to the assigned variant's model
• Track prediction metrics (latency, success/failure) via ABTestingService.track_prediction()

Modify Production Routes (file:apps/backend/app/api/routes/production.py):

• Update production_predict() endpoint to check for active experiments before serving
• Add optional query parameter experiment_id to explicitly participate in an experiment
• Add response header X-Experiment-Variant to indicate which variant was used
• Integrate with traffic router to select appropriate model based on experiment configuration

Add Experiment-Aware Prediction Endpoint (file:apps/backend/app/api/routes/ab_testing.py):

• Create new endpoint POST /api/v1/ab-testing/predict that explicitly routes through A/B testing
• Accept experiment_id and user_identifier in request
• Return variant assignment along with prediction results
• Automatically track metrics for the experiment

Integration Points

• Register traffic router middleware in file:apps/backend/app/main.py before production routes
• Ensure middleware only activates when active experiments exist for the requested model
• Add configuration flag to enable/disable A/B testing traffic routing

2. Performance Comparison Dashboard (Frontend)

Objective: Build comprehensive UI for viewing experiment results and comparing variant performance.

Create Dashboard Components

Experiment List Page (file:apps/frontend/app/experiments/page.tsx):

• Display all experiments with status badges (draft, running, paused, completed)
• Show key metrics: duration, total predictions, winner (if completed)
• Filter by status and search by name
• Quick actions: start, pause, complete, view details
• Use existing Card, Badge, and Button components from shadcn/ui

Experiment Detail Page (file:apps/frontend/app/experiments/[id]/page.tsx):

• Show experiment configuration (variants, traffic split, metrics)
• Display real-time metrics for each variant using existing chart components
• Show statistical comparison between variants
• Visualize traffic distribution with pie chart
• Display timeline of experiment lifecycle events
• Action buttons: pause/resume, complete, export results

Metrics Comparison Component (file:apps/frontend/components/ABTestMetricsComparison.tsx):

• Side-by-side comparison of variant performance
• Use LineChart component for time-series metrics (latency, error rate over time)
• Use BarChart component for aggregate metrics comparison
• Display statistical significance indicators (p-value, confidence level)
• Show lift percentage with visual indicators (positive/negative)
• Highlight winning variant when statistically significant

Variant Performance Cards (file:apps/frontend/components/VariantPerformanceCard.tsx):

• Individual cards for each variant showing key metrics
• Total predictions, error rate, average latency
• Custom metrics visualization
• Traffic percentage allocation
• Status indicator (active, paused, completed)

Dashboard Integration

• Add "Experiments" navigation item to main dashboard (file:apps/frontend/app/dashboard/page.tsx)
• Create quick stats card showing active experiments count
• Add link to experiments page in deployment section
• Use existing StatisticsDashboard component patterns for layout consistency

3. Automatic Winner Selection

Objective: Implement background job to automatically detect and declare experiment winners.

Backend Implementation

Create Experiment Monitor Service (file:apps/backend/app/services/experiment_monitor.py):

• Implement periodic check for running experiments (every 5-10 minutes)
• Use ABTestingService.check_experiment_completion() to evaluate completion criteria
• Automatically call ABTestingService.complete_experiment() when criteria met
• Send notifications (email/webhook) when experiment completes
• Log all automatic actions for audit trail

Add Scheduler Integration (file:apps/backend/app/scheduler.py):

• Use APScheduler or similar library for background job scheduling
• Register experiment monitor job to run periodically
• Ensure single instance execution in distributed environments
• Add health check endpoint for scheduler status

Enhance Completion Logic (file:apps/backend/app/services/ab_testing.py):

• Extend check_experiment_completion() with configurable rules:
  • Minimum sample size per variant reached
  • Statistical significance threshold met
  • Maximum duration exceeded
  • Early stopping criteria (clear winner with high confidence)
• Add configuration for auto-completion behavior per experiment
• Support manual override to prevent auto-completion

Configuration

• Add experiment auto-completion settings to file:apps/backend/app/config.py
• Environment variables for scheduler interval, minimum confidence level
• Feature flag to enable/disable automatic winner selection

4. Gradual Rollout Support

Objective: Enable dynamic traffic percentage adjustment for progressive rollouts.

Backend Enhancements

Add Rollout Schedule Model (file:apps/backend/app/models/ab_test.py):

• Extend ABTest model with rollout_schedule field
• Define schedule as list of stages: [{percentage: 10, duration_hours: 24}, {percentage: 50, duration_hours: 48}, ...]
• Add current_rollout_stage field to track progress
• Add auto_rollout_enabled boolean flag

Implement Rollout Manager (file:apps/backend/app/services/rollout_manager.py):

• Service to manage gradual traffic increases
• Check rollout schedule and update traffic percentages automatically
• Validate performance thresholds before advancing to next stage
• Rollback capability if metrics degrade
• Integration with experiment monitor for automated progression

Add Rollout API Endpoints (file:apps/backend/app/api/routes/ab_testing.py):

• POST /api/v1/ab-testing/experiments/{id}/rollout/configure - Set rollout schedule
• POST /api/v1/ab-testing/experiments/{id}/rollout/advance - Manually advance to next stage
• POST /api/v1/ab-testing/experiments/{id}/rollout/rollback - Rollback to previous stage
• GET /api/v1/ab-testing/experiments/{id}/rollout/status - Get current rollout status

Frontend Components

Rollout Configuration UI (file:apps/frontend/components/RolloutScheduleConfig.tsx):

• Visual editor for defining rollout stages
• Timeline visualization of planned rollout
• Performance threshold configuration per stage
• Enable/disable auto-advancement toggle

Rollout Progress Tracker (file:apps/frontend/components/RolloutProgressTracker.tsx):

• Display current rollout stage with progress bar
• Show metrics for current stage vs. thresholds
• Manual controls for advance/rollback
• Timeline view of completed stages

5. Experiment Management Interface

Objective: Comprehensive UI for creating and managing experiments.

Create Experiment Wizard

Experiment Creation Flow (file:apps/frontend/app/experiments/create/page.tsx):

• Multi-step wizard for experiment setup:
  1. Basic info (name, description, primary metric)
  2. Model selection (choose 2+ models to test)
  3. Traffic configuration (split percentages, user targeting)
  4. Test settings (duration, sample size, confidence level)
  5. Rollout schedule (optional gradual rollout)
  6. Review and create
• Form validation using existing patterns
• Preview of experiment configuration before creation

Experiment Settings Page (file:apps/frontend/app/experiments/[id]/settings/page.tsx):

• Edit experiment configuration (when in draft status)
• Update traffic split for running experiments
• Configure secondary metrics
• Set completion criteria
• Manage rollout schedule

Management Features

Bulk Actions Component (file:apps/frontend/components/ExperimentBulkActions.tsx):

• Select multiple experiments for bulk operations
• Pause/resume multiple experiments
• Archive completed experiments
• Export multiple experiment results

Experiment Templates (file:apps/frontend/components/ExperimentTemplates.tsx):

• Pre-configured templates for common scenarios (50/50 split, champion/challenger, multi-variant)
• Save custom configurations as templates
• Quick start from template

6. Results Export and Reporting

Objective: Enable comprehensive export of experiment results for analysis and reporting.

Backend Export Service

Create Export Service (file:apps/backend/app/services/ab_export_service.py):

• Generate CSV export with variant-level metrics
• Generate JSON export with full experiment data
• Create PDF report with visualizations and statistical analysis
• Support custom date ranges and metric selection
• Include statistical test results and confidence intervals

Add Export Endpoints (file:apps/backend/app/api/routes/ab_testing.py):

• GET /api/v1/ab-testing/experiments/{id}/export?format=csv|json|pdf - Export single experiment
• POST /api/v1/ab-testing/export/batch - Export multiple experiments
• GET /api/v1/ab-testing/experiments/{id}/report - Generate comprehensive report
• Support streaming for large exports

Frontend Export Features

Export Dialog Component (file:apps/frontend/components/ExperimentExportDialog.tsx):

• Select export format (CSV, JSON, PDF)
• Choose metrics to include
• Date range selection
• Include/exclude raw prediction data
• Download or email options

Report Generation (file:apps/frontend/components/ExperimentReportGenerator.tsx):

• Generate shareable HTML report
• Include charts, tables, and statistical analysis
• Executive summary section
• Recommendations based on results
• Export to PDF or share via link

7. Integration Testing and Validation

Objective: Ensure all components work together seamlessly.

Testing Strategy

Backend Tests (file:apps/backend/tests/test_api/test_ab_testing_integration.py):

• Test traffic router with active experiments
• Validate variant assignment consistency
• Test automatic winner selection logic
• Verify gradual rollout progression
• Test export functionality with various formats

Frontend Tests (file:apps/frontend/__tests__/experiments/):

• Test experiment creation wizard flow
• Validate dashboard metrics display
• Test rollout configuration UI
• Verify export functionality

End-to-End Tests:

• Create experiment → Start → Make predictions → View metrics → Complete → Export
• Test gradual rollout from 10% to 100%
• Verify automatic winner selection triggers correctly

8. Documentation and Deployment

Objective: Document the A/B testing framework and prepare for production deployment.

Documentation

User Guide (file:docs/ab-testing/USER_GUIDE.md):

• How to create and configure experiments
• Understanding metrics and statistical significance
• Best practices for A/B testing
• Interpreting results and making decisions

API Documentation (file:docs/ab-testing/API.md):

• Complete API reference for all A/B testing endpoints
• Code examples for integration
• Webhook configuration for notifications

Developer Guide (file:docs/ab-testing/DEVELOPER_GUIDE.md):

• Architecture overview
• Traffic routing implementation details
• Extending with custom metrics
• Troubleshooting guide

Deployment Considerations

• Add A/B testing configuration to environment variables
• Update deployment scripts to include scheduler service
• Configure monitoring and alerting for experiment health
• Set up database indexes for experiment queries
• Plan for data retention and archival of completed experiments

Architecture Diagram

sequenceDiagram
    participant User
    participant TrafficRouter
    participant ABTestService
    participant ModelA
    participant ModelB
    participant MetricsCollector
    participant ExperimentMonitor

    User->>TrafficRouter: POST /predict (model_id)
    TrafficRouter->>ABTestService: Check active experiments
    ABTestService->>ABTestService: Assign variant (hash user_id)
    ABTestService-->>TrafficRouter: Variant B assigned
    TrafficRouter->>ModelB: Forward prediction request
    ModelB-->>TrafficRouter: Prediction result
    TrafficRouter->>MetricsCollector: Track metrics (latency, success)
    TrafficRouter-->>User: Return prediction + variant info
    
    Note over ExperimentMonitor: Background Job (every 5 min)
    ExperimentMonitor->>ABTestService: Check completion criteria
    ABTestService->>ABTestService: Calculate statistics
    alt Criteria Met
        ABTestService->>ABTestService: Declare winner
        ABTestService->>User: Send notification
    end

Loading

Component Dependencies

Component	Dependencies	Purpose
Traffic Router Middleware	ABTestingService, ModelService	Route predictions to correct variant
Experiment Monitor	ABTestingService, Scheduler	Automatic winner selection
Rollout Manager	ABTest model, ABTestingService	Gradual traffic increase
Export Service	ABTest model, Reporting libraries	Generate reports
Dashboard UI	abTesting.ts service, Chart components	Visualize experiment results
Experiment Wizard	abTesting.ts service, Form components	Create experiments

Key Files to Modify/Create

Backend:

• Create: file:apps/backend/app/middleware/ab_traffic_router.py
• Create: file:apps/backend/app/services/experiment_monitor.py
• Create: file:apps/backend/app/services/rollout_manager.py
• Create: file:apps/backend/app/services/ab_export_service.py
• Create: file:apps/backend/app/scheduler.py
• Modify: file:apps/backend/app/api/routes/ab_testing.py (add new endpoints)
• Modify: file:apps/backend/app/api/routes/production.py (integrate traffic router)
• Modify: file:apps/backend/app/models/ab_test.py (add rollout fields)
• Modify: file:apps/backend/app/main.py (register middleware and scheduler)

Frontend:

• Create: file:apps/frontend/app/experiments/page.tsx
• Create: file:apps/frontend/app/experiments/[id]/page.tsx
• Create: file:apps/frontend/app/experiments/create/page.tsx
• Create: file:apps/frontend/components/ABTestMetricsComparison.tsx
• Create: file:apps/frontend/components/VariantPerformanceCard.tsx
• Create: file:apps/frontend/components/RolloutScheduleConfig.tsx
• Create: file:apps/frontend/components/RolloutProgressTracker.tsx
• Create: file:apps/frontend/components/ExperimentExportDialog.tsx
• Modify: file:apps/frontend/app/dashboard/page.tsx (add experiments section)

Tests:

• Create: file:apps/backend/tests/test_middleware/test_ab_traffic_router.py
• Create: file:apps/backend/tests/test_services/test_experiment_monitor.py
• Create: file:apps/backend/tests/test_services/test_rollout_manager.py
• Create: file:apps/frontend/__tests__/experiments/experiment-wizard.test.tsx

Documentation:

• Create: file:docs/ab-testing/USER_GUIDE.md
• Create: file:docs/ab-testing/API.md
• Create: file:docs/ab-testing/DEVELOPER_GUIDE.md

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase already has a solid A/B testing foundation with backend routes (`file:apps/backend/app/api/routes/ab_testing.py`), database models (`file:apps/backend/app/models/ab_test.py`), service layer (`file:apps/backend/app/services/ab_testing.py`), and frontend service (`file:apps/frontend/lib/services/abTesting.ts`). The infrastructure includes experiment creation, variant assignment, metrics tracking, and statistical significance testing. However, key components are missing: production traffic routing integration, dashboard UI, automatic winner selection, gradual rollout capabilities, and results export functionality.

## Approach

The implementation will extend the existing A/B testing infrastructure by integrating it with the production model serving pipeline, building a comprehensive dashboard UI, adding automation for experiment lifecycle management, and implementing gradual rollout capabilities. The approach focuses on minimal disruption to existing code by creating new middleware for traffic routing, new frontend components for the dashboard, background jobs for automation, and export utilities for reporting—all while leveraging the existing experiment management foundation.

## Implementation Steps

### 1. Traffic Router Integration with Production API

**Objective:** Integrate A/B testing with the production model serving endpoint to enable automatic traffic splitting.

#### Backend Changes

**Create Traffic Router Middleware** (`file:apps/backend/app/middleware/ab_traffic_router.py`):
- Implement middleware that intercepts prediction requests to `/production/v1/models/{model_id}/predict`
- Check if the model is part of any active A/B test experiments
- Use `ABTestingService.assign_variant()` to determine which model variant to serve
- Extract user identifier from request headers or generate session-based identifier
- Route request to the assigned variant's model
- Track prediction metrics (latency, success/failure) via `ABTestingService.track_prediction()`

**Modify Production Routes** (`file:apps/backend/app/api/routes/production.py`):
- Update `production_predict()` endpoint to check for active experiments before serving
- Add optional query parameter `experiment_id` to explicitly participate in an experiment
- Add response header `X-Experiment-Variant` to indicate which variant was used
- Integrate with traffic router to select appropriate model based on experiment configuration

**Add Experiment-Aware Prediction Endpoint** (`file:apps/backend/app/api/routes/ab_testing.py`):
- Create new endpoint `POST /api/v1/ab-testing/predict` that explicitly routes through A/B testing
- Accept experiment_id and user_identifier in request
- Return variant assignment along with prediction results
- Automatically track metrics for the experiment

#### Integration Points

- Register traffic router middleware in `file:apps/backend/app/main.py` before production routes
- Ensure middleware only activates when active experiments exist for the requested model
- Add configuration flag to enable/disable A/B testing traffic routing

### 2. Performance Comparison Dashboard (Frontend)

**Objective:** Build comprehensive UI for viewing experiment results and comparing variant performance.

#### Create Dashboard Components

**Experiment List Page** (`file:apps/frontend/app/experiments/page.tsx`):
- Display all experiments with status badges (draft, running, paused, completed)
- Show key metrics: duration, total predictions, winner (if completed)
- Filter by status and search by name
- Quick actions: start, pause, complete, view details
- Use existing Card, Badge, and Button components from shadcn/ui

**Experiment Detail Page** (`file:apps/frontend/app/experiments/[id]/page.tsx`):
- Show experiment configuration (variants, traffic split, metrics)
- Display real-time metrics for each variant using existing chart components
- Show statistical comparison between variants
- Visualize traffic distribution with pie chart
- Display timeline of experiment lifecycle events
- Action buttons: pause/resume, complete, export results

**Metrics Comparison Component** (`file:apps/frontend/components/ABTestMetricsComparison.tsx`):
- Side-by-side comparison of variant performance
- Use `LineChart` component for time-series metrics (latency, error rate over time)
- Use `BarChart` component for aggregate metrics comparison
- Display statistical significance indicators (p-value, confidence level)
- Show lift percentage with visual indicators (positive/negative)
- Highlight winning variant when statistically significant

**Variant Performance Cards** (`file:apps/frontend/components/VariantPerformanceCard.tsx`):
- Individual cards for each variant showing key metrics
- Total predictions, error rate, average latency
- Custom metrics visualization
- Traffic percentage allocation
- Status indicator (active, paused, completed)

#### Dashboard Integration

- Add "Experiments" navigation item to main dashboard (`file:apps/frontend/app/dashboard/page.tsx`)
- Create quick stats card showing active experiments count
- Add link to experiments page in deployment section
- Use existing `StatisticsDashboard` component patterns for layout consistency

### 3. Automatic Winner Selection

**Objective:** Implement background job to automatically detect and declare experiment winners.

#### Backend Implementation

**Create Experiment Monitor Service** (`file:apps/backend/app/services/experiment_monitor.py`):
- Implement periodic check for running experiments (every 5-10 minutes)
- Use `ABTestingService.check_experiment_completion()` to evaluate completion criteria
- Automatically call `ABTestingService.complete_experiment()` when criteria met
- Send notifications (email/webhook) when experiment completes
- Log all automatic actions for audit trail

**Add Scheduler Integration** (`file:apps/backend/app/scheduler.py`):
- Use APScheduler or similar library for background job scheduling
- Register experiment monitor job to run periodically
- Ensure single instance execution in distributed environments
- Add health check endpoint for scheduler status

**Enhance Completion Logic** (`file:apps/backend/app/services/ab_testing.py`):
- Extend `check_experiment_completion()` with configurable rules:
  - Minimum sample size per variant reached
  - Statistical significance threshold met
  - Maximum duration exceeded
  - Early stopping criteria (clear winner with high confidence)
- Add configuration for auto-completion behavior per experiment
- Support manual override to prevent auto-completion

#### Configuration

- Add experiment auto-completion settings to `file:apps/backend/app/config.py`
- Environment variables for scheduler interval, minimum confidence level
- Feature flag to enable/disable automatic winner selection

### 4. Gradual Rollout Support

**Objective:** Enable dynamic traffic percentage adjustment for progressive rollouts.

#### Backend Enhancements

**Add Rollout Schedule Model** (`file:apps/backend/app/models/ab_test.py`):
- Extend `ABTest` model with `rollout_schedule` field
- Define schedule as list of stages: `[{percentage: 10, duration_hours: 24}, {percentage: 50, duration_hours: 48}, ...]`
- Add `current_rollout_stage` field to track progress
- Add `auto_rollout_enabled` boolean flag

**Implement Rollout Manager** (`file:apps/backend/app/services/rollout_manager.py`):
- Service to manage gradual traffic increases
- Check rollout schedule and update traffic percentages automatically
- Validate performance thresholds before advancing to next stage
- Rollback capability if metrics degrade
- Integration with experiment monitor for automated progression

**Add Rollout API Endpoints** (`file:apps/backend/app/api/routes/ab_testing.py`):
- `POST /api/v1/ab-testing/experiments/{id}/rollout/configure` - Set rollout schedule
- `POST /api/v1/ab-testing/experiments/{id}/rollout/advance` - Manually advance to next stage
- `POST /api/v1/ab-testing/experiments/{id}/rollout/rollback` - Rollback to previous stage
- `GET /api/v1/ab-testing/experiments/{id}/rollout/status` - Get current rollout status

#### Frontend Components

**Rollout Configuration UI** (`file:apps/frontend/components/RolloutScheduleConfig.tsx`):
- Visual editor for defining rollout stages
- Timeline visualization of planned rollout
- Performance threshold configuration per stage
- Enable/disable auto-advancement toggle

**Rollout Progress Tracker** (`file:apps/frontend/components/RolloutProgressTracker.tsx`):
- Display current rollout stage with progress bar
- Show metrics for current stage vs. thresholds
- Manual controls for advance/rollback
- Timeline view of completed stages

### 5. Experiment Management Interface

**Objective:** Comprehensive UI for creating and managing experiments.

#### Create Experiment Wizard

**Experiment Creation Flow** (`file:apps/frontend/app/experiments/create/page.tsx`):
- Multi-step wizard for experiment setup:
  1. Basic info (name, description, primary metric)
  2. Model selection (choose 2+ models to test)
  3. Traffic configuration (split percentages, user targeting)
  4. Test settings (duration, sample size, confidence level)
  5. Rollout schedule (optional gradual rollout)
  6. Review and create
- Form validation using existing patterns
- Preview of experiment configuration before creation

**Experiment Settings Page** (`file:apps/frontend/app/experiments/[id]/settings/page.tsx`):
- Edit experiment configuration (when in draft status)
- Update traffic split for running experiments
- Configure secondary metrics
- Set completion criteria
- Manage rollout schedule

#### Management Features

**Bulk Actions Component** (`file:apps/frontend/components/ExperimentBulkActions.tsx`):
- Select multiple experiments for bulk operations
- Pause/resume multiple experiments
- Archive completed experiments
- Export multiple experiment results

**Experiment Templates** (`file:apps/frontend/components/ExperimentTemplates.tsx`):
- Pre-configured templates for common scenarios (50/50 split, champion/challenger, multi-variant)
- Save custom configurations as templates
- Quick start from template

### 6. Results Export and Reporting

**Objective:** Enable comprehensive export of experiment results for analysis and reporting.

#### Backend Export Service

**Create Export Service** (`file:apps/backend/app/services/ab_export_service.py`):
- Generate CSV export with variant-level metrics
- Generate JSON export with full experiment data
- Create PDF report with visualizations and statistical analysis
- Support custom date ranges and metric selection
- Include statistical test results and confidence intervals

**Add Export Endpoints** (`file:apps/backend/app/api/routes/ab_testing.py`):
- `GET /api/v1/ab-testing/experiments/{id}/export?format=csv|json|pdf` - Export single experiment
- `POST /api/v1/ab-testing/export/batch` - Export multiple experiments
- `GET /api/v1/ab-testing/experiments/{id}/report` - Generate comprehensive report
- Support streaming for large exports

#### Frontend Export Features

**Export Dialog Component** (`file:apps/frontend/components/ExperimentExportDialog.tsx`):
- Select export format (CSV, JSON, PDF)
- Choose metrics to include
- Date range selection
- Include/exclude raw prediction data
- Download or email options

**Report Generation** (`file:apps/frontend/components/ExperimentReportGenerator.tsx`):
- Generate shareable HTML report
- Include charts, tables, and statistical analysis
- Executive summary section
- Recommendations based on results
- Export to PDF or share via link

### 7. Integration Testing and Validation

**Objective:** Ensure all components work together seamlessly.

#### Testing Strategy

**Backend Tests** (`file:apps/backend/tests/test_api/test_ab_testing_integration.py`):
- Test traffic router with active experiments
- Validate variant assignment consistency
- Test automatic winner selection logic
- Verify gradual rollout progression
- Test export functionality with various formats

**Frontend Tests** (`file:apps/frontend/__tests__/experiments/`):
- Test experiment creation wizard flow
- Validate dashboard metrics display
- Test rollout configuration UI
- Verify export functionality

**End-to-End Tests**:
- Create experiment → Start → Make predictions → View metrics → Complete → Export
- Test gradual rollout from 10% to 100%
- Verify automatic winner selection triggers correctly

### 8. Documentation and Deployment

**Objective:** Document the A/B testing framework and prepare for production deployment.

#### Documentation

**User Guide** (`file:docs/ab-testing/USER_GUIDE.md`):
- How to create and configure experiments
- Understanding metrics and statistical significance
- Best practices for A/B testing
- Interpreting results and making decisions

**API Documentation** (`file:docs/ab-testing/API.md`):
- Complete API reference for all A/B testing endpoints
- Code examples for integration
- Webhook configuration for notifications

**Developer Guide** (`file:docs/ab-testing/DEVELOPER_GUIDE.md`):
- Architecture overview
- Traffic routing implementation details
- Extending with custom metrics
- Troubleshooting guide

#### Deployment Considerations

- Add A/B testing configuration to environment variables
- Update deployment scripts to include scheduler service
- Configure monitoring and alerting for experiment health
- Set up database indexes for experiment queries
- Plan for data retention and archival of completed experiments

## Architecture Diagram

```mermaid
sequenceDiagram
    participant User
    participant TrafficRouter
    participant ABTestService
    participant ModelA
    participant ModelB
    participant MetricsCollector
    participant ExperimentMonitor

    User->>TrafficRouter: POST /predict (model_id)
    TrafficRouter->>ABTestService: Check active experiments
    ABTestService->>ABTestService: Assign variant (hash user_id)
    ABTestService-->>TrafficRouter: Variant B assigned
    TrafficRouter->>ModelB: Forward prediction request
    ModelB-->>TrafficRouter: Prediction result
    TrafficRouter->>MetricsCollector: Track metrics (latency, success)
    TrafficRouter-->>User: Return prediction + variant info
    
    Note over ExperimentMonitor: Background Job (every 5 min)
    ExperimentMonitor->>ABTestService: Check completion criteria
    ABTestService->>ABTestService: Calculate statistics
    alt Criteria Met
        ABTestService->>ABTestService: Declare winner
        ABTestService->>User: Send notification
    end
```

## Component Dependencies

| Component | Dependencies | Purpose |
|-----------|-------------|---------|
| Traffic Router Middleware | ABTestingService, ModelService | Route predictions to correct variant |
| Experiment Monitor | ABTestingService, Scheduler | Automatic winner selection |
| Rollout Manager | ABTest model, ABTestingService | Gradual traffic increase |
| Export Service | ABTest model, Reporting libraries | Generate reports |
| Dashboard UI | abTesting.ts service, Chart components | Visualize experiment results |
| Experiment Wizard | abTesting.ts service, Form components | Create experiments |

## Key Files to Modify/Create

**Backend:**
- Create: `file:apps/backend/app/middleware/ab_traffic_router.py`
- Create: `file:apps/backend/app/services/experiment_monitor.py`
- Create: `file:apps/backend/app/services/rollout_manager.py`
- Create: `file:apps/backend/app/services/ab_export_service.py`
- Create: `file:apps/backend/app/scheduler.py`
- Modify: `file:apps/backend/app/api/routes/ab_testing.py` (add new endpoints)
- Modify: `file:apps/backend/app/api/routes/production.py` (integrate traffic router)
- Modify: `file:apps/backend/app/models/ab_test.py` (add rollout fields)
- Modify: `file:apps/backend/app/main.py` (register middleware and scheduler)

**Frontend:**
- Create: `file:apps/frontend/app/experiments/page.tsx`
- Create: `file:apps/frontend/app/experiments/[id]/page.tsx`
- Create: `file:apps/frontend/app/experiments/create/page.tsx`
- Create: `file:apps/frontend/components/ABTestMetricsComparison.tsx`
- Create: `file:apps/frontend/components/VariantPerformanceCard.tsx`
- Create: `file:apps/frontend/components/RolloutScheduleConfig.tsx`
- Create: `file:apps/frontend/components/RolloutProgressTracker.tsx`
- Create: `file:apps/frontend/components/ExperimentExportDialog.tsx`
- Modify: `file:apps/frontend/app/dashboard/page.tsx` (add experiments section)

**Tests:**
- Create: `file:apps/backend/tests/test_middleware/test_ab_traffic_router.py`
- Create: `file:apps/backend/tests/test_services/test_experiment_monitor.py`
- Create: `file:apps/backend/tests/test_services/test_rollout_manager.py`
- Create: `file:apps/frontend/__tests__/experiments/experiment-wizard.test.tsx`

**Documentation:**
- Create: `file:docs/ab-testing/USER_GUIDE.md`
- Create: `file:docs/ab-testing/API.md`
- Create: `file:docs/ab-testing/DEVELOPER_GUIDE.md`

---

Execution Information

Branch: main
Commit: 1963a18

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

_{You can configure auto-planning by selecting labels in the issue_enrichment configuration.}

🔗 Similar Issues

Related Issues

• [BLOCKER] Implement Comprehensive Model Evaluation Dashboard with Performance Metrics #79
• [LOW] Implement Model Performance Degradation Detection and Alerts #96
• [BLOCKER] Implement Model Versioning and History Tracking System #78
• [HIGH] Implement One-Click REST API Deployment #84
• [HIGH] Implement Deployment Monitoring Dashboard #85

👤 Suggested Assignees

• frankbria

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

New Plan

Observations

The codebase already has a solid A/B testing foundation with backend routes (file:apps/backend/app/api/routes/ab_testing.py), database models (file:apps/backend/app/models/ab_test.py), service layer (file:apps/backend/app/services/ab_testing.py), and frontend service (file:apps/frontend/lib/services/abTesting.ts). The infrastructure includes experiment creation, variant assignment, metrics tracking, and statistical significance testing. However, key components are missing: production traffic routing integration, dashboard UI, automatic winner selection, gradual rollout capabilities, and results export functionality.

Approach

The implementation will extend the existing A/B testing infrastructure by integrating it with the production model serving pipeline, building a comprehensive dashboard UI, adding automation for experiment lifecycle management, and implementing gradual rollout capabilities. The approach focuses on minimal disruption to existing code by creating new middleware for traffic routing, new frontend components for the dashboard, background jobs for automation, and export utilities for reporting—all while leveraging the existing experiment management foundation.

Implementation Steps

1. Traffic Router Integration with Production API

Objective: Integrate A/B testing with the production model serving endpoint to enable automatic traffic splitting.

Backend Changes

Create Traffic Router Middleware (file:apps/backend/app/middleware/ab_traffic_router.py):

• Implement middleware that intercepts prediction requests to /production/v1/models/{model_id}/predict
• Check if the model is part of any active A/B test experiments
• Use ABTestingService.assign_variant() to determine which model variant to serve
• Extract user identifier from request headers or generate session-based identifier
• Route request to the assigned variant's model
• Track prediction metrics (latency, success/failure) via ABTestingService.track_prediction()

Modify Production Routes (file:apps/backend/app/api/routes/production.py):

• Update production_predict() endpoint to check for active experiments before serving
• Add optional query parameter experiment_id to explicitly participate in an experiment
• Add response header X-Experiment-Variant to indicate which variant was used
• Integrate with traffic router to select appropriate model based on experiment configuration

Add Experiment-Aware Prediction Endpoint (file:apps/backend/app/api/routes/ab_testing.py):

• Create new endpoint POST /api/v1/ab-testing/predict that explicitly routes through A/B testing
• Accept experiment_id and user_identifier in request
• Return variant assignment along with prediction results
• Automatically track metrics for the experiment

Integration Points

• Register traffic router middleware in file:apps/backend/app/main.py before production routes
• Ensure middleware only activates when active experiments exist for the requested model
• Add configuration flag to enable/disable A/B testing traffic routing

2. Performance Comparison Dashboard (Frontend)

Objective: Build comprehensive UI for viewing experiment results and comparing variant performance.

Create Dashboard Components

Experiment List Page (file:apps/frontend/app/experiments/page.tsx):

• Display all experiments with status badges (draft, running, paused, completed)
• Show key metrics: duration, total predictions, winner (if completed)
• Filter by status and search by name
• Quick actions: start, pause, complete, view details
• Use existing Card, Badge, and Button components from shadcn/ui

Experiment Detail Page (file:apps/frontend/app/experiments/[id]/page.tsx):

• Show experiment configuration (variants, traffic split, metrics)
• Display real-time metrics for each variant using existing chart components
• Show statistical comparison between variants
• Visualize traffic distribution with pie chart
• Display timeline of experiment lifecycle events
• Action buttons: pause/resume, complete, export results

Metrics Comparison Component (file:apps/frontend/components/ABTestMetricsComparison.tsx):

• Side-by-side comparison of variant performance
• Use LineChart component for time-series metrics (latency, error rate over time)
• Use BarChart component for aggregate metrics comparison
• Display statistical significance indicators (p-value, confidence level)
• Show lift percentage with visual indicators (positive/negative)
• Highlight winning variant when statistically significant

Variant Performance Cards (file:apps/frontend/components/VariantPerformanceCard.tsx):

• Individual cards for each variant showing key metrics
• Total predictions, error rate, average latency
• Custom metrics visualization
• Traffic percentage allocation
• Status indicator (active, paused, completed)

Dashboard Integration

• Add "Experiments" navigation item to main dashboard (file:apps/frontend/app/dashboard/page.tsx)
• Create quick stats card showing active experiments count
• Add link to experiments page in deployment section
• Use existing StatisticsDashboard component patterns for layout consistency

3. Automatic Winner Selection

Objective: Implement background job to automatically detect and declare experiment winners.

Backend Implementation

Create Experiment Monitor Service (file:apps/backend/app/services/experiment_monitor.py):

• Implement periodic check for running experiments (every 5-10 minutes)
• Use ABTestingService.check_experiment_completion() to evaluate completion criteria
• Automatically call ABTestingService.complete_experiment() when criteria met
• Send notifications (email/webhook) when experiment completes
• Log all automatic actions for audit trail

Add Scheduler Integration (file:apps/backend/app/scheduler.py):

• Use APScheduler or similar library for background job scheduling
• Register experiment monitor job to run periodically
• Ensure single instance execution in distributed environments
• Add health check endpoint for scheduler status

Enhance Completion Logic (file:apps/backend/app/services/ab_testing.py):

• Extend check_experiment_completion() with configurable rules:
  • Minimum sample size per variant reached
  • Statistical significance threshold met
  • Maximum duration exceeded
  • Early stopping criteria (clear winner with high confidence)
• Add configuration for auto-completion behavior per experiment
• Support manual override to prevent auto-completion

Configuration

• Add experiment auto-completion settings to file:apps/backend/app/config.py
• Environment variables for scheduler interval, minimum confidence level
• Feature flag to enable/disable automatic winner selection

4. Gradual Rollout Support

Objective: Enable dynamic traffic percentage adjustment for progressive rollouts.

Backend Enhancements

Add Rollout Schedule Model (file:apps/backend/app/models/ab_test.py):

• Extend ABTest model with rollout_schedule field
• Define schedule as list of stages: [{percentage: 10, duration_hours: 24}, {percentage: 50, duration_hours: 48}, ...]
• Add current_rollout_stage field to track progress
• Add auto_rollout_enabled boolean flag

Implement Rollout Manager (file:apps/backend/app/services/rollout_manager.py):

• Service to manage gradual traffic increases
• Check rollout schedule and update traffic percentages automatically
• Validate performance thresholds before advancing to next stage
• Rollback capability if metrics degrade
• Integration with experiment monitor for automated progression

Add Rollout API Endpoints (file:apps/backend/app/api/routes/ab_testing.py):

• POST /api/v1/ab-testing/experiments/{id}/rollout/configure - Set rollout schedule
• POST /api/v1/ab-testing/experiments/{id}/rollout/advance - Manually advance to next stage
• POST /api/v1/ab-testing/experiments/{id}/rollout/rollback - Rollback to previous stage
• GET /api/v1/ab-testing/experiments/{id}/rollout/status - Get current rollout status

Frontend Components

Rollout Configuration UI (file:apps/frontend/components/RolloutScheduleConfig.tsx):

• Visual editor for defining rollout stages
• Timeline visualization of planned rollout
• Performance threshold configuration per stage
• Enable/disable auto-advancement toggle

Rollout Progress Tracker (file:apps/frontend/components/RolloutProgressTracker.tsx):

• Display current rollout stage with progress bar
• Show metrics for current stage vs. thresholds
• Manual controls for advance/rollback
• Timeline view of completed stages

5. Experiment Management Interface

Objective: Comprehensive UI for creating and managing experiments.

Create Experiment Wizard

Experiment Creation Flow (file:apps/frontend/app/experiments/create/page.tsx):

• Multi-step wizard for experiment setup:
  1. Basic info (name, description, primary metric)
  2. Model selection (choose 2+ models to test)
  3. Traffic configuration (split percentages, user targeting)
  4. Test settings (duration, sample size, confidence level)
  5. Rollout schedule (optional gradual rollout)
  6. Review and create
• Form validation using existing patterns
• Preview of experiment configuration before creation

Experiment Settings Page (file:apps/frontend/app/experiments/[id]/settings/page.tsx):

• Edit experiment configuration (when in draft status)
• Update traffic split for running experiments
• Configure secondary metrics
• Set completion criteria
• Manage rollout schedule

Management Features

Bulk Actions Component (file:apps/frontend/components/ExperimentBulkActions.tsx):

• Select multiple experiments for bulk operations
• Pause/resume multiple experiments
• Archive completed experiments
• Export multiple experiment results

Experiment Templates (file:apps/frontend/components/ExperimentTemplates.tsx):

• Pre-configured templates for common scenarios (50/50 split, champion/challenger, multi-variant)
• Save custom configurations as templates
• Quick start from template

6. Results Export and Reporting

Objective: Enable comprehensive export of experiment results for analysis and reporting.

Backend Export Service

Create Export Service (file:apps/backend/app/services/ab_export_service.py):

• Generate CSV export with variant-level metrics
• Generate JSON export with full experiment data
• Create PDF report with visualizations and statistical analysis
• Support custom date ranges and metric selection
• Include statistical test results and confidence intervals

Add Export Endpoints (file:apps/backend/app/api/routes/ab_testing.py):

• GET /api/v1/ab-testing/experiments/{id}/export?format=csv|json|pdf - Export single experiment
• POST /api/v1/ab-testing/export/batch - Export multiple experiments
• GET /api/v1/ab-testing/experiments/{id}/report - Generate comprehensive report
• Support streaming for large exports

Frontend Export Features

Export Dialog Component (file:apps/frontend/components/ExperimentExportDialog.tsx):

• Select export format (CSV, JSON, PDF)
• Choose metrics to include
• Date range selection
• Include/exclude raw prediction data
• Download or email options

Report Generation (file:apps/frontend/components/ExperimentReportGenerator.tsx):

• Generate shareable HTML report
• Include charts, tables, and statistical analysis
• Executive summary section
• Recommendations based on results
• Export to PDF or share via link

7. Integration Testing and Validation

Objective: Ensure all components work together seamlessly.

Testing Strategy

Backend Tests (file:apps/backend/tests/test_api/test_ab_testing_integration.py):

• Test traffic router with active experiments
• Validate variant assignment consistency
• Test automatic winner selection logic
• Verify gradual rollout progression
• Test export functionality with various formats

Frontend Tests (file:apps/frontend/__tests__/experiments/):

• Test experiment creation wizard flow
• Validate dashboard metrics display
• Test rollout configuration UI
• Verify export functionality

End-to-End Tests:

• Create experiment → Start → Make predictions → View metrics → Complete → Export
• Test gradual rollout from 10% to 100%
• Verify automatic winner selection triggers correctly

8. Documentation and Deployment

Objective: Document the A/B testing framework and prepare for production deployment.

Documentation

User Guide (file:docs/ab-testing/USER_GUIDE.md):

• How to create and configure experiments
• Understanding metrics and statistical significance
• Best practices for A/B testing
• Interpreting results and making decisions

API Documentation (file:docs/ab-testing/API.md):

• Complete API reference for all A/B testing endpoints
• Code examples for integration
• Webhook configuration for notifications

Developer Guide (file:docs/ab-testing/DEVELOPER_GUIDE.md):

• Architecture overview
• Traffic routing implementation details
• Extending with custom metrics
• Troubleshooting guide

Deployment Considerations

• Add A/B testing configuration to environment variables
• Update deployment scripts to include scheduler service
• Configure monitoring and alerting for experiment health
• Set up database indexes for experiment queries
• Plan for data retention and archival of completed experiments

Architecture Diagram

sequenceDiagram
    participant User
    participant TrafficRouter
    participant ABTestService
    participant ModelA
    participant ModelB
    participant MetricsCollector
    participant ExperimentMonitor

    User->>TrafficRouter: POST /predict (model_id)
    TrafficRouter->>ABTestService: Check active experiments
    ABTestService->>ABTestService: Assign variant (hash user_id)
    ABTestService-->>TrafficRouter: Variant B assigned
    TrafficRouter->>ModelB: Forward prediction request
    ModelB-->>TrafficRouter: Prediction result
    TrafficRouter->>MetricsCollector: Track metrics (latency, success)
    TrafficRouter-->>User: Return prediction + variant info
    
    Note over ExperimentMonitor: Background Job (every 5 min)
    ExperimentMonitor->>ABTestService: Check completion criteria
    ABTestService->>ABTestService: Calculate statistics
    alt Criteria Met
        ABTestService->>ABTestService: Declare winner
        ABTestService->>User: Send notification
    end

Loading

Component Dependencies

Component	Dependencies	Purpose
Traffic Router Middleware	ABTestingService, ModelService	Route predictions to correct variant
Experiment Monitor	ABTestingService, Scheduler	Automatic winner selection
Rollout Manager	ABTest model, ABTestingService	Gradual traffic increase
Export Service	ABTest model, Reporting libraries	Generate reports
Dashboard UI	abTesting.ts service, Chart components	Visualize experiment results
Experiment Wizard	abTesting.ts service, Form components	Create experiments

Key Files to Modify/Create

Backend:

• Create: file:apps/backend/app/middleware/ab_traffic_router.py
• Create: file:apps/backend/app/services/experiment_monitor.py
• Create: file:apps/backend/app/services/rollout_manager.py
• Create: file:apps/backend/app/services/ab_export_service.py
• Create: file:apps/backend/app/scheduler.py
• Modify: file:apps/backend/app/api/routes/ab_testing.py (add new endpoints)
• Modify: file:apps/backend/app/api/routes/production.py (integrate traffic router)
• Modify: file:apps/backend/app/models/ab_test.py (add rollout fields)
• Modify: file:apps/backend/app/main.py (register middleware and scheduler)

Frontend:

• Create: file:apps/frontend/app/experiments/page.tsx
• Create: file:apps/frontend/app/experiments/[id]/page.tsx
• Create: file:apps/frontend/app/experiments/create/page.tsx
• Create: file:apps/frontend/components/ABTestMetricsComparison.tsx
• Create: file:apps/frontend/components/VariantPerformanceCard.tsx
• Create: file:apps/frontend/components/RolloutScheduleConfig.tsx
• Create: file:apps/frontend/components/RolloutProgressTracker.tsx
• Create: file:apps/frontend/components/ExperimentExportDialog.tsx
• Modify: file:apps/frontend/app/dashboard/page.tsx (add experiments section)

Tests:

• Create: file:apps/backend/tests/test_middleware/test_ab_traffic_router.py
• Create: file:apps/backend/tests/test_services/test_experiment_monitor.py
• Create: file:apps/backend/tests/test_services/test_rollout_manager.py
• Create: file:apps/frontend/__tests__/experiments/experiment-wizard.test.tsx

Documentation:

• Create: file:docs/ab-testing/USER_GUIDE.md
• Create: file:docs/ab-testing/API.md
• Create: file:docs/ab-testing/DEVELOPER_GUIDE.md

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase already has a solid A/B testing foundation with backend routes (`file:apps/backend/app/api/routes/ab_testing.py`), database models (`file:apps/backend/app/models/ab_test.py`), service layer (`file:apps/backend/app/services/ab_testing.py`), and frontend service (`file:apps/frontend/lib/services/abTesting.ts`). The infrastructure includes experiment creation, variant assignment, metrics tracking, and statistical significance testing. However, key components are missing: production traffic routing integration, dashboard UI, automatic winner selection, gradual rollout capabilities, and results export functionality.

## Approach

The implementation will extend the existing A/B testing infrastructure by integrating it with the production model serving pipeline, building a comprehensive dashboard UI, adding automation for experiment lifecycle management, and implementing gradual rollout capabilities. The approach focuses on minimal disruption to existing code by creating new middleware for traffic routing, new frontend components for the dashboard, background jobs for automation, and export utilities for reporting—all while leveraging the existing experiment management foundation.

## Implementation Steps

### 1. Traffic Router Integration with Production API

**Objective:** Integrate A/B testing with the production model serving endpoint to enable automatic traffic splitting.

#### Backend Changes

**Create Traffic Router Middleware** (`file:apps/backend/app/middleware/ab_traffic_router.py`):
- Implement middleware that intercepts prediction requests to `/production/v1/models/{model_id}/predict`
- Check if the model is part of any active A/B test experiments
- Use `ABTestingService.assign_variant()` to determine which model variant to serve
- Extract user identifier from request headers or generate session-based identifier
- Route request to the assigned variant's model
- Track prediction metrics (latency, success/failure) via `ABTestingService.track_prediction()`

**Modify Production Routes** (`file:apps/backend/app/api/routes/production.py`):
- Update `production_predict()` endpoint to check for active experiments before serving
- Add optional query parameter `experiment_id` to explicitly participate in an experiment
- Add response header `X-Experiment-Variant` to indicate which variant was used
- Integrate with traffic router to select appropriate model based on experiment configuration

**Add Experiment-Aware Prediction Endpoint** (`file:apps/backend/app/api/routes/ab_testing.py`):
- Create new endpoint `POST /api/v1/ab-testing/predict` that explicitly routes through A/B testing
- Accept experiment_id and user_identifier in request
- Return variant assignment along with prediction results
- Automatically track metrics for the experiment

#### Integration Points

- Register traffic router middleware in `file:apps/backend/app/main.py` before production routes
- Ensure middleware only activates when active experiments exist for the requested model
- Add configuration flag to enable/disable A/B testing traffic routing

### 2. Performance Comparison Dashboard (Frontend)

**Objective:** Build comprehensive UI for viewing experiment results and comparing variant performance.

#### Create Dashboard Components

**Experiment List Page** (`file:apps/frontend/app/experiments/page.tsx`):
- Display all experiments with status badges (draft, running, paused, completed)
- Show key metrics: duration, total predictions, winner (if completed)
- Filter by status and search by name
- Quick actions: start, pause, complete, view details
- Use existing Card, Badge, and Button components from shadcn/ui

**Experiment Detail Page** (`file:apps/frontend/app/experiments/[id]/page.tsx`):
- Show experiment configuration (variants, traffic split, metrics)
- Display real-time metrics for each variant using existing chart components
- Show statistical comparison between variants
- Visualize traffic distribution with pie chart
- Display timeline of experiment lifecycle events
- Action buttons: pause/resume, complete, export results

**Metrics Comparison Component** (`file:apps/frontend/components/ABTestMetricsComparison.tsx`):
- Side-by-side comparison of variant performance
- Use `LineChart` component for time-series metrics (latency, error rate over time)
- Use `BarChart` component for aggregate metrics comparison
- Display statistical significance indicators (p-value, confidence level)
- Show lift percentage with visual indicators (positive/negative)
- Highlight winning variant when statistically significant

**Variant Performance Cards** (`file:apps/frontend/components/VariantPerformanceCard.tsx`):
- Individual cards for each variant showing key metrics
- Total predictions, error rate, average latency
- Custom metrics visualization
- Traffic percentage allocation
- Status indicator (active, paused, completed)

#### Dashboard Integration

- Add "Experiments" navigation item to main dashboard (`file:apps/frontend/app/dashboard/page.tsx`)
- Create quick stats card showing active experiments count
- Add link to experiments page in deployment section
- Use existing `StatisticsDashboard` component patterns for layout consistency

### 3. Automatic Winner Selection

**Objective:** Implement background job to automatically detect and declare experiment winners.

#### Backend Implementation

**Create Experiment Monitor Service** (`file:apps/backend/app/services/experiment_monitor.py`):
- Implement periodic check for running experiments (every 5-10 minutes)
- Use `ABTestingService.check_experiment_completion()` to evaluate completion criteria
- Automatically call `ABTestingService.complete_experiment()` when criteria met
- Send notifications (email/webhook) when experiment completes
- Log all automatic actions for audit trail

**Add Scheduler Integration** (`file:apps/backend/app/scheduler.py`):
- Use APScheduler or similar library for background job scheduling
- Register experiment monitor job to run periodically
- Ensure single instance execution in distributed environments
- Add health check endpoint for scheduler status

**Enhance Completion Logic** (`file:apps/backend/app/services/ab_testing.py`):
- Extend `check_experiment_completion()` with configurable rules:
  - Minimum sample size per variant reached
  - Statistical significance threshold met
  - Maximum duration exceeded
  - Early stopping criteria (clear winner with high confidence)
- Add configuration for auto-completion behavior per experiment
- Support manual override to prevent auto-completion

#### Configuration

- Add experiment auto-completion settings to `file:apps/backend/app/config.py`
- Environment variables for scheduler interval, minimum confidence level
- Feature flag to enable/disable automatic winner selection

### 4. Gradual Rollout Support

**Objective:** Enable dynamic traffic percentage adjustment for progressive rollouts.

#### Backend Enhancements

**Add Rollout Schedule Model** (`file:apps/backend/app/models/ab_test.py`):
- Extend `ABTest` model with `rollout_schedule` field
- Define schedule as list of stages: `[{percentage: 10, duration_hours: 24}, {percentage: 50, duration_hours: 48}, ...]`
- Add `current_rollout_stage` field to track progress
- Add `auto_rollout_enabled` boolean flag

**Implement Rollout Manager** (`file:apps/backend/app/services/rollout_manager.py`):
- Service to manage gradual traffic increases
- Check rollout schedule and update traffic percentages automatically
- Validate performance thresholds before advancing to next stage
- Rollback capability if metrics degrade
- Integration with experiment monitor for automated progression

**Add Rollout API Endpoints** (`file:apps/backend/app/api/routes/ab_testing.py`):
- `POST /api/v1/ab-testing/experiments/{id}/rollout/configure` - Set rollout schedule
- `POST /api/v1/ab-testing/experiments/{id}/rollout/advance` - Manually advance to next stage
- `POST /api/v1/ab-testing/experiments/{id}/rollout/rollback` - Rollback to previous stage
- `GET /api/v1/ab-testing/experiments/{id}/rollout/status` - Get current rollout status

#### Frontend Components

**Rollout Configuration UI** (`file:apps/frontend/components/RolloutScheduleConfig.tsx`):
- Visual editor for defining rollout stages
- Timeline visualization of planned rollout
- Performance threshold configuration per stage
- Enable/disable auto-advancement toggle

**Rollout Progress Tracker** (`file:apps/frontend/components/RolloutProgressTracker.tsx`):
- Display current rollout stage with progress bar
- Show metrics for current stage vs. thresholds
- Manual controls for advance/rollback
- Timeline view of completed stages

### 5. Experiment Management Interface

**Objective:** Comprehensive UI for creating and managing experiments.

#### Create Experiment Wizard

**Experiment Creation Flow** (`file:apps/frontend/app/experiments/create/page.tsx`):
- Multi-step wizard for experiment setup:
  1. Basic info (name, description, primary metric)
  2. Model selection (choose 2+ models to test)
  3. Traffic configuration (split percentages, user targeting)
  4. Test settings (duration, sample size, confidence level)
  5. Rollout schedule (optional gradual rollout)
  6. Review and create
- Form validation using existing patterns
- Preview of experiment configuration before creation

**Experiment Settings Page** (`file:apps/frontend/app/experiments/[id]/settings/page.tsx`):
- Edit experiment configuration (when in draft status)
- Update traffic split for running experiments
- Configure secondary metrics
- Set completion criteria
- Manage rollout schedule

#### Management Features

**Bulk Actions Component** (`file:apps/frontend/components/ExperimentBulkActions.tsx`):
- Select multiple experiments for bulk operations
- Pause/resume multiple experiments
- Archive completed experiments
- Export multiple experiment results

**Experiment Templates** (`file:apps/frontend/components/ExperimentTemplates.tsx`):
- Pre-configured templates for common scenarios (50/50 split, champion/challenger, multi-variant)
- Save custom configurations as templates
- Quick start from template

### 6. Results Export and Reporting

**Objective:** Enable comprehensive export of experiment results for analysis and reporting.

#### Backend Export Service

**Create Export Service** (`file:apps/backend/app/services/ab_export_service.py`):
- Generate CSV export with variant-level metrics
- Generate JSON export with full experiment data
- Create PDF report with visualizations and statistical analysis
- Support custom date ranges and metric selection
- Include statistical test results and confidence intervals

**Add Export Endpoints** (`file:apps/backend/app/api/routes/ab_testing.py`):
- `GET /api/v1/ab-testing/experiments/{id}/export?format=csv|json|pdf` - Export single experiment
- `POST /api/v1/ab-testing/export/batch` - Export multiple experiments
- `GET /api/v1/ab-testing/experiments/{id}/report` - Generate comprehensive report
- Support streaming for large exports

#### Frontend Export Features

**Export Dialog Component** (`file:apps/frontend/components/ExperimentExportDialog.tsx`):
- Select export format (CSV, JSON, PDF)
- Choose metrics to include
- Date range selection
- Include/exclude raw prediction data
- Download or email options

**Report Generation** (`file:apps/frontend/components/ExperimentReportGenerator.tsx`):
- Generate shareable HTML report
- Include charts, tables, and statistical analysis
- Executive summary section
- Recommendations based on results
- Export to PDF or share via link

### 7. Integration Testing and Validation

**Objective:** Ensure all components work together seamlessly.

#### Testing Strategy

**Backend Tests** (`file:apps/backend/tests/test_api/test_ab_testing_integration.py`):
- Test traffic router with active experiments
- Validate variant assignment consistency
- Test automatic winner selection logic
- Verify gradual rollout progression
- Test export functionality with various formats

**Frontend Tests** (`file:apps/frontend/__tests__/experiments/`):
- Test experiment creation wizard flow
- Validate dashboard metrics display
- Test rollout configuration UI
- Verify export functionality

**End-to-End Tests**:
- Create experiment → Start → Make predictions → View metrics → Complete → Export
- Test gradual rollout from 10% to 100%
- Verify automatic winner selection triggers correctly

### 8. Documentation and Deployment

**Objective:** Document the A/B testing framework and prepare for production deployment.

#### Documentation

**User Guide** (`file:docs/ab-testing/USER_GUIDE.md`):
- How to create and configure experiments
- Understanding metrics and statistical significance
- Best practices for A/B testing
- Interpreting results and making decisions

**API Documentation** (`file:docs/ab-testing/API.md`):
- Complete API reference for all A/B testing endpoints
- Code examples for integration
- Webhook configuration for notifications

**Developer Guide** (`file:docs/ab-testing/DEVELOPER_GUIDE.md`):
- Architecture overview
- Traffic routing implementation details
- Extending with custom metrics
- Troubleshooting guide

#### Deployment Considerations

- Add A/B testing configuration to environment variables
- Update deployment scripts to include scheduler service
- Configure monitoring and alerting for experiment health
- Set up database indexes for experiment queries
- Plan for data retention and archival of completed experiments

## Architecture Diagram

```mermaid
sequenceDiagram
    participant User
    participant TrafficRouter
    participant ABTestService
    participant ModelA
    participant ModelB
    participant MetricsCollector
    participant ExperimentMonitor

    User->>TrafficRouter: POST /predict (model_id)
    TrafficRouter->>ABTestService: Check active experiments
    ABTestService->>ABTestService: Assign variant (hash user_id)
    ABTestService-->>TrafficRouter: Variant B assigned
    TrafficRouter->>ModelB: Forward prediction request
    ModelB-->>TrafficRouter: Prediction result
    TrafficRouter->>MetricsCollector: Track metrics (latency, success)
    TrafficRouter-->>User: Return prediction + variant info
    
    Note over ExperimentMonitor: Background Job (every 5 min)
    ExperimentMonitor->>ABTestService: Check completion criteria
    ABTestService->>ABTestService: Calculate statistics
    alt Criteria Met
        ABTestService->>ABTestService: Declare winner
        ABTestService->>User: Send notification
    end
```

## Component Dependencies

| Component | Dependencies | Purpose |
|-----------|-------------|---------|
| Traffic Router Middleware | ABTestingService, ModelService | Route predictions to correct variant |
| Experiment Monitor | ABTestingService, Scheduler | Automatic winner selection |
| Rollout Manager | ABTest model, ABTestingService | Gradual traffic increase |
| Export Service | ABTest model, Reporting libraries | Generate reports |
| Dashboard UI | abTesting.ts service, Chart components | Visualize experiment results |
| Experiment Wizard | abTesting.ts service, Form components | Create experiments |

## Key Files to Modify/Create

**Backend:**
- Create: `file:apps/backend/app/middleware/ab_traffic_router.py`
- Create: `file:apps/backend/app/services/experiment_monitor.py`
- Create: `file:apps/backend/app/services/rollout_manager.py`
- Create: `file:apps/backend/app/services/ab_export_service.py`
- Create: `file:apps/backend/app/scheduler.py`
- Modify: `file:apps/backend/app/api/routes/ab_testing.py` (add new endpoints)
- Modify: `file:apps/backend/app/api/routes/production.py` (integrate traffic router)
- Modify: `file:apps/backend/app/models/ab_test.py` (add rollout fields)
- Modify: `file:apps/backend/app/main.py` (register middleware and scheduler)

**Frontend:**
- Create: `file:apps/frontend/app/experiments/page.tsx`
- Create: `file:apps/frontend/app/experiments/[id]/page.tsx`
- Create: `file:apps/frontend/app/experiments/create/page.tsx`
- Create: `file:apps/frontend/components/ABTestMetricsComparison.tsx`
- Create: `file:apps/frontend/components/VariantPerformanceCard.tsx`
- Create: `file:apps/frontend/components/RolloutScheduleConfig.tsx`
- Create: `file:apps/frontend/components/RolloutProgressTracker.tsx`
- Create: `file:apps/frontend/components/ExperimentExportDialog.tsx`
- Modify: `file:apps/frontend/app/dashboard/page.tsx` (add experiments section)

**Tests:**
- Create: `file:apps/backend/tests/test_middleware/test_ab_traffic_router.py`
- Create: `file:apps/backend/tests/test_services/test_experiment_monitor.py`
- Create: `file:apps/backend/tests/test_services/test_rollout_manager.py`
- Create: `file:apps/frontend/__tests__/experiments/experiment-wizard.test.tsx`

**Documentation:**
- Create: `file:docs/ab-testing/USER_GUIDE.md`
- Create: `file:docs/ab-testing/API.md`
- Create: `file:docs/ab-testing/DEVELOPER_GUIDE.md`

---

Execution Information

Branch: main
Commit: 1963a18