diff --git a/README.md b/README.md index 49f1dc8..b86f36c 100644 --- a/README.md +++ b/README.md @@ -214,7 +214,7 @@ Once configured, the following endpoints are available: - `/first-party/ad` (GET): returns HTML for a single slot (`slot`, `w`, `h` query params). The server inspects returned creative HTML and rewrites: - All absolute images and iframes to `/first-party/proxy?tsurl=&&tstoken=` (1×1 pixels are detected server‑side heuristically for logging). The `tstoken` is derived from encrypting the full target URL and hashing it. -- `/third-party/ad` (POST): accepts tsjs ad units and proxies to Prebid Server. +- `/auction` (POST): accepts tsjs ad units and runs the auction orchestrator. - `/first-party/proxy` (GET): unified proxy for resources referenced by creatives. - Query params: - `tsurl`: Target URL without query (base URL) — required diff --git a/SEQUENCE.md b/SEQUENCE.md index 235ef0e..210bb45 100644 --- a/SEQUENCE.md +++ b/SEQUENCE.md @@ -130,7 +130,7 @@ sequenceDiagram ## Notes - TSJS - Served first-party at `/static/tsjs-core.min.js` (and `/static/tsjs-ext.min.js` if prebid auto-config is enabled). - - Discovers ad units and renders placeholders; either uses slot-level HTML (`/first-party/ad`) or JSON auction (`/third-party/ad`). + - Discovers ad units and renders placeholders; either uses slot-level HTML (`/first-party/ad`) or JSON auction (`/auction`). - Publisher HTML Rewriting - Injects TSJS loader and rewrites absolute URLs from origin domain to first-party domain during streaming. - Creative HTML Rewriting diff --git a/TESTING.md b/TESTING.md new file mode 100644 index 0000000..b245acd --- /dev/null +++ b/TESTING.md @@ -0,0 +1,184 @@ +# Testing the Auction Orchestration System + +## Quick Test Summary + +The auction orchestration system has been integrated into the existing Prebid endpoints. You can test it right away using the Fastly local server! + +## How to Test + +### 1. Start the Local Server + +```bash +fastly compute serve +``` + +### 2. Test with Existing Endpoint + +The `/auction` endpoint now uses the orchestrator when `auction.enabled = true` in config. + +**Test Request:** +```bash +curl -X POST http://localhost:7676/auction \ + -H "Content-Type: application/json" \ + -d '{ + "adUnits": [ + { + "code": "header-banner", + "mediaTypes": { + "banner": { + "sizes": [[728, 90], [970, 250]] + } + } + }, + { + "code": "sidebar", + "mediaTypes": { + "banner": { + "sizes": [[300, 250], [300, 600]] + } + } + } + ] + }' +``` + +### 3. What You'll See + +**With Orchestrator Enabled** (`auction.enabled = true`): +- Logs showing: `"Using auction orchestrator"` +- Parallel execution of APS (mocked) and Prebid (real) +- GAM mediation (mocked) selecting winning bids +- Final response with winning creatives + +**With Orchestrator Disabled** (`auction.enabled = false`): +- Logs showing: `"Using legacy Prebid flow"` +- Direct Prebid Server call (backward compatible) + +##Configuration + +Edit `trusted-server.toml` to customize the auction: + +```toml +# Enable/disable orchestrator +[auction] +enabled = true +bidders = ["prebid", "aps"] +mediator = "gam" # If set: mediation, if omitted: highest bid wins +timeout_ms = 2000 + +# Mock provider configs +[integrations.aps] +enabled = true +mock = true +mock_price = 2.50 + +[integrations.gam] +enabled = true +mock = true +inject_house_bids = true +gam_win_rate = 30 # GAM wins 30% of the time +``` + +## Test Scenarios + +### Scenario 1: Parallel + Mediation (Default) +**Config:** +```toml +[auction] +enabled = true +bidders = ["prebid", "aps"] +mediator = "gam" # Mediator configured = parallel mediation strategy +``` + +**Expected Flow:** +1. Prebid queries real SSPs +2. APS returns mock bids ($2.50 CPM) +3. GAM mediates between all bids +4. Winning creative returned + +### Scenario 2: Parallel Only (No Mediation) +**Config:** +```toml +[auction] +enabled = true +bidders = ["prebid", "aps"] +# No mediator = parallel only strategy +``` + +**Expected Flow:** +1. Prebid and APS run in parallel +2. Highest bid wins automatically +3. No GAM mediation + +### Scenario 3: Legacy Mode (Backward Compatible) +**Config:** +```toml +[auction] +enabled = false +``` + +**Expected Flow:** +- Original Prebid-only behavior +- No orchestration overhead + +## Debugging + +### Check Logs +The orchestrator logs extensively: +``` +INFO: Using auction orchestrator +INFO: Running auction with strategy: parallel_mediation +INFO: Running 2 bidders in parallel +INFO: Requesting bids from: prebid +INFO: Prebid returned 2 bids (time: 120ms) +INFO: Requesting bids from: aps +INFO: APS (MOCK): returning 2 bids in 80ms +INFO: GAM mediation: slot 'header-banner' won by 'amazon-aps' at $2.50 CPM +``` + +### Verify Provider Registration +Look for these log messages on startup: +``` +INFO: Registering auction provider: prebid +INFO: Registering auction provider: aps +INFO: Registering auction provider: gam +``` + +### Common Issues + +**Issue:** `"Provider 'aps' not registered"` +**Fix:** Make sure `[integrations.aps]` is configured in `trusted-server.toml` + +**Issue:** `"No bidders configured"` +**Fix:** Make sure `bidders = ["prebid", "aps"]` is set in `[auction]` + +**Issue:** Tests fail with WASM errors +**Explanation:** Async tests don't work in WASM test environment. Integration tests via HTTP work fine! + +## Next Steps + +1. **Test with real Prebid Server** - Verify Prebid bids work correctly +2. **Implement real APS** - Replace mock with actual Amazon TAM API calls +3. **Implement real GAM** - Add Google Ad Manager API integration +4. **Add metrics** - Track bid rates, win rates, latency per provider + +## Mock Provider Behavior + +### APS (Amazon) +- Returns bids for all slots +- Default mock price: $2.50 CPM +- Always returns 2 bids +- Response time: ~80ms (simulated) + +### GAM (Google Ad Manager) +- Acts as mediator +- Can inject house ads at $1.75 CPM +- Wins 30% of auctions (configurable) +- Response time: ~40ms (simulated) +- Uses hash-based "randomness" for consistent testing + +### Prebid +- **Real implementation** - makes actual HTTP calls +- Queries configured SSPs +- Returns real bids from real bidders +- Response time: varies (network dependent) diff --git a/crates/common/build.rs b/crates/common/build.rs index d7b020b..a97734b 100644 --- a/crates/common/build.rs +++ b/crates/common/build.rs @@ -1,6 +1,9 @@ #[path = "src/error.rs"] mod error; +#[path = "src/auction_config_types.rs"] +mod auction_config_types; + #[path = "src/settings.rs"] mod settings; diff --git a/crates/common/src/auction/README.md b/crates/common/src/auction/README.md new file mode 100644 index 0000000..3659ffd --- /dev/null +++ b/crates/common/src/auction/README.md @@ -0,0 +1,559 @@ +# Auction Orchestration System + +A flexible, extensible framework for managing multi-provider header bidding auctions with support for parallel execution and mediation. + +## Overview + +The auction orchestration system allows you to: +- Run multiple auction providers (Prebid, Amazon APS, Google GAM, etc.) in parallel or sequentially +- Implement mediation strategies where a primary ad server (like GAM) makes the final decision +- Configure different auction flows for different scenarios +- Easily add new auction providers + +## Architecture + +``` +┌─────────────────────────────────────────────────────────┐ +│ Auction Orchestrator │ +│ - Manages auction workflow & sequencing │ +│ - Combines bids from multiple sources │ +│ - Applies business logic │ +└─────────────────────────────────────────────────────────┘ + │ + │ uses + ▼ +┌─────────────────────────────────────────────────────────┐ +│ AuctionProvider Trait │ +│ - request_bids() │ +│ - provider_name() │ +│ - timeout_ms() │ +│ - is_enabled() │ +└─────────────────────────────────────────────────────────┘ + │ + ┌─────────────────┼─────────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ Prebid │ │ Amazon │ │ Google │ + │ Provider │ │ APS │ │ GAM │ + └──────────┘ └──────────┘ └──────────┘ +``` + +## Request Flow + +When a request arrives at the `/auction` endpoint, it goes through the following steps: + +``` +┌──────────────────────────────────────────────────────────────────────┐ +│ 1. HTTP POST /auction │ +│ - Body: AdRequest (Prebid.js/tsjs format) │ +│ - Headers: User-Agent, cookies, etc. │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 2. Route Matching (crates/fastly/src/main.rs:84) │ +│ - Pattern: (Method::POST, "/auction") │ +│ - Handler: handle_auction(&settings, req).await │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 3. Parse Request Body (mod.rs:149) │ +│ - Deserialize JSON → AdRequest struct │ +│ - Extract ad units with media types │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 4. Generate User IDs (mod.rs:206-214) │ +│ - Create/retrieve synthetic ID (persistent) │ +│ - Generate fresh ID (per-request) │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 5. Transform Request Format (mod.rs:216-240) │ +│ - AdRequest → AuctionRequest │ +│ - AdUnit.code → AdSlot.id │ +│ - mediaTypes.banner.sizes → AdFormat[] │ +│ - Build PublisherInfo, UserInfo, DeviceInfo │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 6. Get Global Orchestrator (mod.rs:161) │ +│ - Retrieve from GLOBAL_ORCHESTRATOR singleton │ +│ - Contains all registered providers (APS, Prebid, etc.) │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 7. Create Auction Context (mod.rs:172-176) │ +│ - Attach settings │ +│ - Attach original request │ +│ - Set timeout from config │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 8. Run Auction Strategy (orchestrator.rs:42) │ +│ ┌────────────────────────────────────────────────────────────┐ │ +│ │ Strategy: parallel_only │ │ +│ │ 1. Launch all bidders concurrently │ │ +│ │ 2. Wait for all responses │ │ +│ │ 3. Select highest bid per slot │ │ +│ └────────────────────────────────────────────────────────────┘ │ +│ ┌────────────────────────────────────────────────────────────┐ │ +│ │ Strategy: parallel_mediation │ │ +│ │ 1. Launch all bidders concurrently │ │ +│ │ 2. Collect all bids │ │ +│ │ 3. Send to mediator (GAM) for final decision │ │ +│ └────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 9. Each Provider Processes Request │ +│ - Transform AuctionRequest → Provider format (e.g., APS TAM) │ +│ - Send HTTP request to provider endpoint │ +│ - Parse provider response │ +│ - Transform → AuctionResponse with Bid[] │ +│ - Return to orchestrator │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 10. Select Winning Bids (orchestrator.rs:363-385) │ +│ - For each slot, find highest CPM bid │ +│ - Create HashMap │ +│ - Log winning selections │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 11. Transform to OpenRTB Response (mod.rs:274-322) │ +│ - Build seatbid array (one per winning bid) │ +│ - Rewrite creative HTML for first-party proxy │ +│ - Add orchestrator metadata (timing, strategy, bid count) │ +└──────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────┐ +│ 12. Return HTTP Response │ +│ - Status: 200 OK │ +│ - Content-Type: application/json │ +│ - Body: OpenRTB BidResponse │ +└──────────────────────────────────────────────────────────────────────┘ +``` + +### Step-by-Step Breakdown + +#### 1. Request Arrival +Client (browser, Prebid.js, tsjs) sends a POST request to `/auction` with ad unit definitions: + +```json +{ + "adUnits": [ + { + "code": "header-banner", + "mediaTypes": { + "banner": { + "sizes": [[728, 90], [970, 250]] + } + } + } + ] +} +``` + +#### 2. Format Transformation +The system transforms the Prebid.js format into an internal `AuctionRequest`: + +```rust +// From: AdUnit with sizes [[728, 90], [970, 250]] +// To: AdSlot with formats +AdSlot { + id: "header-banner", + formats: vec![ + AdFormat { width: 728, height: 90, media_type: Banner }, + AdFormat { width: 970, height: 250, media_type: Banner }, + ], + floor_price: None, + targeting: HashMap::new(), +} +``` + +#### 3. Provider Execution +Each registered provider (APS, Prebid, etc.) receives the `AuctionRequest` and: +- Transforms it to their specific format (e.g., APS TAM, OpenRTB) +- Makes HTTP request to their endpoint +- Parses the response +- Returns `AuctionResponse` with `Bid[]` + +For example, APS provider: +```rust +// Transform AuctionRequest → ApsBidRequest +let aps_request = ApsBidRequest { + pub_id: "5128", + slots: vec![ + ApsSlot { + slot_id: "header-banner", + sizes: vec![[728, 90], [970, 250]], + slot_name: Some("header-banner"), + } + ], + page_url: Some("https://example.com"), + ua: Some("Mozilla/5.0..."), + timeout: Some(800), +}; + +// HTTP POST to http://localhost:6767/e/dtb/bid +// Parse response → AuctionResponse +``` + +#### 4. Response Assembly +The orchestrator collects all bids and creates an OpenRTB response: + +```json +{ + "id": "auction-response", + "seatbid": [ + { + "seat": "amazon-aps", + "bid": [ + { + "id": "amazon-aps-header-banner", + "impid": "header-banner", + "price": 2.5, + "adm": "