SPEAKER.md

Speaker Guide: Logging and Analysis Demo

Audience promise

"In 20 minutes, you will see exactly why logs-only troubleshooting is painful, and how metrics + logs + traces together make incident triage dramatically faster."

Talk structure (suggested 20-25 min)

1. Setup and framing (2 min)
2. Simple endpoint demo (4 min)
3. Complex checkout demo (6 min)
4. Failure investigation story (8 min)
5. Wrap-up and Q&A (3-5 min)

Pre-talk checklist

1. Start stack:

docker compose up --build

2. Open:

   • API: http://localhost:8080
   • Grafana: http://localhost:3000

3. In Grafana:

   • Confirm dashboard Observability Demo Overview is visible.

• Open Explore and verify datasource health with these checks:
  • Loki datasource selected, query: {app="demo-api"}, time range: Last 15 minutes.
  • Tempo datasource selected, search service name logging-analysis-demo-api, time range: Last 15 minutes.
  • If both return No data, generate one request with Invoke-RestMethod "http://localhost:8080/demo/simple/ping?delayMs=20" and retry.
  • "No data" is healthy only when datasource test passes and no traffic has been sent yet.

4. Keep a terminal ready for API calls.

Step-by-step live flow

1) Frame the problem

Talking points:

• "Most teams start with logs, but logs alone break down under concurrency and distributed workflows."
• "We need three lenses: metrics for shape, logs for detail, traces for causality."
• "This app has both a simple and a complex scenario so we can compare the experience."

2) Show the simple path first

Run:

Invoke-RestMethod "http://localhost:8080/demo/simple/ping?delayMs=20"
Invoke-RestMethod "http://localhost:8080/demo/simple/ping?delayMs=200"

Narrate:

• "Same endpoint, different latency."
• "Prometheus captures timing distributions."
• "Loki captures contextual logs for each request."

In Grafana dashboard, point at:

• Request Rate
• Checkout Latency panel (it is expected to show No data until /demo/checkout or /demo/load/spike is called; use this to explain metric scope)

3) Trigger intentional errors

Run:

Invoke-RestMethod "http://localhost:8080/demo/simple/error" -SkipHttpErrorCheck

Narrate:

• "This is a deliberate failure to simulate a user-visible issue."
• "Errors panel rises quickly - metrics tell us there is a problem, not why."

In Grafana:

• Show error panel movement
• Open logs panel and point out warning/error entries

4) Move to complex behavior (why traces matter)

Run one checkout:

$body = @{
  orderId = "ord-talk-001"
  tenant = "retail"
  totalAmount = 650.00
  items = @(
    @{ sku = "SKU-500"; quantity = 2; unitPrice = 125.00 },
    @{ sku = "SKU-900"; quantity = 1; unitPrice = 400.00 }
  )
} | ConvertTo-Json -Depth 5

Invoke-RestMethod "http://localhost:8080/demo/checkout" -Method Post -Body $body -ContentType "application/json"

Narrate:

• "Checkout is multi-step: inventory, payment, shipment."
• "A single result code hides where time was actually spent."

In Grafana Explore:

1. Set datasource to Loki and time range to Last 15 minutes.
2. Run log query: {app="demo-api"} |= "checkout"
3. If there are too many rows, narrow to your test order id:

• {app="demo-api"} |= "ord-talk-001"

4. Open one matching row and point out:

• Message text like "Checkout started" or "Checkout completed"
• Context fields: orderId, tenant, and itemCount
• Trace fields: TraceId and SpanId

5. Pivot to trace:

• Click the TraceId link from the log row (or copy TraceId into Tempo search).

6. Switch datasource to Tempo and show the trace details:

• Root span: checkout.process
• Child spans: checkout.reserve_inventory, checkout.capture_payment, checkout.create_shipment
• Use span duration to identify the slow or failing step.

7. Narrate the correlation story:

• "Metric spike tells us something changed."
• "Logs tell us which order/tenant was impacted."
• "Trace shows exactly which step consumed time or failed."

If you still see no rows in Loki after a checkout call:

1. Confirm time range is Last 15 minutes and not a custom stale range.
2. Run broader query: {app="demo-api"}.
3. Trigger a fresh checkout call and rerun query after 5-10 seconds.
4. Validate API is receiving requests: docker compose logs --tail=50 demo-api.

Key line:

• "Metrics told us something changed, logs gave context, traces told us exactly where and in what order."

5) Generate load and tell incident story

First, force one failure mode so the incident is deterministic:

$faults = @{
  forceInventoryFailure = $false
  forcePaymentFailure = $true
} | ConvertTo-Json

Invoke-RestMethod "http://localhost:8080/demo/admin/faults" -Method Post -Body $faults -ContentType "application/json"

Then run load:

Invoke-RestMethod "http://localhost:8080/demo/load/spike?seconds=20&concurrency=12" -Method Post

Narrate while charts move in the dashboard:

• "This simulates a traffic spike."
• "Watch request rate and p95 latency rise together."
• "Because payment failure is forced, failures are now deterministic and easy to investigate live."

In Grafana, demonstrate investigation path:

1. Dashboard symptom:

• Errors Last 5m increases.
• Checkout Outcomes shows failure rate climbing.

2. Logs in Explore (Loki datasource):

• Query: {app="demo-api"} |= "CAUSE checkout failure"
• Highlight SourceHint field value, for example CheckoutEngine.CapturePaymentAsync.

3. Traces in Explore (Tempo datasource):

• Pivot from TraceId in the same log row.
• Show failed trace and span where checkout.capture_payment is the failing step.

4. Exact source lines in code:

Select-String -Path ".\src\Demo.Api\Program.cs" -Pattern "CAUSE checkout failure at capture_payment|CapturePaymentAsync" |
  ForEach-Object { "{0}:{1}: {2}" -f $_.Path, $_.LineNumber, $_.Line.Trim() }

5. Explain remediation:

• "Root cause is isolated to payment step logic."
• "We can now fix this method directly instead of guessing from generic failures."

After the demo, reset fault toggles:

$reset = @{
  forceInventoryFailure = $false
  forcePaymentFailure = $false
} | ConvertTo-Json

Invoke-RestMethod "http://localhost:8080/demo/admin/faults" -Method Post -Body $reset -ContentType "application/json"

Suggested incident narrative:

1. "Pager fires: checkout failures increased."
2. "Dashboards confirm symptom and blast radius."
3. "Logs show impacted tenant/order patterns."
4. "Logs include SourceHint to the failing method: CheckoutEngine.CapturePaymentAsync."
5. "Traces isolate the failure path to checkout.capture_payment span."
6. "Mitigation: rollback fault flag or deploy fix, then verify metrics recovery."

6) Close with practical takeaway

Talking points:

• "Use metrics for fast detection and SLOs."
• "Use logs for rich event context."
• "Use traces for path-level root cause analysis."
• "The value is not any one tool; it is correlation across all three."

Optional deeper demo extensions

• Add tenant-specific alerting and compare behavior by tenant label.
• Add a downstream fake dependency API and propagate trace context over HTTP.
• Add exemplars and link high-latency metric points directly to traces.

If something fails live

1. Validate containers:

docker compose ps

2. Tail API logs:

docker compose logs -f demo-api

3. Restart stack quickly:

docker compose down
docker compose up --build