Skip to content

fix(historical): resolve current-day lookups at now, not future end-of-day - #49

Merged
matheus1lva merged 4 commits into
mainfrom
matheus1lva/clamp-current-day-live-resolve
Aug 27, 2026
Merged

fix(historical): resolve current-day lookups at now, not future end-of-day#49
matheus1lva merged 4 commits into
mainfrom
matheus1lva/clamp-current-day-live-resolve

Conversation

@matheus1lva

@matheus1lva matheus1lva commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Problem

Since the 08/25 deploy, Kong's current-day price lookups fail ~50% of the time (incident report: https://artifacts.yearn.dev/94e791faa5bb87e377c76cb2cc7a4cfe.md).

parseTimestampSegment normalizes every requested timestamp to end-of-day 23:59:59. That key is right for the DB read (today's rows are stored at EOD), but the exact route passed the same normalized value to the live-resolve fallback. For a block a few minutes old that timestamp is up to ~24h in the future: DeFiLlama's 6h search width around a future timestamp finds nothing until late in the UTC day, and the on-chain source maps a future timestamp to a block. Result: 404s and edge timeouts on nearly every current-day miss, worst right after UTC midnight (87% failure in the 00:00 hour on Aug 27).

Fix

Clamp the resolve timestamp with the existing toFetchTimestamp helper (already used by the batch warmup source): current-day requests resolve at now, past days are unchanged. The DB lookup key, response timestamp field, and cache-control behavior stay as they were.

No service-side backfill needed

  • The broken path never wrote to the DB — it only returned 404s/504s, so no bad rows exist.
  • Warmup applies its own today→now clamp (scripts/warmup-prices.ts) and kept writing correct rows throughout.
  • Mis-dated rows from the earlier alignment bug were already repaired by PR feat(prices): fix DeFiLlama day alignment and backfill mis-dated prices #45's backfill.
  • False 404s edge-cached during the incident expire within 1h on their own.

After merge

  1. Deploy the worker.
  2. On Kong (separate repo/infra):
    • Decide on USE_PRICE_SERVICE (Render env). This fix removes the current-day regression (~38% of the zeros), but ~55% are tokens the service has never carried — those still return 404 and become priceUsd: 0 in Kong until the coverage gap is closed or the flag is rolled back.
    • Repair the zeroed TVL rows (46 mainnet vaults, ~$1.3M, Aug 27 snapshot) with kong's packages/scripts/src/quality-assurance/ pair: tvl-detect-gaps.ts --start 2026-08-25 to list affected vaults, then tvl-backfill.ts --update tvls to rewrite them.
  3. Before any re-cutover to service-only mode: close the coverage gap — the ~9,375 warmup gap: tokens per run are the worklist; the historical gap backfill (feat(backfill): add historical price gap backfill #40) is the tool. Gate cutover on a measured coverage comparison against the legacy chain.

Scope

This PR is only the current-day live-resolve fix. Coverage-gap work, Kong-side caching, and alerting on zero-price rates are follow-ups.

@matheus1lva

Copy link
Copy Markdown
Collaborator Author

APPROVE

Fix matches the incident's root cause: only the live-resolve argument is clamped with the existing
toFetchTimestamp; DB key, response timestamp, and cache behavior untouched. Test pins the
resolve-at-now behavior with fake timers. No defects found.


How This Was Reviewed

Reviewed with the review-pr-workflow skill
2 review lenses, each finding independently verified by claude. 0 candidate findings were refuted and dropped.

@matheus1lva

Copy link
Copy Markdown
Collaborator Author

/review-workflow

@github-actions

Copy link
Copy Markdown

Review started (review-pr-workflow): https://github.com/yearn/yearn-prices/actions/runs/33075013630

@github-actions

Copy link
Copy Markdown

Review: PR #49 — fix(historical): resolve current-day lookups at now, not future end-of-day

Verdict: APPROVE

toFetchTimestamp was already the established helper for this exact problem (src/sources/defillama/batch.ts:31), so the exact route now matches the batch path instead of asking upstream for a timestamp up to 24h in the future.

Issues

None. No defect survived verification.

gitconfig-mask: sentinel


How This Was Reviewed

Reviewed with the review-pr-workflow skill
5 review lenses, each finding independently verified by claude. 0 candidate findings were refuted and dropped.


tier: full — 2 files / ~30 net lines would downgrade to light, escalated because the change alters timestamp semantics feeding both the upstream fetch and cache-control selection; verify-agent: claude.
stats — confirmed 0, refuted 0, unverified 0, discarded 1 (a non-defect clarity suggestion), advisories 0; no duplicates to collapse. Four of five lenses returned nothing, so no finding actually reached a verifier.
Not verifiable in CI: bun run lint (biome is not installed and there is no network); visual verification skipped — this is a Workers JSON API with no UI.
Coverage gaps from the critic (not part of the review, for your call on whether to re-run a lens):

  • On-chain source shares the fallback registry and now also receives now: src/sources/onchain/context.ts:155 rejects blockTimestamp > target.timestamp, a guard that previously had ~24h of slack; and the block cache key src/clients/rpc.ts:152 goes from one key per UTC day to one per second. Unexamined by every lens.
  • docs/routes.md:48-54,132 still documents current-day fallback as a day-granularity lookup; no lens opened the docs directory.
  • The new test asserts only the outbound URL — not the echoed timestamp nor the CACHE_CONTROL_TODAY header, and it is the only test reaching that branch.
  • No test pins the untouched branch (non-today timestamps passed through unmodified); there is no test file for src/utils/time.ts.
  • vi.useFakeTimers() is restored inline at test/prices-historical.test.ts:189 rather than in afterEach, unlike every other fake-timer suite in the repo; a rejection would leak faked time into the retry test at line 214.
  • Commit 1b608eb exists solely for lint/format and contributes one reformatted line, but biome could not be run to confirm the branch passes the gate.

@murderteeth murderteeth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: PR #49 — fix(historical): resolve current-day lookups at now, not future end-of-day

Verdict: APPROVE

The clamp itself is right and matches the batch path's existing use of toFetchTimestamp. Lint is green as of faaa513; all four CI jobs pass.

Issues

  • Current-day on-chain fallback got slower (medium) — resolving at now means the chain head has usually moved past that timestamp by the time the on-chain source runs, so the cheap "target is in the future, return head" shortcut in estimateBlockByTimestamp no longer fires. In a cold isolate the block lookup becomes a sleeping binary search (~26 sequential getBlock calls instead of 1), on exactly the current-day requests this PR is repairing — worst on fast-block L2s. It amortizes to ~3 calls once the isolate has samples, so this is added latency, not failure. Done when: with a stubbed client whose head timestamp is one second newer than the resolve timestamp and no remembered samples, a current-day on-chain lookup resolves in single-digit getBlock calls, and a test pins that count. Provenance: d63d0ba

How This Was Reviewed

Reviewed with the review-pr-workflow skill
5 review lenses, each finding independently verified by claude. 0 candidate findings were refuted and dropped.

@matheus1lva
matheus1lva merged commit d05be36 into main Aug 27, 2026
4 checks passed
@matheus1lva
matheus1lva deleted the matheus1lva/clamp-current-day-live-resolve branch August 27, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants