refactor(historical): drop on-miss upstream resolution, serve from table only (#54)

Author: matheus1lva

README.md

@@ -1,6 +1,6 @@
 # price-service
 
-Cloudflare Worker that serves spot and historical token prices for Yearn. It aggregates prices from DefiLlama, on-chain oracles, Curve, Bob's API, and Enso, and persists historical prices to a Neon Postgres database.
+Cloudflare Worker that serves Enso spot prices and historical prices from Neon Postgres. Historical rows are written by offline warmup and backfill jobs, not by the worker.
 
 ## Requirements
 
@@ -49,16 +49,14 @@ Full route reference, request/response shapes, error codes, and caching behavior
 
 ## Price sources
 
-Prices are fetched through a pluggable source layer that tries providers in priority order until one returns a result. Currently:
-
 - **Spot prices**: Enso (live prices for any token on supported chains)
-- **Historical prices**: DefiLlama → Chainlink → derived (on-chain) → DefiLlama alias, tried in that order on single-token lookups when the DB has no record
+- **Historical prices**: read from `token_prices` only. No historical route calls an upstream provider; a row that is not in the table returns `404` (single token) or is omitted from the response (batch, range), and stays that way until an offline job writes it.
 
-Batch and range historical endpoints remain DB-only (a registry fallback inside a large batch would generate many upstream requests).
+Historical rows are written by `scripts/warmup-prices.ts` (hourly: DefiLlama, Curve, derived), `scripts/backfill-historical-gaps.ts` and `scripts/backfill-defillama-day-alignment.ts`. One-off migrations (`scripts/backfill-token-address-checksums.ts`) copy existing rows and add no new prices. `docs/routes.md` lists which job writes each `source` value.
 
 ### Adding a new price source
 
-Sources are pluggable adapters under `src/sources/`, registered in `src/registries/spot.ts` or `src/registries/historical.ts`. See [`src/sources/README.md`](src/sources/README.md) for the full authoring guide (interface, registration steps, test checklist).
+Sources are pluggable adapters under `src/sources/`. Spot sources register in `src/registries/spot.ts` and are served live. Historical sources register in `src/registries/historical.ts` but have no request-path consumer — extend a warmup or backfill job so rows reach the table. See [`src/sources/README.md`](src/sources/README.md).
 
 ## Authentication
 

docs/routes.md

@@ -66,7 +66,15 @@ Price routes accept an optional `source` query parameter. Supported values are:
 - `defillama-alias`
 - `enso`
 
-`source` filters the stored price rows only. `chainlink` and `defillama-alias` are resolved live. When request-path persistence fills a gap for a closed past day, the result is stored under the resolving source's name, so filters match those rows afterward. Before any such fill, filtering returns `404` for the single-token route and `200` with an empty `coins` object for `batchHistorical` and `rangeHistorical`.
+`source` filters the stored price rows only. Rows reach the table from the offline jobs:
+
+- `defillama`: `scripts/warmup-prices.ts`, `scripts/backfill-historical-gaps.ts` (alias hits are stored as `defillama`) and `scripts/backfill-defillama-day-alignment.ts` (re-aligns existing rows).
+- `curve`, `derived`: `scripts/warmup-prices.ts`.
+- `chainlink`, `defillama-alias`: no writer. Only rows stored before the request-path registry was removed can match; a token priceable only through Chainlink has no price on any route.
+- `on-chain-oracle`, `bobs-api`: no writer in this repo.
+- `enso`: spot only; the spot route never writes `token_prices`, so no historical row carries this source.
+
+Filtering on a source with no stored rows returns `404` for the single-token route and `200` with an empty `coins` object for `batchHistorical` and `rangeHistorical`.
 
 When `source` is omitted, the API returns the first available row by priority:
 
@@ -135,13 +143,9 @@ Response:
 }
 ```
 
-If no stored row exists for the normalized timestamp and no `source` filter was given, the route falls back to the live historical source registry for that day. A fallback resolved for a closed past day is stored in `token_prices` under the normalized day key, so the next request is a table hit; current-day and future-day results are never persisted. A non-chainlink resolution is stored only when its underlying observation falls within the DeFiLlama search width of the day key; an out-of-window result is served but not stored, so it resolves upstream again on the next cache miss. A `source` filter disables the fallback: the route answers from stored rows only.
-
-The fallback shares the request-path DeFiLlama client with `batchHistorical`: each upstream attempt is capped at 2.5 seconds and DeFiLlama `429` responses are not retried, so a rate-limited lookup fails on the first attempt instead of backing off. `5xx` responses still retry with backoff, so one degraded call can take up to three attempts. Offline jobs keep the retrying client.
+If no stored row exists for the normalized timestamp, the route returns `NOT_FOUND`. It never resolves upstream on the request path. The only scheduled writer is the hourly warmup, which covers Kong `origin=yearn` vault and underlying tokens over the trailing 7 days. A token or day outside that set only gets a row when someone runs a backfill script by hand, so a `404` there does not resolve on its own.
 
-If the fallback upstream is degraded, the request fails with `INTERNAL_ERROR` (`500`, `no-store`) rather than `NOT_FOUND` — a not-found response is cached for an hour, so a transient blip must not be recorded as "no price exists".
-
-If neither the stored rows nor the fallback have a price, the route returns:
+When no stored row exists, the route returns:
 
 ```json
 {
@@ -281,9 +285,9 @@ Response:
 }
 ```
 
-When no `source` filter is given, up to `10` pairs missing from the table are resolved through the source registry. The initial DeFiLlama lookup uses its provider-native batch endpoint; unresolved pairs then fall through to the full source registry, so a pair the batch matcher drops is still tried against the single-coin lookup this route's exact counterpart uses. DeFiLlama `429` responses are not retried on the request path, and each upstream attempt is capped at 2.5 seconds (`5xx` responses retry with backoff inside the route deadline). A pair whose own payload group failed is not retried against the DeFiLlama single-coin lookup; pairs from groups that answered are, even when the group matched no sample for them. Payload groups are requested concurrently, and the whole batch stage is capped at 2.5 seconds, so a slow or retrying provider cannot consume the whole deadline and starve the fallback: pairs still pending at that cap fall through to the remaining sources, and only those whose own group never answered skip the single-coin lookup. Upstream resolution has a five-second deadline measured from route entry, after which prices already completed are returned and the rest stay absent. Results for closed past days are stored in `token_prices` and returned in the same response. A non-chainlink resolution is stored only when its underlying observation falls within the DeFiLlama search width of the day key; an out-of-window result is served but not stored, so it resolves upstream again on the next cache miss. A `source` filter disables that resolution: the route answers from stored rows only.
+The route answers from stored rows only. It never resolves upstream on the request path. The hourly warmup covers Kong `origin=yearn` vault and underlying tokens over the trailing 7 days; any other token or older day only gets a row from a manually run backfill, so an omitted pair there stays omitted.
 
-Only found prices are returned. Pairs that upstream cannot resolve, and misses past the `10` per-request limit, are omitted from the response.
+Only found prices are returned. Pairs missing from the table are omitted from the response, and the batch is marked partial.
 
 ## `GET /api/prices/rangeHistorical`
 
@@ -384,14 +388,16 @@ Price responses set cache headers based on the requested timestamps and whether
 - Historical non-today exact price: `public, max-age=31536000, immutable`
 - Requests involving today's UTC day, or a batch pair whose day has not closed yet: `public, s-maxage=300, max-age=3600, stale-while-revalidate=14400`
 - Fully resolved batch or range for past days: `public, max-age=31536000, immutable`
-- Partially resolved batch or range for past days: `public, max-age=3600`
-- Historical not found responses: `public, max-age=3600, stale-while-revalidate=14400`
+- Partially resolved batch or range for past days: `public, s-maxage=300, max-age=300`
+- Historical not found responses: `public, s-maxage=300, max-age=300`
 - Spot: `public, s-maxage=120, stale-while-revalidate=600`
 
+A historical `404`, or a pair omitted from a partial batch or range, means the row is not in `token_prices` yet. It self-heals only for what the hourly warmup covers (Kong `origin=yearn` vault and underlying tokens, trailing 7 days); anything else waits on a manually run backfill. The 300s negative TTL applies to a `404` and to a partial batch or range made entirely of closed days — it is kept below the warmup cadence so a covered consumer sees the row soon after a job lands it. A batch or range that touches today's UTC day takes the today policy instead, so a missing closed day in that response carries the 1h browser `max-age` and 4h `stale-while-revalidate`.
+
 ## Edge caching
 
 Worker-generated responses do not populate Cloudflare's edge cache from a `Cache-Control` header alone — that header only drives the client/browser cache. Successful `GET` responses for all price routes are therefore stored in the edge cache (`caches.default`) explicitly and served from it on subsequent requests, using the TTLs above. A request is served from the edge before any Enso fetch or database query runs.
 
 Spot has no upstream cache policy (Enso sends only a weak `etag`), so its `s-maxage=120` is a chosen shared-cache TTL — short enough to keep prices fresh, long enough to absorb bursts — mirroring the Enso proxy already shipping in yearn.fi.
 
-The store/TTL decision is delegated to the Cache API: `caches.default.put()` reads the response's `Cache-Control`, refusing `no-store`/`private` and deriving the edge TTL from `s-maxage` (falling back to `max-age`, then `Expires`). Only successful responses are offered to `put()` — errors return straight from the worker's catch block and are never edge-stored (generic errors additionally carry `no-store` for downstream caches; historical not-found is the deliberate exception, returning a browser-cacheable negative result). Today's data sets `s-maxage=300` so the shared edge refreshes every ~5min, tracking the hourly warmup far more closely than the 1h browser `max-age`. The cache key is the request URL canonicalized first (sorted query params, and `coins` re-serialized with sorted keys and lowercased addresses) so requests that differ only in JSON ordering, whitespace, or address casing share one entry. Positional arrays — a range's `[start, end]` and a batch token's timestamp list — are never reordered, so two requests that differ in those never collide.
+The store/TTL decision is delegated to the Cache API: `caches.default.put()` reads the response's `Cache-Control`, refusing `no-store`/`private` and deriving the edge TTL from `s-maxage` (falling back to `max-age`, then `Expires`). Only successful responses are offered to `put()` — errors return straight from the worker's catch block and are never edge-stored (generic errors additionally carry `no-store` for downstream caches; historical not-found is the deliberate exception, returning a short-lived cacheable negative result). Today's data sets `s-maxage=300` so the shared edge refreshes every ~5min, tracking the hourly warmup far more closely than the 1h browser `max-age`. The cache key is the request URL canonicalized first (sorted query params, and `coins` re-serialized with sorted keys and lowercased addresses) so requests that differ only in JSON ordering, whitespace, or address casing share one entry. Positional arrays — a range's `[start, end]` and a batch token's timestamp list — are never reordered, so two requests that differ in those never collide.

src/cache/headers.ts

@@ -5,15 +5,21 @@ export const CACHE_CONTROL_IMMUTABLE = 'public, max-age=31536000, immutable'
 // shared edge refresh every ~5min — far tighter than the warmup cadence — while browsers
 // keep the gentler 1h max-age.
 export const CACHE_CONTROL_TODAY = 'public, s-maxage=300, max-age=3600, stale-while-revalidate=14400'
-export const CACHE_CONTROL_PARTIAL = 'public, max-age=3600'
-export const CACHE_CONTROL_NOT_FOUND = 'public, max-age=3600, stale-while-revalidate=14400'
+// A historical miss (a 404, or a pair omitted from a batch/range) means the row is not
+// in the table yet, not that no price exists: the hourly warmup (Kong origin=yearn vaults
+// and underlyings, trailing 7 days) or a manually run backfill can fill it. Keep both
+// negative TTLs under the warmup cadence so a client stops seeing the gap soon after the
+// row lands. CACHE_CONTROL_PARTIAL only reaches responses made entirely of closed days —
+// a batch/range touching today takes CACHE_CONTROL_TODAY before the partial check runs.
+export const CACHE_CONTROL_PARTIAL = 'public, s-maxage=300, max-age=300'
+export const CACHE_CONTROL_NOT_FOUND = 'public, s-maxage=300, max-age=300'
 // Spot is a live proxy with no upstream cache policy. Short shared-cache TTL so the
 // edge absorbs bursts without serving long-stale prices; mirrors yearn.fi's Enso proxy.
 export const CACHE_CONTROL_SPOT = 'public, s-maxage=120, stale-while-revalidate=600'
 // Generic error responses set this so no client or shared cache retains them. (Errors
 // also never reach writeEdgeCache — they return from the request handler's catch block —
 // so they don't populate the edge cache regardless.) Historical not-found is the one
-// deliberate exception: it returns a cacheable negative result (CACHE_CONTROL_NOT_FOUND).
+// deliberate exception: it returns a short-lived negative result (CACHE_CONTROL_NOT_FOUND).
 export const CACHE_CONTROL_NO_STORE = 'no-store'
 
 export function cacheControlForHistorical(timestamp: number): string {

src/lander.ts

@@ -156,7 +156,7 @@ export function renderLandingPage(env: Env, baseUrl: string): string {
 <main>
   <header>
     <h1>Yearn Price Service</h1>
-    <p class="lede">Spot and historical token prices for Yearn, aggregated from DefiLlama, on-chain oracles, Curve, Bob's API, and Enso.</p>
+    <p class="lede">Spot token prices for Yearn from Enso, and historical prices from DefiLlama, Curve, and on-chain derived pricing.</p>
   </header>
 
   <section>
@@ -215,7 +215,7 @@ export function renderLandingPage(env: Env, baseUrl: string): string {
       <li>Spot: <code>public, s-maxage=120, stale-while-revalidate=600</code>.</li>
       <li>Historical prices for past days: <code>public, max-age=31536000, immutable</code>.</li>
       <li>Historical requests for today's UTC day: <code>public, s-maxage=300, max-age=3600, stale-while-revalidate=14400</code>.</li>
-      <li>Partially resolved batch or range for past days: <code>public, max-age=3600</code>.</li>
+      <li>Partially resolved batch or range for past days, and historical not-found: <code>public, s-maxage=300, max-age=300</code>.</li>
       <li>Responses are edge-cached at Cloudflare with canonicalized URLs, so requests that differ only in query order, whitespace, or address casing share one entry.</li>
     </ul>
   </section>

src/registries/historical.ts

@@ -35,8 +35,8 @@ export function createHistoricalSources(env?: Env): HistoricalPriceSource[] {
   // Per-attempt timeout only; 5xx responses still retry with backoff, so the
   // batch stage is capped separately by BATCH_STAGE_BUDGET_MS. Retrying every
   // member of a rate-limited batch in lockstep amplifies the provider burst;
-  // offline jobs keep their retry policy. The exact route shares this client,
-  // so it also loses 429 retries — see docs/routes.md.
+  // offline jobs keep their retry policy. No route calls this registry today;
+  // the budgets bound any future request-path caller.
   const client = new DefiLlamaClient(undefined, undefined, { timeoutMs: 2_500, retryRateLimits: false })
   const marketSources = [
     createDefiLlamaHistoricalSource(client),
@@ -55,9 +55,9 @@ export function createHistoricalSources(env?: Env): HistoricalPriceSource[] {
 }
 
 /**
- * Half the route's five-second resolution budget. A slow or retrying provider
- * batch call must not consume the whole deadline: the rest of the source chain
- * still needs time to price the pairs the batch never returned.
+ * Cap on the provider batch stage. A slow or retrying provider batch call must
+ * not consume a caller's whole budget: the rest of the source chain still needs
+ * time to price the pairs the batch never returned.
  */
 const BATCH_STAGE_BUDGET_MS = 2_500