Repository navigation
feat(shape): z-normalized shape search — find any pattern across the history - #89
Conversation
…history Roadmap v3's flagship: brush any region (or pass times, or press Find shape on the visible window) and findShape() ranks every occurrence of that shape across the whole loaded history. The math (shapeSearch in core.js, exported) is MASS-style: one O(n log n) FFT round-trip computes all sliding dot products against the z-normalized query, prefix sums give candidate-window stats, and distances fall out as 2m(1-rho) where rho is the Pearson correlation of the z-normalized shapes (reported as 'score' — 1 identical, -1 inverted). Windows overlapping the query or an accepted match are excluded greedily, so near-duplicate neighbours collapse. Query length clamps to 8..750; flat queries and flat windows are principled no-matches. Measured: 100k-bar history searched in ~30 ms — interactive at any size, so the roadmap's worker offload was not needed (noted in ROADMAP.md). The hand-rolled iterative radix-2 FFT is the only new machinery. Chart surface: findShape()/clearShape()/shapeResult, the wick:shape event with the full payload, query/match translucent bands behind the series, and setData() clearing (indices are data-bound). Covered three ways: differential unit tests against an independent brute-force O(n·m) reference, golden vectors vs a numpy brute-force reference (follows in its own commit), and e2e paint/clear specs. The golden gate caught a SuperTrend regression mid-development — a botched edit of the flip-gap line — before it could ship; the method works.
shapeSearch's FFT + z-norm math is ~1.9 KB in core; the chart surface (findShape/clearShape/shapeResult, wick:shape payload, band rendering) adds ~1.2 KB. Landed at 70.3 KB.
…ference A numpy O(n·m) reference (direct per-window z-normalization and dot products — no FFT anywhere) pins the FFT path's match list: starts exact, scores/distances within tolerance. The coverage guard now knows shapeSearch is a math kernel (not calc*-named), 22/22 covered.
…ap strikes the row docs.html gains a live findShape(visible range) playground; the demo toolbar gets a Find shape toggle that searches the visible window's shape and toasts the match count/duration/best score; README documents the method, event and section; ROADMAP strikes the shipped row with the sync-not-worker note.
CI's wider Linux fallback fonts laid the labelled button out past 412px (426px, zoomed-out page). Moving it into cluster-view restores the Market cluster's exact pre-feature width — strictly narrower than the layout that already passed on any font — and the button gains the established icon-only + screen-reader-label + aria-pressed toggle pattern.
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e19a49853
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| // selection indices are data-bound; a replacement invalidates them | ||
| if (this._brushSel || this._brushDrag) this.clearBrush(); | ||
| if (this._shape) this._shape = null; // indices are data-bound too |
There was a problem hiding this comment.
When setData([]) is used, the early branch calls clearData() before reaching this new reset, so the old index-based _shape remains stored. If streaming subsequently repopulates the chart through update(), shapeResult can report unrelated bars or throw when a stored match index exceeds the new dataset; historical inserts and _maybeLoadMore() prepends similarly shift these indices without rebasing them. Clear or rebase shape state on every data path that empties or shifts the primary series.
Useful? React with 👍 / 👎.
| const matches = shapeSearch( | ||
| d.map((b) => b.close), | ||
| i0, | ||
| i1 - i0 + 1, |
There was a problem hiding this comment.
For a requested range longer than 750 bars, shapeSearch() silently searches only the first 750 bars, but this wrapper retains the original i1 in the result and query band. Consequently the payload misidentifies the searched shape, and a match starting at bar 750 can even be returned despite overlapping the reported query range. Clamp or reject the chart-level range before both searching and storing it.
Useful? React with 👍 / 👎.
| const mean = (csum[i + m] - csum[i]) / m; | ||
| const sig = Math.sqrt(Math.max(0, (csq[i + m] - csq[i]) / m - mean * mean)); |
There was a problem hiding this comment.
When prices have a large baseline relative to their movement, computing variance as E[x²] - E[x]² suffers catastrophic cancellation. For example, adding 1e9 to an otherwise identical repeated pattern can make this search return no matches, even though z-normalized similarity must be invariant to that offset. Use a centered or otherwise numerically stable rolling variance calculation so high-valued instruments are not treated as flat or assigned corrupted scores.
Useful? React with 👍 / 👎.
…rting, offset-stable variance All three Codex findings, all legitimate: 1. setData([]) early-returns through clearData(), so the shape reset added to setData never ran on that path — stale index-bound state survived an empty-then-repopulated chart. clearData() now clears it. Backfill prepends (loadMore) shift every index by 'added'; the shape result rebases instead of dropping, so bands survive loading more history in front of them. 2. A query range longer than the 750-bar clamp searched only the first 750 bars while reporting the full range — the payload misidentified the shape, and a returned match could overlap the *reported* query. findShape now clamps i1 to i0 + SHAPE_MAX_WINDOW - 1 (the cap is exported from core) before searching, storing, or reporting. 3. Window variance came from raw moments (E[x²] − E[x]²), which cancel catastrophically when the price baseline dwarfs the movement — a repeated pattern on a 1e9-baseline series returned zero matches. The series is now shifted by its first value once, so every statistic runs at movement scale; z-normalized similarity was already offset-invariant in exact arithmetic, and now is in float64. Each carries a regression test (offset-invariance at 1e9, clamp reporting + overlap, clearData/rebase source contracts).
|
All three findings addressed in 1b22dc3 — each was legitimate, thank you: 1. Data-reset invalidation — 2. Honest clamp reporting — the cap is now exported ( 3. Offset-stable variance — the series is shifted by its first value once, so the raw-moment window statistics run at movement scale instead of price scale. The 1e9-baseline scenario from the review is now a regression test: the repeated pattern finds all its matches at score ≈ 1 (it previously returned none).
|
1 participant
Roadmap v3's flagship feature, PR #2 of the recommended five: brush any region and find every occurrence of that shape across the whole loaded history — matrix-profile-class similarity search shipped inside a web chart.
The math —
shapeSearch(exported fromwickchart/core)MASS-style, one O(n log n) FFT round-trip (hand-rolled iterative radix-2): sliding dot products against the z-normalized query + prefix sums for window stats → distances fall out as
2m(1−ρ)where ρ is the Pearson correlation of the z-normalized shapes (reported asscore: 1 identical, −1 inverted, 0.8 strong). Both query and candidates are mean-removed and unit-variance, so matches are about form — the dip you're looking at matches at any price or amplitude. Greedy exclusion collapses overlapping matches; query length clamps 8–750; flat queries/windows are principled no-matches.Measured: a 100k-bar history searches in ~30 ms — interactive at any size, so the roadmap's planned worker offload wasn't needed (ROADMAP row struck with the note). The unit suite pins this with a perf sanity.
The surface
findShape({from, to, maxMatches, minScore})(no args → the brush selection) ·clearShape()·shapeResultgetter ·wick:shapeevent · translucent query/match bands behind the series ·setData()clears (indices are data-bound).Verification — the flagship lands on tested ground
npm test— 704/704 · e2e — 80 passed / 4 skipped.A story worth the PR body
Mid-development the golden gate caught a SuperTrend regression — a botched edit of the flip-gap line during this feature's development — before it could ship. The correctness harness built in PR #88 paid for itself two PRs later.
Docs: README section + method/event rows, docs.html section with a live playground (TOC updated), demo toolbar Find shape toggle, ROADMAP row struck.