Commit f4517bd
comix: first-class search, --multi-source, and image-quality probe (#62)
* comix: first-class search, --multi-source, and image-quality probe
Promote the comix.to handler from a URL-seed-only, un-probed,
multi-source-excluded source to a full participant.
- Keyword search: the token-gated + encrypted /api/v1/manga?keyword=
endpoint can't be reproduced from Python, so search() now drives the
header typeahead in the persistent Patchright browser
(fetch_search_via_dom) and maps the dropdown rows to SearchHits. All
failures swallow to [] (never raise) -- the orchestrator's persistent
ProbeFailureCache would otherwise blocklist comix.to for an hour after
two flaky searches, and comix issues no HTTP request here.
- Multi-source: drop SKIP_MULTI_SOURCE so comix competes as an alternate
and contributes to the aligned chapter map.
- Image-quality probe: drop SKIP_QUALITY_PROBE and add a custom
single-chapter _probe_chapter_aggregate override. The base 8-chapter
breadth probe is infeasible -- each chapter renders fully in the
single-threaded browser bridge, blowing the 240s probe deadline. Probe
ONE chapter (chapter 1 by preference, _pick_probe_chapter) with a capped
page render (new max_capture_pages on fetch_chapter_images_via_dom) and
score the latter half of the captured pages by median. A single early
page mis-scored the flagship series at 0.1 on a sparse cold-open; the
latter-half median lands it at ~0.79, in line with peer sites.
_fetch_probe_item_bytes is overridden to read the browser image_cache
first so synthetic comix-page:// canvas URLs score fairly instead of 0.0.
The calibrated 0.74 seed is now only the probe's fallback.
- UI: remove comix's stale "broken handler" red marker (the canvas-timeout
warning was retired with the 2026-07-11 webp rewrite).
The SKIP_QUALITY_PROBE / SKIP_MULTI_SOURCE getattr hooks remain in
search_orchestrator.py and aio_search_cli.py as generic opt-outs that no
handler sets today (comment-only updates there).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* search: memoized/overlapped pipeline, clamped-probe cache, quality_basis UI flag
Cross-site search + --multi-source discovery took 40-100s+ even with
--seeded-only: a dead fan-out timeout, strictly-serial phases, the same
chapter list fetched up to 3x, and clamped probes re-paid every search.
Frieren benchmark (--seeded-only --search-json --multi-source, warm
caches): 219.9s -> 75.7s.
- Fan-out: daemon-thread pool (was ThreadPoolExecutor whose join waited
for the slowest handler), width 16, comix enqueued first
(SEARCH_COST_HINT), 60s soft barrier that stops waiting but never
kills a straggler -- late completions merge post-T3 with title-match
+ seed/cached rating only.
- Clamped (non-top-candidate) probes now cache with a 7-day TTL and a
shared _desired_max_samples rule so a promoted top candidate always
re-probes full and overwrites unclamped. comix's PROBE_SAMPLES_FIXED
removes its every-search browser re-probe entirely.
- probe_candidate_limit caps how many ranked candidates get probed at
all: 1 for --auto-pick/direct-URL (only candidates[0] is ever read),
2 for UI searches.
- New sites/fetch_memo.py: per-run memo shared by the probe phase, T3
pairwise, and the winner chapter fetch so each (site, url)'s context
+ chapter list is fetched from the network at most once.
- T3 pairwise parallelizes its per-source page fetches (serial ==
parallel adjustments, verified offline).
- mangadex's per-hit DMCA probe capped at 5 hits (was up to 20).
- BaseSiteHandler.PROBE_SOURCE_BUDGET_S bounds one chapter-probe call
to 120s so a timing-out host (mangakatana) can't hold the probe
phase to its 240s deadline on every search and never cache.
- New quality_basis field ("chapter_probe"/"cover"/"seed") drives a
red AlertTriangle in SearchSourceCard.jsx next to any source whose
rating isn't grounded in a measured chapter page.
- Per-phase and per-site timing lines on stderr for future tuning.
Offline regression: tools/_test_search_perf_opts.py (74 checks,
gitignored). Ranking parity confirmed against baseline JSON (same
winner, same aligned chapter count) across cold/warm/auto-pick runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* search: parallelize the per-source image-probe breadth pass
The image-quality probe's breadth sampler (1 page from each of up to 8
chapters, in BaseSiteHandler._probe_chapter_aggregate) ran serially. For
HTML-scraped sites whose page-image list lives in the chapter HTML —
mangakatana pays a full Cloudflare-fronted origin GET per sample — that was
~8x serial round trips: ~43s cold-cache for a probe an actual chapter
download does in ~4s, and the dominant cost behind the "slow site" search
verdict.
Run the breadth samples on a bounded daemon pool (PROBE_BREADTH_CONCURRENCY,
default 4, env AIO_PROBE_BREADTH_CONCURRENCY; a subclass sets 1 to force
serial). Byte-identical to serial: which chapter+page each sample fetches
is fixed before any I/O by the two pure pickers, aggregation is
order-independent, and each worker writes a preallocated slot that a
post-join sweep rebuilds into the four index-aligned lists + the compacted
per_chapter_metas in pick order. probe_deadline now bounds both task-start
and the wall-clock join; the throttle tail stays serial and post-join (it
is the sequential-throttle measurement). comix is a whole-method override
(untouched); rizzcomic post-processes the return tuple (transparent).
Verified live (mangakatana 35-ch: identical score at concurrency 1 vs 4)
and offline via a new parallel==serial equivalence check; the per-source
budget test was reworked for the concurrent path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* search: recommend and disable slow/down search sites
Some supported sites are unreachable from a given connection or just down.
The search fan-out + image-quality probe already measure exactly who the
culprits are every run, but that data was write-only to stderr with no way
to act on it. This surfaces it: a non-blocking callout in the Search tab
recommends disabling chronically slow/down sites, backed by a persistent
Settings list. Disabling excludes a site from the search fan-out/probe
AND from multi-source download alternatives.
Backend (search_orchestrator.py): search_all gains exclude_sites +
diagnostics kwargs. A return-time _emit_site_health() assembles a
slow/down verdict per site from timing the phases already collect (fan-out
duration, probe duration, errors, late/never-finished, ProbeFailureCache
blocks) — zero added network cost for a healthy search. Flagged sites then
get a second-stage bounded reachability probe (single liveness GET per
candidate host, 5s/host, 12s phase cap, skipped for constitutionally-slow/
official handlers) that corrects mechanism-based mislabeling: a site whose
search endpoint times out then raises but whose downloads work fine
(mangakatana) softens from down/error to slow/search_error; a site that
swallows an all-mirrors-down failure to [] (zeroscans - now propagates
instead) or is slow-but-actually-dead (omegascans) promotes to
down/unreachable. A confirmed-unreachable host also feeds the
ProbeFailureCache so it trends toward auto-block. A healthy all-ok search
probes nothing, so the 2026-07-12 perf work is unaffected.
aio_search_cli.py / aio-dl.py: --disable-sites flag + parse_disable_sites,
wired into both search_all call sites and a guard-filter over the
assembled multi-source alternatives dict (also scrubs the persisted
run_params.json cache so a disabled site can't get re-cached).
UI: Electron emits --disable-sites from settings (searcher.js +
downloader.js). useDownloader tracks a durable disabledSites list plus an
in-memory searchSiteHealth strike/decay map (down +2, slow +1 capped at 4,
decays -1 when tested-and-healthy, deleted at 0) - deliberately NOT
persisted to settings, since that would fire SettingsTab's hydration
effect mid-search and clobber an unsaved Settings draft. SearchTab renders
an amber SlowSitesCallout above results (per-site checkboxes, status pill,
diagnostic latency bar, "Disable & re-search"). SettingsTab gets an
immediate-persist "Search Sources" section (disabled chips, sites flagged
this session, add-by-name) plus diff-aware settings hydration so that
immediate write can no longer clobber an in-progress unsaved edit
elsewhere in the form.
Verified: tools/_test_site_health.py (63 checks, incl. the reachability
truth table + a mocked end-to-end probe) and tools/_test_search_perf_opts.py
(77 checks) offline; live --search-json / --disable-sites smoke tests;
npm run build (1265 modules); real-app E2E.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>1 parent 9454596 commit f4517bd
16 files changed
Lines changed: 3237 additions & 389 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
252 | 252 | | |
253 | 253 | | |
254 | 254 | | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
255 | 269 | | |
256 | 270 | | |
257 | 271 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
68 | 81 | | |
69 | 82 | | |
70 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
35 | 40 | | |
36 | 41 | | |
37 | 42 | | |
| |||
102 | 107 | | |
103 | 108 | | |
104 | 109 | | |
105 | | - | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
106 | 116 | | |
107 | 117 | | |
108 | 118 | | |
| |||
193 | 203 | | |
194 | 204 | | |
195 | 205 | | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
196 | 211 | | |
197 | 212 | | |
198 | 213 | | |
| |||
229 | 244 | | |
230 | 245 | | |
231 | 246 | | |
| 247 | + | |
| 248 | + | |
232 | 249 | | |
233 | 250 | | |
234 | 251 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
54 | 72 | | |
55 | 73 | | |
56 | 74 | | |
| |||
76 | 94 | | |
77 | 95 | | |
78 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
79 | 100 | | |
80 | 101 | | |
81 | 102 | | |
| |||
208 | 229 | | |
209 | 230 | | |
210 | 231 | | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
211 | 251 | | |
212 | 252 | | |
213 | 253 | | |
| |||
0 commit comments