Skip to content

Commit f4517bd

Browse files
Thundia2claude
andauthored
comix: first-class search, --multi-source, and image-quality probe (#62)
* comix: first-class search, --multi-source, and image-quality probe Promote the comix.to handler from a URL-seed-only, un-probed, multi-source-excluded source to a full participant. - Keyword search: the token-gated + encrypted /api/v1/manga?keyword= endpoint can't be reproduced from Python, so search() now drives the header typeahead in the persistent Patchright browser (fetch_search_via_dom) and maps the dropdown rows to SearchHits. All failures swallow to [] (never raise) -- the orchestrator's persistent ProbeFailureCache would otherwise blocklist comix.to for an hour after two flaky searches, and comix issues no HTTP request here. - Multi-source: drop SKIP_MULTI_SOURCE so comix competes as an alternate and contributes to the aligned chapter map. - Image-quality probe: drop SKIP_QUALITY_PROBE and add a custom single-chapter _probe_chapter_aggregate override. The base 8-chapter breadth probe is infeasible -- each chapter renders fully in the single-threaded browser bridge, blowing the 240s probe deadline. Probe ONE chapter (chapter 1 by preference, _pick_probe_chapter) with a capped page render (new max_capture_pages on fetch_chapter_images_via_dom) and score the latter half of the captured pages by median. A single early page mis-scored the flagship series at 0.1 on a sparse cold-open; the latter-half median lands it at ~0.79, in line with peer sites. _fetch_probe_item_bytes is overridden to read the browser image_cache first so synthetic comix-page:// canvas URLs score fairly instead of 0.0. The calibrated 0.74 seed is now only the probe's fallback. - UI: remove comix's stale "broken handler" red marker (the canvas-timeout warning was retired with the 2026-07-11 webp rewrite). The SKIP_QUALITY_PROBE / SKIP_MULTI_SOURCE getattr hooks remain in search_orchestrator.py and aio_search_cli.py as generic opt-outs that no handler sets today (comment-only updates there). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * search: memoized/overlapped pipeline, clamped-probe cache, quality_basis UI flag Cross-site search + --multi-source discovery took 40-100s+ even with --seeded-only: a dead fan-out timeout, strictly-serial phases, the same chapter list fetched up to 3x, and clamped probes re-paid every search. Frieren benchmark (--seeded-only --search-json --multi-source, warm caches): 219.9s -> 75.7s. - Fan-out: daemon-thread pool (was ThreadPoolExecutor whose join waited for the slowest handler), width 16, comix enqueued first (SEARCH_COST_HINT), 60s soft barrier that stops waiting but never kills a straggler -- late completions merge post-T3 with title-match + seed/cached rating only. - Clamped (non-top-candidate) probes now cache with a 7-day TTL and a shared _desired_max_samples rule so a promoted top candidate always re-probes full and overwrites unclamped. comix's PROBE_SAMPLES_FIXED removes its every-search browser re-probe entirely. - probe_candidate_limit caps how many ranked candidates get probed at all: 1 for --auto-pick/direct-URL (only candidates[0] is ever read), 2 for UI searches. - New sites/fetch_memo.py: per-run memo shared by the probe phase, T3 pairwise, and the winner chapter fetch so each (site, url)'s context + chapter list is fetched from the network at most once. - T3 pairwise parallelizes its per-source page fetches (serial == parallel adjustments, verified offline). - mangadex's per-hit DMCA probe capped at 5 hits (was up to 20). - BaseSiteHandler.PROBE_SOURCE_BUDGET_S bounds one chapter-probe call to 120s so a timing-out host (mangakatana) can't hold the probe phase to its 240s deadline on every search and never cache. - New quality_basis field ("chapter_probe"/"cover"/"seed") drives a red AlertTriangle in SearchSourceCard.jsx next to any source whose rating isn't grounded in a measured chapter page. - Per-phase and per-site timing lines on stderr for future tuning. Offline regression: tools/_test_search_perf_opts.py (74 checks, gitignored). Ranking parity confirmed against baseline JSON (same winner, same aligned chapter count) across cold/warm/auto-pick runs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * search: parallelize the per-source image-probe breadth pass The image-quality probe's breadth sampler (1 page from each of up to 8 chapters, in BaseSiteHandler._probe_chapter_aggregate) ran serially. For HTML-scraped sites whose page-image list lives in the chapter HTML — mangakatana pays a full Cloudflare-fronted origin GET per sample — that was ~8x serial round trips: ~43s cold-cache for a probe an actual chapter download does in ~4s, and the dominant cost behind the "slow site" search verdict. Run the breadth samples on a bounded daemon pool (PROBE_BREADTH_CONCURRENCY, default 4, env AIO_PROBE_BREADTH_CONCURRENCY; a subclass sets 1 to force serial). Byte-identical to serial: which chapter+page each sample fetches is fixed before any I/O by the two pure pickers, aggregation is order-independent, and each worker writes a preallocated slot that a post-join sweep rebuilds into the four index-aligned lists + the compacted per_chapter_metas in pick order. probe_deadline now bounds both task-start and the wall-clock join; the throttle tail stays serial and post-join (it is the sequential-throttle measurement). comix is a whole-method override (untouched); rizzcomic post-processes the return tuple (transparent). Verified live (mangakatana 35-ch: identical score at concurrency 1 vs 4) and offline via a new parallel==serial equivalence check; the per-source budget test was reworked for the concurrent path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * search: recommend and disable slow/down search sites Some supported sites are unreachable from a given connection or just down. The search fan-out + image-quality probe already measure exactly who the culprits are every run, but that data was write-only to stderr with no way to act on it. This surfaces it: a non-blocking callout in the Search tab recommends disabling chronically slow/down sites, backed by a persistent Settings list. Disabling excludes a site from the search fan-out/probe AND from multi-source download alternatives. Backend (search_orchestrator.py): search_all gains exclude_sites + diagnostics kwargs. A return-time _emit_site_health() assembles a slow/down verdict per site from timing the phases already collect (fan-out duration, probe duration, errors, late/never-finished, ProbeFailureCache blocks) — zero added network cost for a healthy search. Flagged sites then get a second-stage bounded reachability probe (single liveness GET per candidate host, 5s/host, 12s phase cap, skipped for constitutionally-slow/ official handlers) that corrects mechanism-based mislabeling: a site whose search endpoint times out then raises but whose downloads work fine (mangakatana) softens from down/error to slow/search_error; a site that swallows an all-mirrors-down failure to [] (zeroscans - now propagates instead) or is slow-but-actually-dead (omegascans) promotes to down/unreachable. A confirmed-unreachable host also feeds the ProbeFailureCache so it trends toward auto-block. A healthy all-ok search probes nothing, so the 2026-07-12 perf work is unaffected. aio_search_cli.py / aio-dl.py: --disable-sites flag + parse_disable_sites, wired into both search_all call sites and a guard-filter over the assembled multi-source alternatives dict (also scrubs the persisted run_params.json cache so a disabled site can't get re-cached). UI: Electron emits --disable-sites from settings (searcher.js + downloader.js). useDownloader tracks a durable disabledSites list plus an in-memory searchSiteHealth strike/decay map (down +2, slow +1 capped at 4, decays -1 when tested-and-healthy, deleted at 0) - deliberately NOT persisted to settings, since that would fire SettingsTab's hydration effect mid-search and clobber an unsaved Settings draft. SearchTab renders an amber SlowSitesCallout above results (per-site checkboxes, status pill, diagnostic latency bar, "Disable & re-search"). SettingsTab gets an immediate-persist "Search Sources" section (disabled chips, sites flagged this session, add-by-name) plus diff-aware settings hydration so that immediate write can no longer clobber an in-progress unsaved edit elsewhere in the form. Verified: tools/_test_site_health.py (63 checks, incl. the reachability truth table + a mocked end-to-end probe) and tools/_test_search_perf_opts.py (77 checks) offline; live --search-json / --disable-sites smoke tests; npm run build (1265 modules); real-app E2E. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 9454596 commit f4517bd

16 files changed

Lines changed: 3237 additions & 389 deletions

UI-source/electron/downloader.js

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,20 @@ function buildCliArgs(args) {
252252
cliArgs.push("--collapse-splits");
253253
}
254254

255+
// User-disabled sites also drop out as MULTI-SOURCE download alternatives —
256+
// not just from search (the user's "search + downloads" scope decision). Same
257+
// chokepoint shape as collapseSplits above: comma-joined handler names →
258+
// --disable-sites. useDownloader.queueDownload injects args.disabledSites from
259+
// settings.disabledSites on EVERY spawn (manual / search / library / queue),
260+
// so a disabled site is skipped by find_alternatives_for_direct_url and the
261+
// guard-filter of _multi_source_alternatives in aio-dl.py (which also scrubs
262+
// it from any prefetched/disk-cached alt list). Harmless no-op on
263+
// single-source downloads — Python only consults the exclusion under
264+
// --multi-source. Array+length guarded, mirroring searcher.js:buildSearchArgs.
265+
if (Array.isArray(args.disabledSites) && args.disabledSites.length) {
266+
cliArgs.push("--disable-sites", args.disabledSites.join(","));
267+
}
268+
255269
// LINE Webtoon recompression valued knobs (Phase 1, 2026-05-11). Only
256270
// emit when the master toggle is on AND the value differs from the
257271
// Python-side argparse default (85 for quality, 4 for method). Without

UI-source/electron/searcher.js

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,19 @@ function buildSearchArgs(query, opts = {}) {
6565
// _ML_RATING_ENABLED docstring for the full rationale.
6666
if (opts.enableMlRating) args.push("--enable-ml-rating");
6767

68+
// User-disabled search sites (Settings → Search → "Search Sources", or the
69+
// SlowSitesCallout's "Disable & re-search"). Comma-joined handler names →
70+
// --disable-sites, which drops them from the fan-out AND the image-quality
71+
// probe (Python: aio_search_cli.parse_disable_sites → search_all
72+
// exclude_sites). useDownloader.runSearch injects opts.disabledSites from
73+
// settings.disabledSites on every search; the callout ALSO passes an explicit
74+
// list in opts to dodge the settings-ref update race on its immediate
75+
// re-search (opts wins in the hook's finalOpts merge). Array+length guarded,
76+
// mirroring the download-path emitter in downloader.js:buildCliArgs.
77+
if (Array.isArray(opts.disabledSites) && opts.disabledSites.length) {
78+
args.push("--disable-sites", opts.disabledSites.join(","));
79+
}
80+
6881
return args;
6982
}
7083

UI-source/src/App.jsx

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,11 @@ const TABS = [
3232
export default function App() {
3333
const [activeTab, setActiveTab] = useState("new");
3434
const [downloadDraft, setDownloadDraft] = useState(null);
35+
// When the SearchTab slow-sites callout's "Manage in Settings" link fires, we
36+
// jump to the Settings tab AND preselect its Search category. Null the rest of
37+
// the time so a manual Settings visit (via the rail) lands on the default pane
38+
// — the rail button clears it below.
39+
const [settingsCategory, setSettingsCategory] = useState(null);
3540

3641
// Central hook that manages all download state and Electron IPC
3742
const dl = useDownloader();
@@ -102,7 +107,12 @@ export default function App() {
102107
return (
103108
<button
104109
key={tab.id}
105-
onClick={() => setActiveTab(tab.id)}
110+
onClick={() => {
111+
// A manual Settings visit resets any callout-forced category so
112+
// it opens on the default (General) pane, not Search.
113+
if (tab.id === "settings") setSettingsCategory(null);
114+
setActiveTab(tab.id);
115+
}}
106116
title={tab.label}
107117
className={cn(
108118
"relative flex flex-col items-center justify-center w-12 h-12 rounded-lg",
@@ -193,6 +203,11 @@ export default function App() {
193203
onSaveSettings={dl.saveSettings}
194204
resumable={dl.resumable}
195205
onResumeDownload={dl.resumeDownload}
206+
searchSiteHealth={dl.searchSiteHealth}
207+
onManageSources={() => {
208+
setSettingsCategory("search");
209+
setActiveTab("settings");
210+
}}
196211
onStartDownload={(url, args) => {
197212
// Fix A (2026-05-07): merge settings.defaults so search- and
198213
// library-driven downloads inherit format / quality / scaling /
@@ -229,6 +244,8 @@ export default function App() {
229244
<SettingsTab
230245
settings={dl.settings}
231246
onSave={dl.saveSettings}
247+
searchSiteHealth={dl.searchSiteHealth}
248+
initialCategory={settingsCategory}
232249
/>
233250
)}
234251
</div>

UI-source/src/components/SearchSourceCard.jsx

Lines changed: 54 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -37,20 +37,38 @@ function qualityTier(score) {
3737
// Add/remove entries here when handlers break / get fixed. Each value
3838
// is the tooltip text the user sees on hover.
3939
//
40-
// Entries:
41-
// - comix (added 2026-05-27): the canvas-scrape fallback for
42-
// unscrambled chapter images can't render all pages within the 300s
43-
// per-chapter time budget. Chapter downloads time out at single-
44-
// digit page counts (e.g. 3/193 pages in the 2026-05-27 Shangri-La
45-
// Frontier run). The Patchright bridge is single-threaded so
46-
// increasing parallelism doesn't help. See sites/comix.py for the
47-
// in-progress work and the PR-31 context doc's "fix comix" pointer.
48-
const BROKEN_HANDLERS = {
49-
comix: (
50-
"Currently broken: canvas-scrape can't capture all pages within the "
51-
+ "per-chapter time budget. Downloads typically time out at a small "
52-
+ "number of pages. You can still try, but expect failure."
53-
),
40+
// Entries: none currently. comix was flagged here (2026-05-27) while its
41+
// chapter images came off a single-threaded canvas scrape that timed out
42+
// mid-chapter; that warning was retired with comix's 2026-07-11 rewrite to
43+
// plain directly-fetchable webp <img> pages plus its 2026-07-12 promotion to
44+
// a first-class search / --multi-source / quality-probed source. The empty
45+
// map keeps the render affordance (the icon just renders for nothing) ready
46+
// for the next handler that breaks.
47+
const BROKEN_HANDLERS = {};
48+
49+
// Where this source's displayed rating came from. The orchestrator emits
50+
// `quality_basis` per source (search_orchestrator.py:_quality_basis —
51+
// "chapter_probe" | "cover" | "seed"); anything that ISN'T a real
52+
// chapter-page measurement gets a red AlertTriangle next to the site name
53+
// (user request 2026-07-12, same affordance BROKEN_HANDLERS uses). That
54+
// covers: seed-prior-only ratings (unprobed candidates under the top-2
55+
// probe scope, late-merged fan-out stragglers, cache misses) and
56+
// cover-image fallbacks / all-samples-failed probes. The fallback inference
57+
// below handles JSON from an older spawn that predates the field: null
58+
// score means the UI is showing seed_quality; metadata without a successful
59+
// sample means cover/failed-probe.
60+
function qualityBasis(source) {
61+
if (source.quality_basis) return source.quality_basis;
62+
if (source.img_quality_score == null) return "seed";
63+
const m = source.img_quality_metadata;
64+
if (m && (m.samples_succeeded ?? 0) > 0) return "chapter_probe";
65+
return "cover";
66+
}
67+
68+
const QUALITY_BASIS_WARNINGS = {
69+
seed: "Rating is the per-site prior — no chapter pages were measured for this source in this search.",
70+
cover:
71+
"Rating was not measured from chapter pages — it comes from the cover image only, or every sampled page failed to fetch.",
5472
};
5573

5674
export default function SearchSourceCard({
@@ -76,6 +94,9 @@ export default function SearchSourceCard({
7694

7795
const final = source.img_quality_score != null ? source.img_quality_score : source.seed_quality;
7896
const tier = qualityTier(final);
97+
// Red-triangle warning when the displayed rating isn't grounded in real
98+
// chapter pages. null for "chapter_probe" (no icon).
99+
const basisWarning = QUALITY_BASIS_WARNINGS[qualityBasis(source)] || null;
79100
// Phase H aggregate metadata from sites/base.py:_probe_chapter_aggregate.
80101
// null when un-measured (cache miss + probe failed). Drives the format chip
81102
// beside the site name, the bpp/B&W/outlier breakdown in the quality-bar
@@ -208,6 +229,25 @@ export default function SearchSourceCard({
208229
/>
209230
</span>
210231
)}
232+
{/* Quality-basis danger icon (2026-07-12): red AlertTriangle when
233+
the displayed rating is seed-prior-only or a cover fallback —
234+
i.e. NOT measured from real chapter pages. Same affordance as
235+
the BROKEN_HANDLERS icon above (and distinct from the yellow
236+
outlier triangle on the score row, which means "measured but
237+
suspect"). Data contract + derivation:
238+
search_orchestrator.py:_quality_basis; helper qualityBasis at
239+
the top of this file handles pre-field JSON. */}
240+
{basisWarning && (
241+
<span
242+
className="shrink-0 inline-flex items-center"
243+
title={basisWarning}
244+
>
245+
<AlertTriangle
246+
className="w-3 h-3 text-red-500"
247+
aria-label={`${source.site} rating basis warning: ${basisWarning}`}
248+
/>
249+
</span>
250+
)}
211251
{formatLabel && (
212252
// Format chip from Phase H metadata. Lowercase font-mono lines up
213253
// with the site name visually; muted color keeps it secondary.

0 commit comments

Comments
 (0)