CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Research Synergy (ReSyn) is a Rust application for Literature Based Discovery (LBD). It aggregates academic papers from arXiv and InspireHEP, persists them to SurrealDB, constructs knowledge graphs from citation relationships, and visualizes them as interactive force-directed graphs.

Build & Run Commands

The workspace has multiple binaries. Always use cargo run --bin resyn with the appropriate subcommand — bare cargo run is ambiguous.

cargo build                  # Debug build (all crates)
cargo build --release        # Release build
cargo test                   # Run all tests
cargo test <test_name>       # Run a specific test
cargo test -- --nocapture    # Run tests with stdout visible
cargo check                  # Type-check without building
cargo fmt --all -- --check   # Check formatting
cargo clippy --all-targets --all-features  # Lint (CI runs with -Dwarnings)

Subcommands (cargo run --bin resyn -- <subcommand> [args]):

# Crawl
cargo run --bin resyn -- crawl --paper-id 2503.18887 --db surrealkv://./data
cargo run --bin resyn -- crawl --paper-id 2301.12345 --max-depth 2 --db surrealkv://./data
cargo run --bin resyn -- crawl --source inspirehep --paper-id 2301.12345 --db surrealkv://./data

# Analyze (NLP + optionally LLM + gap analysis)
cargo run --bin resyn -- analyze --db surrealkv://./data
cargo run --bin resyn -- analyze --db surrealkv://./data --llm-provider claude

# Export Louvain community graph to JSON (for external tooling, e.g. Kuramoto-LBD notebook)
cargo run --bin resyn -- export-louvain-graph --db surrealkv://./data --output graph.json
cargo run --bin resyn -- export-louvain-graph --db surrealkv://./data \
    --output research_synergy_pre2015.json \
    --published-before 2014-12-31 \
    --tfidf-top-n 50

# Serve web UI
cargo run --bin resyn -- serve --db surrealkv://./data

# Bulk-ingest papers from OpenAlex REST API (Phase RS-08)
# Default filter: ML+stat.ML+NeuralNet papers hosted on arXiv (~1.5M works)
# NOTE: OpenAlex itself does not need a key — the unauthenticated polite pool (mailto UA)
# still works (verified 2026-07-05). But `bulk_ingest::run` currently hard-exits when
# --api-key/OPENALEX_API_KEY is absent, so the flag is required by *this* CLI, not by the API.
cargo run --release --bin resyn -- bulk-ingest --db surrealkv://./data-openalex \
    --api-key "$OPENALEX_API_KEY"
# Physics corpus filter (Condensed matter + Statistical physics on arXiv):
cargo run --release --bin resyn -- bulk-ingest --db surrealkv://./data-physics \
    --api-key "$OPENALEX_API_KEY" \
    --filter "primary_location.source.id:S4306400194,concepts.id:C26873012|C121864883"

# Frontend (separate from the backend binary)
cd resyn-app && trunk serve

Architecture

The application pipeline: select data source → fetch seed paper → BFS crawl references → persist to SurrealDB → NLP + graph analytics + gap analysis → serve the results in a Leptos web UI (or export JSON for external tooling).

There is no top-level src/. This is a four-crate Cargo workspace (edition 2024, resolver = "2"):

Crate	Role
resyn-core	All domain logic. WASM-safe by default; server-only pieces are behind the ssr feature.
resyn-app	Leptos 0.8 frontend (cdylib + rlib); csr and ssr features.
resyn-server	The resyn binary (clap subcommands) + axum/leptos_axum SSR host.
resyn-worker	WASM web-worker crate (gloo-worker) running the force-layout simulation off the main thread.

resyn-core/src/ — domain layers

Always available (WASM-safe): analysis, data_processing, datamodels, error, gap_analysis, graph_analytics, nlp, utils, validation. Behind ssr: data_aggregation, database, llm. (See resyn-core/src/lib.rs — the feature gating is load-bearing: single-clustering pulls in rayon, which is not WASM-compatible.)

• data_aggregation/ (ssr) — source abstraction and implementations:

  • traits.rs — PaperSource async trait (fetch_paper, fetch_references, fetch_citing_papers default no-op, source_name)
  • arxiv_source.rs / arxiv_api.rs / arxiv_utils.rs / html_parser.rs — arXiv metadata + HTML bibliography scraping; arxiv_utils.rs holds the BFS crawler (recursive_paper_search_by_references, takes &mut dyn PaperSource)
  • inspirehep_api.rs — InspireHepClient (350ms rate limit)
  • semantic_scholar_api.rs — S2 source, the only one implementing forward citations (--bidirectional)
  • openalex_source.rs / openalex_bulk.rs — OpenAlex single-paper source and the OpenAlexBulkLoader behind bulk-ingest
  • chained_source.rs — comma-separated source chaining (--source arxiv,inspirehep)
  • rate_limiter.rs (governor), text_extractor.rs, search_query_handler.rs

• database/ (ssr) — SurrealDB v3 persistence: client.rs (connect, connect_memory, connect_local), schema.rs (versioned apply_migration_N + schema_migrations table), queries.rs (PaperRepository), crawl_queue.rs (persistent BFS frontier).

• datamodels/ — paper.rs (Paper, Reference, Link, Journal, DataSource; transient citing_papers), plus analysis.rs, community.rs, community_graph.rs (export-only CommunityGraph/ExportedNode/ExportedEdge/LouvainParams), enrichment.rs, extraction.rs, gap_finding.rs, graph_metrics.rs, llm_annotation.rs, progress.rs, similarity.rs.

• data_processing/ — graph_creation.rs converts Vec<Paper> into a petgraph::StableGraph<Paper, f32, Directed> from citation relationships; dedups via strip_version_suffix().

• nlp/ — preprocessing.rs (tokenization, stop-words) and tfidf.rs (TF-IDF / c-TF-IDF vectors used for community labels and similarity).

• graph_analytics/ — community.rs (Louvain via single-clustering, ssr-gated), pagerank.rs, betweenness.rs.

• gap_analysis/ — literature-based-discovery gap finding: abc_bridge.rs (Swanson ABC), similarity.rs, contradiction.rs, output.rs (ssr).

• analysis/ — aggregation.rs (roll-up of per-paper analysis into corpus-level results) and highlight.rs.

• llm/ (ssr) — provider abstraction for semantic extraction: traits.rs, claude.rs, ollama.rs, noop.rs, plus prompt.rs / gap_prompt.rs.

• error.rs — ResynError enum: ArxivApi, HtmlDownload, HttpRequest (ssr only), PaperNotFound, InvalidPaperId, NoArxivLink, InspireHepApi, Database, LlmApi, SemanticScholarApi, OpenAlexApi. Implements Display, Error, From<reqwest::Error>.

• validation.rs — arXiv ID validation (new YYMM.NNNNN and old category/NNNNNNN, optional version suffix) and URL validation.

• utils.rs — strip_version_suffix(), create_http_client() (30s timeout).

resyn-server/src/

main.rs (the single #[tokio::main]) → commands/: crawl.rs, analyze.rs, export.rs, bulk_ingest.rs, serve.rs — one module per CLI subcommand.

resyn-app/src/

Leptos SSR/CSR app: pages/ (dashboard, graph, papers, gaps, methods, open_problems), components/ (search_bar, graph_controls, analysis_controls, crawl_progress, gap_card, heatmap), layout/ (sidebar, drawer), server_fns/ (leptos server functions per domain: papers, graph, gaps, community, similarity, metrics, methods, problems, analysis), and graph/ — the renderer stack: canvas_renderer.rs, webgl_renderer.rs, lod.rs, label_collision.rs, viewport_fit.rs, kmeans.rs, interaction.rs, layout_state.rs, worker_bridge.rs.

resyn-worker/src/

forces.rs + barnes_hut.rs force-directed layout, driven from bin/resyn_worker.rs as a gloo web worker.

Key Dependencies

Workspace-level (Cargo.toml [workspace.dependencies]) unless noted.

• leptos / leptos_router / leptos_meta / leptos-use 0.8 — the web UI (there is no desktop GUI; egui/eframe/fdg were removed)
• leptos_axum + axum + tower-http — SSR host and static serving
• tokio — async runtime (single #[tokio::main] in resyn-server/src/main.rs)
• surrealdb v3 — embedded document-graph DB (kv-mem for tests, kv-surrealkv for local persistence)
• petgraph 0.7 (default-features off; graphmap/stable_graph/matrix_graph/serde-1) — graph data structures; re-exported from resyn-core so downstream crates don't depend on it directly
• single-clustering 0.6 — Louvain; ssr-only because it pulls rayon (not WASM-safe)
• arxiv-rs — arXiv API client
• reqwest / scraper / async-trait — HTTP, HTML parsing, async traits (all ssr-gated in core)
• governor — rate limiting
• wasm-bindgen / web-sys / js-sys / gloo-worker / wasm-bindgen-futures — WASM frontend + worker plumbing (WebGL2 + Canvas2D features)
• clap (derive + env) — CLI parsing; dotenvy — .env loading
• stop-words, sha2, chrono, rand, anyhow, regex
• tracing / tracing-subscriber — structured logging
• serde / serde_json — serialization for all data models
• wiremock, tokio-test, http, tower (dev) — HTTP mocking and server test harness

Data Flow Details

• arXiv source: Each paper requires two HTTP requests — one to arXiv API (metadata) and one to the arXiv HTML page (bibliography references). Both go through ArxivHTMLDownloader rate limiting (default 3s).
• InspireHEP source: Single API call per paper returns metadata + references with direct arXiv eprint IDs (no HTML scraping needed). Rate limit: 350ms between requests.
• Reference extraction from arXiv HTML parses <span class="ltx_bibblock"> elements. Titles extracted from <em> tags when present, falling back to comma-splitting.
• The BFS crawler (recursive_paper_search_by_references) accepts &mut dyn PaperSource, enabling source-agnostic crawling.
• Only arXiv-to-arXiv citation edges are followed. References to Nature/PhysRev/other journals are stored but not crawled.
• SurrealDB persistence: papers stored as paper:⟨arxiv_id⟩ records, citations as cites relation edges. Schema is auto-initialized on connection.

CLI Subcommands

Run cargo run --bin resyn -- <subcommand> --help for full argument lists. Key arguments per subcommand:

crawl

Argument	Default	Description
--paper-id / -p	2503.18887	arXiv seed paper ID
--max-depth / -d	3	BFS crawl depth
--rate-limit-secs / -r	3	Rate limit between requests (seconds)
--source	arxiv	Data source: arxiv, inspirehep, semantic_scholar, or comma-separated chain
--bidirectional	false	Fetch forward citations (citing papers) in addition to references; semantic_scholar source only
--max-forward-citations	500	Cap on citing papers fetched per paper when --bidirectional is set
--db	surrealkv://./data	DB connection string

analyze

Argument	Default	Description
--db	surrealkv://./data	DB connection string
--llm-provider	none	LLM for semantic extraction: claude, ollama, noop
--force	false	Re-analyze already-analyzed papers

export-louvain-graph — exports the Louvain community graph to JSON for external tooling

Argument	Default	Description
--db	surrealkv://./data	DB connection string
--output	(required)	Output JSON file path
--published-before	none	ISO-8601 date cutoff e.g. 2014-12-31 (inclusive, lexicographic)
--tfidf-top-n	50	Max TF-IDF terms per node

Output schema: {louvain_params, corpus_fingerprint, nodes: [{id, community_id, tfidf_vec}], communities: [{community_id, size, tfidf_vec}], edges: [{src, dst, weight}]}. "Other" community papers (community_id = u32::MAX-1) are excluded. Edge weight is 1.0 (uniform). The communities field carries per-community c-TF-IDF vectors for EXP-RS-07 (Sheaves-LBD); old consumers ignore it via serde default. Requires communities to be computed first (analyze runs community detection). See resyn-core/src/datamodels/community_graph.rs for the full type definitions.

Typical Kuramoto-LBD v03 workflow:

# 1. Crawl a corpus (one-time)
cargo run --bin resyn -- crawl --paper-id <seed> --db surrealkv://./data --max-depth 3

# 2. Run analysis (NLP + community detection)
cargo run --bin resyn -- analyze --db surrealkv://./data

# 3. Export for the Python notebook
cargo run --bin resyn -- export-louvain-graph \
    --db surrealkv://./data \
    --output prototypes/data/research_synergy_pre2015.json \
    --published-before 2014-12-31 \
    --tfidf-top-n 50

# 4. Run the Python LBD prototypes in ./prototypes/ (venv: prototypes/.venv)

Research thread wiring (garden)

Besides the product track, this repo hosts the Dynamical LBD (Gen-4) research thread, managed by the vault at ~/Repositories/garden. For research phases (exploratory LBD experiments, EXP-RS-*), follow the vault's research structure on top of the GSD scaffolding:

• Thread state (read by the vault's /cartographer): .planning/research/THREAD.md (hard core, live hypotheses, kill criteria, claims) + .planning/research/CONVENTIONS.md (append-only convention lock — honor it; append, never edit). Update THREAD.md same-day after every experiment run.
• Experiment registry: EXP-RS-* rows in the vault's wiki/meta/agentic-experiments-research.md. Predictions are pre-registered there before running — never adjusted post-hoc.
• Baseline rule: no dynamical method is interesting until it beats the brute-force baseline (vault: wiki/concepts/brute-force-lbd-baseline.md) on the shared 10-pair benchmark.
• Skills: /consult research "<question>" — read-only vault lookups; /commission --research — independent falsification of a claim/verdict before accepting it; /chronicler at session end (self-invoking; auto-files lessons to the vault).
• Cartographer channel: .cartographer-notes.md (gitignored) — proposals from vault research reviews land there; go/kill/pivot decisions are the human's, recorded in the vault hypothesis-ledger.
• Prototype workspace (in-repo): ALL research/LBD implementation — Python + Rust prototypes, scripts, data/, figures/ — lives in-repo at ./prototypes/ (venv rebuilt from prototypes/requirements-lock.txt, gitignored; run scripts via prototypes/.venv/bin/python). The garden holds only management/ideas/decisions (thread state in .planning/, brainstorms, conventions) — no implementation. (Consolidated 2026-07-07, human decision; supersedes the old vault-prototypes split — see CONVENTIONS.md C-21.)

Important Notes

• arXiv rate limiting: ArxivHTMLDownloader enforces configurable delays (default 3s) between requests using tokio::time::sleep. Violating this causes request blocks.
• InspireHEP rate limiting: InspireHepClient enforces 350ms between requests.
• Bidirectional crawl mode (--bidirectional, S2 only): Fetches both backward (references) and forward (citations) for each paper from the Semantic Scholar /citations endpoint and enqueues newly-discovered citing papers in the BFS queue. Forward-citation edges are persisted via PaperRepository::upsert_inverse_citations_batch with the correct direction (citing -> cited). The --max-forward-citations N flag caps pagination per paper (default 500). Use this mode for pre-2015 cond-mat seeds where S2 backward citations have externalIds: null (the graph would otherwise terminate after one hop).
• OpenAlex bulk ingest (bulk-ingest subcommand): Ingests arXiv-indexed papers in bulk from the OpenAlex REST API (polite pool, ~10 req/s). Auth: OpenAlex's unauthenticated polite pool (mailto UA) still works — no key needed by the API (verified 2026-07-05; the .env key is a commented-out placeholder). However bulk_ingest::run (resyn-server/src/commands/bulk_ingest.rs) still process::exit(1)s when --api-key/OPENALEX_API_KEY is missing, and sends it as an Authorization: Bearer header, so the flag is a requirement of this CLI rather than of OpenAlex. Skips per-paper HTTP calls entirely. upsert_citations_batch does not check target paper existence (dangling edges OK). arxiv_id() extracts arXiv IDs from both 10.48550/arxiv.* DOIs and locations[].landing_page_url matching arxiv.org/abs/. Concept IDs: C154945302=ML, C121332964=stat.ML, C41008148=Neural Networks, C26873012=Condensed matter physics, C121864883=Statistical physics.
• Paper IDs may have version suffixes (e.g., "2301.12345v2") — these are stripped via utils::strip_version_suffix() during crawl dedup, graph construction, and DB upserts.
• Rust edition 2024, stable toolchain (pinned via rust-toolchain.toml).
• Single async runtime: main.rs has the only #[tokio::main]. All API/HTML functions are async fn.
• Error handling uses ResynError with ? propagation. The crawler logs warnings for individual failures and continues.
• SurrealDB kv-mem feature compiles as a Rust dependency — no external server needed for tests.
• CI runs fmt check, clippy with -Dwarnings, tests, and coverage via tarpaulin.

Testing

• 46 tests total: 32 unit tests + 8 integration tests + 6 database tests
  • Unit: paper, graph_creation, arxiv_utils, search_query_handler, validation, utils, inspirehep_api deserialization/conversion
  • Integration: wiremock-based arXiv HTML parsing (3), InspireHEP API mocking (5)
  • Database: SurrealDB in-memory (upsert, idempotent, exists, version dedup, citations, graph traversal)
• All DB tests use connect_memory() — no external DB required
• ArxivHTMLDownloader::with_rate_limit(Duration::from_millis(0)) and InspireHepClient::with_rate_limit(Duration::from_millis(0)) disable rate limiting in tests

Known pre-existing failures — do NOT chase these as regressions: 7 schema-migration tests fail and cargo clippy --all-targets --all-features reports 4 errors. Both predate any current work (Phase 24-01, migration 12) and are recorded as known-failing in the vault registry. A red suite on a fresh checkout is expected; only failures beyond these are yours.