You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
description: Use when the user wants to ingest one or more web URLs into ScholarAIO, convert rendered web content into Markdown through qt-web-extractor, or turn an online page/PDF into a normal document-ingest workflow.
4
+
---
5
+
6
+
# Ingest Web Links
7
+
8
+
Use this skill when the user already has one or more URLs and wants them pulled into the ScholarAIO library as document-style records.
9
+
10
+
## When to Use
11
+
12
+
Use this skill when the user wants to:
13
+
14
+
- ingest a webpage directly into the local knowledge base
15
+
- ingest an online PDF or report from a URL
16
+
- capture technical documentation, standards, manuals, or web articles as `document` items
17
+
- route rendered web content through the normal ScholarAIO ingest/index flow
18
+
19
+
Do not use this skill when:
20
+
21
+
- the task is only to search the local library; use `search`
22
+
- the task is mainly about arXiv preprints; use `arxiv`
23
+
- the user only wants broad web discovery without ingesting; use external web search first, then come back if ingestion is desired
24
+
25
+
## Core Workflow
26
+
27
+
### 1. Confirm the source shape
28
+
29
+
- If the user already provides URL(s), go straight to `scholaraio ingest-link`
30
+
- If the user only has a topic but no URL, discover URLs first through external web search, then ingest selected results
Copy file name to clipboardExpand all lines: AGENTS.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -48,6 +48,7 @@ The right mental model is to treat skills as "reusable workflows": when the user
48
48
Knowledge base management:
49
49
-`search` - When the user wants to find papers, search authors, or run keyword / semantic / hybrid retrieval, start with this skill.
50
50
-`arxiv` - When the user wants to browse arXiv preprints, search arXiv directly, or fetch a preprint PDF into inbox or ingest, use this skill.
51
+
-`ingest-link` - When the user wants to ingest one or more rendered web URLs or online PDFs into the library through `qt-web-extractor`, use this skill.
51
52
-`show` - When the user wants to read paper metadata, abstract, conclusion, or full text, use this skill for progressive L1-L4 loading.
52
53
-`enrich` - When the user wants to add TOC, conclusion, abstract, citation counts, or other enrichment fields, use this skill.
53
54
-`ingest` - When the user wants to process inbox items, ingest PDF / Office / Markdown files, and rebuild indexes, use this skill.
- Enrichment and repair: `enrich-toc`, `enrich-l3`, `backfill-abstract`, `refetch`, `repair`
197
-
- Data maintenance: `attach-pdf`
198
+
- Data maintenance: `attach-pdf`, `ingest-link`
198
199
- Workspace: `ws` (subcommands such as `init`, `add`, `remove`, `show`, `search`, `export`, and more)
199
200
- Proceedings: `proceedings` (`apply-split`, `build-clean-candidates`, `apply-clean`) and `fsearch --scope proceedings`
200
201
- External and scientific runtime: `arxiv`, `toolref`, `insights`, `style`, `document`
@@ -387,7 +388,7 @@ data/explore/<name>/
387
388
### `sources/` Abstraction Layer
388
389
389
390
`papers.py` is the path-helper layer for the local library under `data/papers/`, and modules use it directly to iterate paper directories and read `meta.json`.
390
-
`sources/` holds external-source adapters such as arXiv, Endnote, and Zotero.
391
+
`sources/` holds external-source adapters such as arXiv, Endnote, Zotero, and webtools-style HTTP backends.
Copy file name to clipboardExpand all lines: CHANGELOG.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,11 +13,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
13
13
-**Academic writing router** ([#55](https://github.com/ZimoLiao/scholaraio/issues/55)): Added the `academic-writing` skill as a stable top-level entry point that routes users by deliverable and writing stage instead of forcing them to guess among multiple writing skills
14
14
-**Deliverable-first writing workflows** ([#55](https://github.com/ZimoLiao/scholaraio/issues/55)): Added lightweight `poster` and `technical-report` skills so conference posters, poster-style summaries, topic reports, and research briefings are first-class workflows rather than implicit combinations of lower-level skills
15
15
-**Writing workflow regression coverage**: Added tests for skill frontmatter validity, router references, approximate host-style skill selection, and 11 rounds of documentation-alignment checks across docs, agent instructions, and marketplace metadata
16
+
-**Rendered web URL ingestion** ([#52](https://github.com/ZimoLiao/scholaraio/issues/52)): Added the native `scholaraio ingest-link` CLI, `ingest-link` skill, and `sources.webtools` connector flow so webpages and online PDFs can be ingested through an external `qt-web-extractor` daemon while preserving provenance fields such as `source_url`, `source_type`, `extracted_at`, and `extraction_method`
16
17
17
18
### Fixed
18
19
19
20
-**macOS semantic/unified search crash** ([#65](https://github.com/ZimoLiao/scholaraio/issues/65)): main-library and explore semantic search now embed and normalize the query before loading or searching FAISS indexes, avoiding a known `faiss` / `sentence-transformers` import-order segfault pattern on macOS while preserving existing ranking behavior
20
21
-**Academic writing docs alignment** ([#55](https://github.com/ZimoLiao/scholaraio/issues/55)): Synchronized `docs/guide/writing.md`, `README.md`, `README_CN.md`, `docs/index.md`, `AGENTS.md`, `AGENTS_CN.md`, `CLAUDE.md`, and `clawhub.yaml` around a router-first writing model so poster/report workflows and the academic-writing entry point are discoverable consistently across user and agent surfaces
22
+
-**`ingest-link` reliability and isolation** ([#52](https://github.com/ZimoLiao/scholaraio/issues/52)): URL ingest now preserves extractor PDF autodetect unless `--pdf` is explicitly requested, isolates both temporary inboxes from the real library, skips only failed URLs in multi-link batches, keeps warning-bearing extractions with usable text, retries transient extraction failures with exponential backoff, and avoids overlong fallback filenames for title-less URLs
Copy file name to clipboardExpand all lines: CLAUDE.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -48,6 +48,7 @@ The right mental model is to treat skills as "reusable workflows": when the user
48
48
Knowledge base management:
49
49
-`search` - When the user wants to find papers, search authors, or run keyword / semantic / hybrid retrieval, start with this skill.
50
50
-`arxiv` - When the user wants to browse arXiv preprints, search arXiv directly, or fetch a preprint PDF into inbox or ingest, use this skill.
51
+
-`ingest-link` - When the user wants to ingest one or more rendered web URLs or online PDFs into the library through `qt-web-extractor`, use this skill.
51
52
-`show` - When the user wants to read paper metadata, abstract, conclusion, or full text, use this skill for progressive L1-L4 loading.
52
53
-`enrich` - When the user wants to add TOC, conclusion, abstract, citation counts, or other enrichment fields, use this skill.
53
54
-`ingest` - When the user wants to process inbox items, ingest PDF / Office / Markdown files, and rebuild indexes, use this skill.
- Enrichment and repair: `enrich-toc`, `enrich-l3`, `backfill-abstract`, `refetch`, `repair`
197
-
- Data maintenance: `attach-pdf`
198
+
- Data maintenance: `attach-pdf`, `ingest-link`
198
199
- Workspace: `ws` (subcommands such as `init`, `add`, `remove`, `show`, `search`, `export`, and more)
199
200
- Proceedings: `proceedings` (`apply-split`, `build-clean-candidates`, `apply-clean`) and `fsearch --scope proceedings`
200
201
- External and scientific runtime: `arxiv`, `toolref`, `insights`, `style`, `document`
@@ -387,7 +388,7 @@ data/explore/<name>/
387
388
### `sources/` Abstraction Layer
388
389
389
390
`papers.py` is the path-helper layer for the local library under `data/papers/`, and modules use it directly to iterate paper directories and read `meta.json`.
390
-
`sources/` holds external-source adapters such as arXiv, Endnote, and Zotero.
391
+
`sources/` holds external-source adapters such as arXiv, Endnote, Zotero, and webtools-style HTTP backends.
Copy file name to clipboardExpand all lines: docs/guide/cli-reference.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,6 +35,7 @@ scholaraio top-cited
35
35
36
36
```text
37
37
scholaraio pipeline [preset]
38
+
scholaraio ingest-link <url> [<url> ...]
38
39
scholaraio enrich-toc
39
40
scholaraio enrich-l3
40
41
scholaraio backfill-abstract
@@ -44,6 +45,7 @@ scholaraio attach-pdf
44
45
```
45
46
46
47
-`pipeline` is the main composable ingest entrypoint.
48
+
-`ingest-link` pulls one or more rendered web URLs or online PDFs through an external `qt-web-extractor` service and routes them into the existing document ingest flow.
47
49
- Current preset values are `full`, `ingest`, `enrich`, and `reindex`.
48
50
- Run `scholaraio pipeline --help` for pipeline options such as `--steps`, `--dry-run`, `--no-api`, and `--rebuild`.
ScholarAIO is agent-first: users talk to an agent, and the agent orchestrates local ScholarAIO skills.
4
-
If you also want live web search/extraction, you can integrate [AnterCreeper/claude-webtools](https://github.com/AnterCreeper/claude-webtools) as an external capability layer.
4
+
If you also want live web search/extraction, ScholarAIO can integrate the same backend daemons used by[AnterCreeper/claude-webtools](https://github.com/AnterCreeper/claude-webtools) as an external capability layer.
5
5
6
6
## When to use this
7
7
8
8
- You need **internet discovery** (news, latest announcements, online docs) in addition to the local paper KB.
9
9
- You want the agent to combine:
10
10
- ScholarAIO local retrieval (`/scholaraio:search`, `/scholaraio:show`, etc.)
11
-
- external web lookup from claude-webtools.
11
+
- external web lookup from `GUILessBingSearch` / `qt-web-extractor`.
12
+
13
+
## Native ScholarAIO entrypoint
14
+
15
+
ScholarAIO now provides a native URL-ingest command:
16
+
17
+
```bash
18
+
scholaraio ingest-link https://example.com/page
19
+
```
20
+
21
+
This command:
22
+
23
+
1. Calls a running `qt-web-extractor` service.
24
+
2. Pulls rendered page content instead of only raw HTML source.
25
+
3. Writes extracted Markdown into a temporary document inbox.
26
+
4. Reuses the existing ScholarAIO document ingest pipeline.
27
+
28
+
In practice, this means `scholaraio ingest-link` can ingest:
29
+
30
+
- JavaScript-rendered pages that a plain HTTP fetch would miss
31
+
- online PDFs and report URLs
32
+
- technical docs, manuals, standards, and web articles as normal `document` records
33
+
34
+
The current command expects `qt-web-extractor` to be reachable through:
35
+
36
+
-`WEBEXTRACT_URL` if set
37
+
- otherwise `http://127.0.0.1:8766`
12
38
13
39
## Recommended setup
14
40
15
-
1. Install and configure `claude-webtools` by following its README.
41
+
1. Install and configure the backend services:
42
+
-`qt-web-extractor` for rendered URL/PDF extraction
43
+
- optional `GUILessBingSearch` for search-first workflows
16
44
2. Keep ScholarAIO as the authoritative local knowledge pipeline (ingest/index/enrich).
17
45
3. In agent workflows:
18
46
- use ScholarAIO first for reproducible local evidence;
19
47
- use webtools only when freshness or external coverage is required.
20
48
49
+
`qt-web-extractor` is an external daemon, not a built-in ScholarAIO fetcher. ScholarAIO delegates browser rendering to that service and then continues with its own ingest pipeline.
50
+
21
51
## Operational guidelines
22
52
23
53
- Prefer local KB evidence for stable academic claims.
24
54
- For time-sensitive facts, cross-check via webtools and record access date.
25
55
- When webtools is unavailable, agent should degrade gracefully to local-only ScholarAIO workflows.
56
+
- Prefer the default automatic URL handling first; use `scholaraio ingest-link --pdf <url>` only when a PDF URL needs an explicit hint.
0 commit comments