feat: 0.3.0 - typed result objects, list/nested extraction, pooled browser

Picks up the three deferred capability items from 0.2.0.

Typed results (scrapo.results):
- scrape() returns ScrapeResult, crawl() returns CrawlResult, result.extraction
  is an ExtractionView. Pydantic models: attribute access, validation,
  model_dump(). They also keep dict-style read access (result["key"],
  result.get(), "key" in result) so existing 0.1/0.2 code is unaffected;
  isinstance(result, dict) is no longer true. MCP server serializes via
  model_dump(mode="json").

List / nested extraction:
- schema fields typed list[BaseModel] are extracted as repeated DOM elements:
  the LLM returns {"field": {"__list__": "<repeating el>", "<sub>": "<rel sel>"}},
  verified against the live page, cached, and replayed with zero tokens like
  scalar fields. New scrapo.extract.schema.list_fields() does the detection and
  feeds a hint into the prompt.

Browser pooling (scrapo.access.browser_pool.BrowserPool):
- a TierRouter lazily launches one headless Chromium and reuses it across
  fetches; proxy settings move to the context level so one browser serves
  rotating proxies. crawl() shares one router across all pages instead of cold-
  launching per page. TierRouter.aclose() tears it down; scrape() closes the
  router it creates; scrape() gained a router= kwarg for explicit reuse.
- playwright-stealth is applied to the page before navigation (was racing a
  context event) and tries both the old and new plugin APIs.

Tests: typed-result and crawl-result assertions, list extraction (LLM path +
cache replay), list_fields detection, browser-pool/router teardown (94 tests,
still fully offline). ruff + mypy --strict clean. Docs and CHANGELOG updated; no
em-dashes in Markdown. Bump to 0.3.0.

Author: vikast908

CHANGELOG.md

@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-05-10
+
+Capability release: typed results, list/nested extraction, and a reused browser.
+
+### Added
+
+- **Typed result objects** (`scrapo.results`): `scrape()` returns `ScrapeResult`, `crawl()` returns `CrawlResult`, and `extraction` on a result is an `ExtractionView`. They are Pydantic models, so you get attribute access (`result.markdown`), validation, and `result.model_dump()` for serialization. They also support `result["key"]`, `result.get("key", default)`, and `"key" in result` so code written against the 0.1/0.2 dict shape keeps working unchanged.
+- **List / nested extraction**: schema fields typed `list[SomeBaseModel]` are now extracted as repeated DOM elements. The LLM returns a container selector plus per-subfield selectors (`{"products": {"__list__": "ul.grid > li", "name": "h3", "price": ".price"}}`), those are verified against the live page, cached, and replayed on later runs with zero LLM tokens, exactly like scalar fields. `scrapo.extract.schema.list_fields()` exposes the detection.
+- **Browser-context pooling** (`scrapo.access.browser_pool.BrowserPool`): a `TierRouter` now lazily launches one Chromium and reuses it across fetches (proxy settings move to the context level so a single browser serves rotating proxies). A crawl no longer cold-launches a browser per page. `TierRouter.aclose()` tears it down; `scrape()` closes the router it creates, and `crawl()` shares one router across all pages. `scrape()` gained a `router=` keyword for callers that want to reuse one explicitly.
+- The flaky `playwright-stealth` integration is applied to the page before navigation (instead of via a context event that raced the first page) and tries both the old and new plugin entry points.
+
+### Changed
+
+- `scrape()` / `crawl()` return Pydantic models instead of plain `dict`. Dict-style read access still works; `isinstance(result, dict)` does not. The MCP server serializes results with `model_dump(mode="json")`.
+
 ## [0.2.0] - 2026-05-10
 
 Hardening release: makes the "cost-aware" and "production crawling" claims real,
@@ -73,6 +88,7 @@ Initial public release.
 - The `robots.txt` gate is opt-in: set `SCRAPO_RESPECT_ROBOTS=1` (or `Config(respect_robots=True)`) to enable it. You are responsible for complying with each site's terms of use and applicable law.
 - Alpha status: the public API and core subsystems are stable, but the T4 agent driver, full action caching, an S3 snapshot adapter, and a hosted control plane are intentionally lightweight or not yet implemented.
 
-[Unreleased]: https://github.com/vikast908/Scrapo/compare/v0.2.0...HEAD
+[Unreleased]: https://github.com/vikast908/Scrapo/compare/v0.3.0...HEAD
+[0.3.0]: https://github.com/vikast908/Scrapo/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/vikast908/Scrapo/compare/v0.1.0...v0.2.0
 [0.1.0]: https://github.com/vikast908/Scrapo/releases/tag/v0.1.0

README.md

@@ -107,13 +107,14 @@ Plus a feature nobody else ships: **deterministic replay** of every fetch, so ex
 
 ```
 scrapo/
-├── access/      # (1) 5-tier router + Bright Data / Oxylabs / Scrapfly / Zyte
-├── extract/     # (2) hybrid selector + LLM, model pinning, cost-aware budget
+├── access/      # (1) 5-tier router + pooled browser + Bright Data / Oxylabs / Scrapfly / Zyte
+├── extract/     # (2) hybrid selector + LLM (scalar & list fields), model pinning, cost-aware budget
 ├── shape/       # (3) selectolax markdown + heading chunker
 ├── replay/      # (4) snapshot store + field-level diff
 ├── policy/      # (5) robots, PII (flag or redact), geo, append-only audit
 ├── crawl/       # persistent SQLite queue + async scheduler
 ├── agent/       # (6) MCP server + tool schemas
+├── results.py   # typed ScrapeResult / CrawlResult / ExtractionView
 ├── security.py  # SSRF guard for fetch targets
 ├── _db.py       # tuned SQLite connections (WAL, busy timeout)
 ├── logging.py   # structlog setup for the CLI / MCP server
@@ -139,33 +140,38 @@ playwright install chromium
 import asyncio, scrapo
 
 async def main():
-    res = await scrapo.scrape("https://example.com/")
-    print(res["markdown"])
-    print("run_id:", res["run_id"])
+    res = await scrapo.scrape("https://example.com/")   # res is a typed ScrapeResult
+    print(res.markdown)
+    print("run_id:", res.run_id)
+    # res["markdown"] / res.get("status") still work too (back-compat with the 0.1 dict)
 
 asyncio.run(main())
 ```
 
-### 2. Typed extraction (LLM once, selectors forever)
+### 2. Typed extraction, including lists (LLM once, selectors forever)
 
 ```python
 import asyncio, scrapo
 from pydantic import BaseModel
 
-class Product(BaseModel):
+class Offer(BaseModel):
     name: str
     price: str
 
+class Listing(BaseModel):
+    page_title: str
+    offers: list[Offer] = []        # array fields become repeated-element extraction
+
 async def main():
-    res = await scrapo.scrape("https://example.com/widget", schema=Product)
-    print(res["extraction"]["data"])      # {'name': 'Widget Pro', 'price': '$42'}
-    print(res["extraction"]["method"])    # 'llm' on the first run, 'selector' after
-    print(res["cost_usd"])                # 0.0 once selectors are cached
+    res = await scrapo.scrape("https://example.com/shop", schema=Listing)
+    print(res.extraction.data)      # {'page_title': '...', 'offers': [{'name': ..., 'price': ...}, ...]}
+    print(res.extraction.method)    # 'llm' on the first run, 'selector' after
+    print(res.cost_usd)             # 0.0 once selectors are cached
 
 asyncio.run(main())
 ```
 
-> First call uses the LLM and **caches the selectors it learns** (keyed by host + schema). Every subsequent call against that host + schema uses cached selectors and **zero LLM tokens**. When the layout drifts, validation fails, Scrapo falls back to the LLM, re-derives selectors, and self-heals; a cache entry that keeps failing is evicted automatically.
+> First call uses the LLM and **caches the selectors it learns** (keyed by host + schema; for `list[Model]` fields it caches a container selector plus per-subfield selectors). Every subsequent call against that host + schema uses cached selectors and **zero LLM tokens**. When the layout drifts, validation fails, Scrapo falls back to the LLM, re-derives selectors, and self-heals; a cache entry that keeps failing is evicted automatically.
 
 ### 3. Recursive crawl
 
@@ -205,15 +211,15 @@ Escalation triggers: Cloudflare/Akamai/PerimeterX/DataDome/Distil fingerprints,
 </details>
 
 <details>
-<summary><b>Hybrid selector + LLM extractor</b></summary>
+<summary><b>Hybrid selector + LLM extractor (scalar and list fields)</b></summary>
 
 ```
 cache hit + validates    -> return (method=selector, llm_calls=0, cost_usd=0)
 miss / fail / over budget -> LLM with schema -> validate -> verify + persist selectors -> return (method=llm)
 repeated cache failures   -> evict the stale entry, re-derive next run
 ```
 
-The LLM is asked to return both the JSON payload *and* CSS selectors per field. Returned selectors are verified against the live HTML before being cached, so a hallucinated selector never poisons the cache. The cache is keyed by host (not registered domain), so `blog.example.com` and `shop.example.com` never collide.
+The LLM is asked to return both the JSON payload *and* CSS selectors per field. A scalar field gets a string selector; a `list[Model]` field gets `{"__list__": "<repeating element>", "<subfield>": "<selector relative to it>", ...}`, which Scrapo applies as `tree.css(container)` then per-subfield extraction inside each match. Returned selectors are verified against the live HTML before being cached, so a hallucinated selector never poisons the cache. The cache is keyed by host (not registered domain), so `blog.example.com` and `shop.example.com` never collide.
 
 </details>
 
@@ -283,6 +289,7 @@ diff 9f3e1c...  vs  abc123...
 - **SSRF guard.** Every fetch target is checked before a request goes out; loopback, link-local (including `169.254.169.254`), private RFC 1918 / ULA ranges, and well-known local hostnames are refused. Set `allow_private_hosts=True` (or `SCRAPO_ALLOW_PRIVATE_HOSTS=1`) for internal scraping. Crawl link discovery applies the same filter and skips obvious binary URLs.
 - **Bounded HTTP retries.** Transient `429 / 5xx` and transport errors are retried with exponential backoff and jitter before the router escalates to a heavier tier (`SCRAPO_HTTP_RETRIES`, default `2`).
 - **Concurrency-safe storage.** All SQLite stores (replay, selector cache, crawl queue) open in WAL mode with a busy timeout, so concurrent crawl workers do not trip over each other.
+- **Browser reuse.** A `TierRouter` launches one headless Chromium lazily and reuses it across fetches (proxy applied per context), so a crawl is not paying a cold browser launch per page. `TierRouter.aclose()` tears it down; `scrape()` and `crawl()` handle that for you.
 - **Cost accounting.** LLM cost is computed per call, recorded on the run, and enforceable via `Budget(max_llm_calls=..., max_cost_usd=...)`.
 - **PII handling.** Flag PII in the audit log (`SCRAPO_PII_FILTER=1`), or redact it from the stored snapshot, markdown, and chunks (`SCRAPO_REDACT_SNAPSHOTS=1`).
 - **Local UI hardening.** `scrapo serve` binds `127.0.0.1` by default, validates the `Host` header against an allowlist (anti DNS-rebinding), serializes scrapes, and warns loudly if you bind a public interface.
@@ -436,7 +443,7 @@ Issues and PRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the dev setup
 
 ## Project status
 
-Alpha. The public API (`scrape`, `extract`, `crawl`) is stable; tier escalation, model pinning, replay schema, and the MCP tool surface are stable. Parts that are intentionally lightweight today and slated for hardening: T4 agent driver, list/nested extraction, browser-context pooling, full Stagehand-style action caching, S3 snapshot adapter, hosted control plane.
+Alpha. The public API (`scrape`, `extract`, `crawl`) is stable; tier escalation, model pinning, replay schema, typed results, list extraction, and the MCP tool surface are stable. Parts that are intentionally lightweight today and slated for hardening: a batteries-included T4 agent driver, full Stagehand-style action caching, in-browser request interception, pagination/sitemap following, content-type routing (PDF/JSON/RSS), an S3 snapshot adapter, and a hosted control plane.
 
 See [CHANGELOG.md](CHANGELOG.md) for release notes.
 

layman.md

@@ -22,7 +22,7 @@ If you have ever copied information off a web page by hand, or wished a program
 ## What's possible
 
 - Get clean, readable text out of most public web pages.
-- Get structured data (the specific fields you define) out of pages, reliably and repeatably.
+- Get structured data (the specific fields you define) out of pages, reliably and repeatably, including lists (every product on a listing page, every row of a table) when you describe them as a list of records.
 - Scrape JavaScript-heavy pages that do not work with a plain download.
 - Get past light anti-bot defenses on your own; get past tougher ones by plugging in a commercial proxy service (Bright Data, Oxylabs, Scrapfly, and Zyte are supported out of the box).
 - Crawl a whole site: follow links automatically, with limits on depth and page count, skipping duplicates.
@@ -40,9 +40,8 @@ If you have ever copied information off a web page by hand, or wished a program
 - **It is not a no-code, point-and-click product.** You need to write a little Python or use the command line. The built-in web page is intentionally minimal; it is for trying things, not a polished app.
 - **It cannot magically beat every site's defenses.** Aggressive bot protection and CAPTCHAs are genuinely hard. A proxy provider helps a lot, but nothing is guaranteed. The most advanced mode (an AI that drives a browser through logins and CAPTCHAs) exists but is lightweight and experimental today, and ships without a default driver.
 - **It will not log into sites for you by default.** You can supply credentials or a saved login session, but automated login flows are still experimental.
-- **There is no built-in list/table extraction yet.** Today it extracts one record per page well; pulling every row out of a listing page is on the roadmap, not in the box.
 - **There is no hosted dashboard or scheduler.** Scrapo does not run your jobs in the cloud, send alerts, or give you a web console to manage everything. You run and schedule it yourself.
-- **It is alpha software.** The core works and is stable, but expect rough edges. Some pieces (browser pooling, cloud snapshot storage, advanced action caching, a hosted control plane) are planned, not built.
+- **It is alpha software.** The core works and is stable, but expect rough edges. Some pieces (a ready-made agent driver, in-browser request interception, following "next page" links and sitemaps, cloud snapshot storage, advanced action caching, a hosted control plane) are planned, not built.
 - **AI extraction costs money.** The first run on a new site (or after a layout change) calls a paid AI model. Scrapo is designed to minimize this (most runs use the free cached recipe) but it is not literally free.
 - **It is not legal advice or a compliance guarantee.** The robots rules, personal-data flagging, geo limits, and audit log are *tools* to help you scrape responsibly. You are still responsible for following each site's terms and the law. (Note: robots-rule enforcement is off by default; you have to turn it on.)
 - **It is Python-only.** No JavaScript, Java, Go, etc. versions. Requires Python 3.11 or newer.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "scrapo"
-version = "0.2.0"
+version = "0.3.0"
 description = "AI-native, agent-first web scraping library with deterministic replay"
 readme = "README.md"
 requires-python = ">=3.11"

scrapo/__init__.py

@@ -2,6 +2,7 @@
 
 from scrapo.api import crawl, extract, scrape
 from scrapo.config import Config
+from scrapo.results import ChunkView, CrawlResult, ExtractionView, ScrapeResult
 from scrapo.types import (
     Budget,
     ChunkedDocument,
@@ -12,16 +13,20 @@
     Tier,
 )
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 __all__ = [
     "Budget",
+    "ChunkView",
     "ChunkedDocument",
     "Config",
+    "CrawlResult",
     "ExtractionResult",
+    "ExtractionView",
     "FetchResult",
     "ProvenanceTag",
     "RunRecord",
+    "ScrapeResult",
     "Tier",
     "crawl",
     "extract",

scrapo/access/agent_tier.py

@@ -36,6 +36,10 @@ def __init__(
         self.config = config
         self.driver = driver
 
+    async def aclose(self) -> None:
+        """No persistent resources today; defined so TierRouter.aclose can call it."""
+        return
+
     async def fetch(
         self,
         url: str,

scrapo/access/browser_pool.py

@@ -0,0 +1,66 @@
+"""A lazily-launched, reused Playwright browser.
+
+Launching Chromium costs roughly a second or two. When a :class:`TierRouter` is
+reused across many fetches (most importantly during a crawl), this keeps one
+browser process alive and hands out a fresh context per fetch instead of
+cold-launching every time. It stays lazy: the browser is launched the first time
+a browser-tier fetch actually runs, never on import or construction.
+
+The browser is launched without a proxy; proxy settings (which may rotate per
+fetch) are applied at the context level, so a single pooled browser can serve
+every proxy.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+from typing import Any
+
+
+class BrowserPool:
+    def __init__(self, *, headless: bool = True) -> None:
+        self._headless = headless
+        self._lock = asyncio.Lock()
+        self._pw: Any = None
+        self._browser: Any = None
+
+    async def _ensure_browser(self) -> Any:
+        if self._browser is not None:
+            return self._browser
+        async with self._lock:
+            if self._browser is not None:
+                return self._browser
+            from playwright.async_api import async_playwright
+
+            pw = await async_playwright().start()
+            try:
+                browser = await pw.chromium.launch(headless=self._headless)
+            except Exception:
+                await pw.stop()
+                raise
+            self._pw = pw
+            self._browser = browser
+            return self._browser
+
+    @asynccontextmanager
+    async def context(self, **context_kwargs: Any) -> AsyncIterator[Any]:
+        browser = await self._ensure_browser()
+        ctx = await browser.new_context(**context_kwargs)
+        try:
+            yield ctx
+        finally:
+            await ctx.close()
+
+    async def aclose(self) -> None:
+        if self._browser is not None:
+            try:
+                await self._browser.close()
+            finally:
+                self._browser = None
+        if self._pw is not None:
+            try:
+                await self._pw.stop()
+            finally:
+                self._pw = None