MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.
-
Updated
Mar 27, 2026 - Python
#
trafilatura
Here are 55 public repositories matching this topic...
Self-hosted search + markdown harvester for AI agents. SearXNG (100+ engines) + FastAPI + trafilatura. Tavily-compatible /search plus /extract with size presets and pagination. One-command Docker Compose.
markdown docker-compose self-hosted web-scraping search-api ai-agents rag fastapi searxng llm llm-tools trafilatura tavily agentic-ai
-
Updated
Apr 27, 2026 - Python
pebkac Chrome Nonautomation - A Local LLM-Driven Web Co-Browser using Smolagents, Zendriver, Trafilatura.
chrome automation ai openai webscraping atlas claude llms claude-ai trafilatura nodriver ai-browser-automation smolagents ai-browser-control aibrowser zendriver ai-browser pebkac
-
Updated
Apr 14, 2026 - Python
Fast, accurate web content extraction in Rust. ML page-type classification, per-type extraction, confidence scoring. F1=0.966 on ScrapingHub (#1), F1=0.859 across 2,008 annotated pages (1,497 development + 511 held-out test
-
Updated
Apr 3, 2026 - Rust
Tools for LLMs to anonymously search and browse the web
mcp selenium tor duckduckgo selenium-webdriver tor-network geckodriver tor-hidden-services trafilatura mcp-server fastmcp ddgs tbselenium
-
Updated
Mar 17, 2026 - Python
Fast and accurate web content extraction
-
Updated
Jul 18, 2026 - Rust
web Scrapper In Python
-
Updated
Sep 6, 2023 - Python
Keyless, uv-native web search + read for AI agents: ddgs multi-engine search with de-correlated rank fusion, Trafilatura extraction to paginated Markdown, keyless arxiv and github search. Self-hosted SearXNG without Docker, an optional Tor layer for .onion search and fetch, SOCKS5 egress, one-call init, and a doctor.
python tor self-hosted arxiv socks5-proxy content-extraction ai-agents web-search uv keyless onion-services ahmia searxng rank-fusion prompt-injection trafilatura agent-skills ddgs
-
Updated
Jul 28, 2026 - Python
Convert any URL into clean, token-efficient Markdown for LLMs. CLI, Python API, MCP server.
-
Updated
Jun 14, 2026 - Python
ChatGPT AI Clone
python dart websockets gemini web-scraping flutter fastapi sentence-transformers flutter-web-app trafilatura tavily ai-clones
-
Updated
Jul 3, 2026 - Dart
Telegram Mini App that saves internet articles to read them later
-
Updated
Feb 19, 2026 - Python
Selective web content extraction for AI agents — URL + query returns only the chunks that matter (Python library + MCP server)
python mcp embeddings web-scraping content-extraction ai-agents rag playwright llm trafilatura bge-m3 mcp-server
-
Updated
Jul 7, 2026 - Python
Fast, permissive, self-hosted web extraction service — URL → clean LLM-ready markdown. Async HTTP/2, conditional Playwright rendering, multi-strategy extraction, PDF/academic pipeline. No LLM required, no copyleft deps. Apache-2.0.
python markdown web-crawler self-hosted web-scraping content-extraction fastapi playwright llm trafilatura
-
Updated
Jul 29, 2026 - Python
A pipe-based news article scraping and metadata extraction library for Python
python crawler data-science scraper spider data-collection news-scraper osint-python llms article-extraction trafilatura newspaper4k
-
Updated
Mar 20, 2026 - Python
Real-time AI search and chat backend with WebSocket streaming, powered by Tavily web search and Google Gemini for Flutter apps.
-
Updated
Aug 2, 2025 - Python
Safe, config-driven Python web ingestion pipeline with extraction, evidence-gated AI generation, provenance ledger, RAG chunks, data cards, and multi-provider exports.
python pypi provenance gemini web-scraping openai developer-tools python-package audit-trail html-extraction structured-output rag research-software ai-pipeline llm trafilatura deepseek web-ingestion data-cards ssrf-protection
-
Updated
Jul 25, 2026 - Python
Local-first AI trend intelligence dashboard with GitHub Radar, Research Radar, Startup Gap Finder, Scrapy/Trafilatura crawler, SQLite, Streamlit, and Ollama support
sqlite web-crawler developer-tools scrapy market-intelligence local-first streamlit startup-ideas trafilatura ollama ai-dashboard trend-intelligence github-radar research-radar hackathon-radar
-
Updated
May 29, 2026 - Python
Protocole de collecte et d'analyse d'archives de la Wayback Machine pour une analyse textuelle et statistique
-
Updated
Apr 20, 2026 - Python
HTML main-content extraction for Rust — ports of Mozilla Readability, Trafilatura, and htmldate.
html rust text-extraction web-scraping html-parser readability content-extraction metadata-extraction boilerplate-removal article-extraction trafilatura htmldate
-
Updated
May 26, 2026 - HTML
FastAPI service that classifies publisher websites for affiliate campaigns using an LLM pipeline (scrape → signals → RAG → scoring). Detects cashback, adult, gambling, scams. Supports OpenAI/Ollama, Redis cache, Docker.
docker redis openai python-asyncio affiliate-marketing rag pydantic httpx sentence-transformers llm website-scraping chromadb trafilatura ollama scam-detection publisher-website-classifier gambling-detection adult-content-filter cashback-detection
-
Updated
Mar 30, 2026 - Python
Improve this page
Add a description, image, and links to the trafilatura topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the trafilatura topic, visit your repo's landing page and select "manage topics."