🧠💻 Reimagine the Internet as Self-Organizing Mind Map

Critical times call for critical thinkers to create a crowdsourced argument reasoning dataset, for AI models to recommend research quotes, to evolve crowdsourced chain-of-thought reasoning, to unlock faster ways to read long articles, to monitor developments by topic modeling a knowledge base graph, and to provide a public service of answers to research. Language Models can distill the essence of collective thought into a vector space where every point has a weighted value representing its contribution to the overall decision-making process. AI will show its reasoning based on what sentences and cites it used from the collective research, so that people can see it is aligned with our interests. Research Agents recommend articles for human researchers working alongside AI to develop a summarized topic outline as a public service. The agents monitor for any related articles via web searches for keywords associated with that Topic Model. Imagine uploading a research paper, then the app extracts full text of reference cites and creates topic model and keyword summaries, then monitors that literature base and stores highlights. People will make personal knowledge bases of what influences them to create AI assistants cloning their mind-uploaded perspective and interests in a self-organizing mind map.

# Download Source
bun x git0 vtempest/qwksearch-research-agent

# Import API Client
bun i qwksearch-api-client

🤖🔎 STREAM: Search with Top Result Extraction & Answer Model

Reimagine The Web as Self-Organizing Mind Map

• 🔍 Web Search - 100+ popular sites search across 10 categories: Web Search, Academic, Videos, Images, Files, News, etc
• 📝 Article Preview - Extract, format with APA cite, and summarize articles, PDFs, Youtube, and URLs before reading them
• 🤖 Choice of Language Model - Claude, GPT, Gemini, Grok, Llama, Chinese LLMs, etc
• 📄 File Upload Support - Ask questions about PDFs, URLs, DOCX, Google Docs, and Youtube
• 📚 Search History - All searches saved with memories, except for privacy mode
• ❓ Follow-up Questions - Generate follow-up questions to ask language models

REASON Writing Agent

Research Editor for Annotated Summaries in Outline Notation

• 📝 Complex Rich Text Editor: full featured alternative to Google Docs based on Meta's Lexical with easy minimalist UI
• 📂 Nested Document Tree: organize research notes with a nested document organizer with drag-and-drop, tabs, and custom storage sources
• 🖱️ Context Menu: right-click to access quick actions for seamless document management
• 🔍 Full-Text Search: instantly find documents by title or content with full-text search
• ✨ AI Rewriting: leverage AI to rewrite and improve your text directly within the editor
• 👥 Team Management: collaborate with team members and manage access rights
• 🔄 View Modes: switch between Formatted, parsing HTML, and Markdown views for versatile editing
• 🛠️ Find & Replace: powerful search and replace functionality with match highlighting
• 📥 Google Docs Integration: seamless export, import, and sharing capabilities
• 💾 Persistent Storage: reliable SQLite storage ensures your data is safe and accessible
• ⌨️ Keyboard Navigation: efficient keyboard shortcuts for power users
• 💬 Research Quotes: capture and organize key quotes and insights from your research
• 🏗️ Built with: Claude Code, Lexical Editor, shadcn/ui, Cloudflare, Next.js, Google Docs API

🚜📜 Tractor the Text Extractor

1. Main Content Detection: Extract the main content from a URL by combining Mozilla Readability and Postlight Mercury algorithms, utilizing over 100 custom adapters for major sites for article, author, date HTML classes.
2. YouTube Transcript Processing: When a YouTube video URL is detected, retrieve the complete video transcript including both manual captions and auto-generated subtitles, maintaining proper timestamp synchronization.
3. PDF to HTML: Extracts formatted text from PDF with parsing of linebreaks , page headers, footnotes, and section headings. Supports fonts, links, bold, italics, lists, headings, headers, footnotes, and Table of Contents, Quotes, and Code Blocks. Removes repeated headers, links footnote anchors to the footnote, and preserves number of the PDF page with invisible I element. This function uses pdfjs-serverless to work in more environments than PDF.js-based tools: Cloudflare workers, serverless, node.js, and front-end only.
4. Basic HTML Standardization: Transform complex HTML into a simplified reading-mode format of basic HTML, making it ideal for research note archival and focused reading, with headings, images and links.
5. Cite: Identify and extract citation metadata including author names, publication dates, sources, and titles using HTML meta tags and common class name patterns. The system validates author names against a comprehensive database of 90,000 first and last names, distinguishing between personal and organizational authors to properly format citations.

🖥️ Desktop App

QwkSearch Desktop — Built with Tauri. Select any text on screen, press ` to instantly search the web without leaving your workflow.

🧩 Browser Extension

QwkSearch Tab Manager AI — AI-powered tab manager that organizes, searches, and manages browser tabs intelligently.

🧠 VS Code Extension

QwkSearch for VS Code — Ask cited research questions from a sidebar in your editor, backed by the same QwkSearch API. Sign in with your account's API key, or use it signed-out as a guest.

📦 Packages

• chat-agent-toolkit — A multi-provider AI agent toolkit that generates language responses, searches the web, extracts content, and manages memory across 10+ LLM providers. It integrates the Vercel AI SDK, Mastra framework, and MCP protocol to orchestrate research agent workflows.

• domain-rank — Looks up top-ranked domains from the Tranco List and CommonCrawl backlink data to retrieve their human-readable source label, influence rank, and favicon. Useful for search/URL autocomplete, bookmark launchers, and domain reputation scoring.

• extract-pdf — Converts a PDF from a URL or ArrayBuffer into clean HTML with structural tagging including headings, lists, footnotes, and code blocks. Works in Node.js, Cloudflare Workers, and browser environments with zero runtime dependencies.

• extract-pdf-docling — Converts PDF documents to HTML using IBM's granite-docling-258M AI model via Hugging Face Transformers, preserving layout and structure through OCR. Supports recognition of code, formulas, tables, lists, charts, and figures via a Hono-based HTTP API.

• extract-webpage — Searches, extracts, cites, and outlines web content for a topic using an AI Research Agent. Combines PDF extraction, YouTube transcript extraction, DOM parsing, and LLM-based summarization to produce structured content from arbitrary web pages.

• extract-youtube — A fast, no-browser, serverless-optimized YouTube transcript extractor that fetches subtitles and captions without requiring a headless browser. Supports multiple output formats (SRT, WebVTT) and runs on edge/serverless platforms.

• notebooklm-api-client — Placeholder package for a future API client for Google's NotebookLM service.

• qwksearch-api-client — An auto-generated TypeScript API client for the QwkSearch platform, built from an OpenAPI specification. Provides typed fetch-based bindings for interacting with the QwkSearch backend API.

• reason-editor — A formatted text editor built on Lexical (React) with a toolbar, documents manager, and note outlines. Includes drag-and-drop, collaborative editing via Yjs, file management, and rich UI components for a full-featured writing/research interface.

• render-url-to-html — A collection of URL-to-HTML rendering strategies using Cloudflare Browser Rendering, Puppeteer with stealth plugins, and JSDOM. Fetches URLs and returns fully-rendered DOM as HTML, capable of bypassing bot-detection on JavaScript-rendered pages.

• research-agent-ui — The chat research agent UI: conversation window, article reader, search config, file uploads, and chat history, along with the shadcn primitives and icons it depends on. Drops into a Next.js app behind a small config/injection surface for auth, branding, and media-search preferences.

• search-web-api — Provides access to 70+ search engines across 10 categories (web, academic, news, images, etc.) plus a scrape/extract API served via a Hono HTTP server. Includes Hugging Face Transformers integration for AI-powered processing.

• searxng-search-cloudflare — A deployment configuration for running a private SearXNG metasearch engine proxy in Docker. Aggregates results from multiple search engines without tracking the user, providing a privacy-respecting search backend.

• shadcn-app-dock — A prop-driven, macOS-style category dock React component with icon magnification on hover and a built-in shadcn theme switcher. Uses Framer Motion for animations and integrates with next-themes for light/dark mode toggling.

• language-model-training — A from-scratch GPT-style transformer implementation built on Tinygrad that trains a next-word-prediction language model with a full Wikipedia pipeline. Ships with a FastAPI control API, Docker Compose orchestration, and a Next.js dashboard for monitoring training jobs.

• write-language — A multi-provider language generation toolkit using the Vercel AI SDK that generates text responses via 10+ LLM providers including OpenAI, Anthropic, Google, Groq, and more. Provides a unified interface for streaming and non-streaming text generation.

Further Research

• ThoughtSource Reasoning Datasets
• Awesome-LLMs-Datasets
• GPT Researcher
• NLP Papers Latest Updates
• Anthropic Persuasion Overview
• NLP Research Progress
• NLP Datasets
• Google Search Algorithm
• Can LLMs Generate Novel Research Ideas?
• Graph Algorithms Playground
• CommonCrawl C4 Download
• Knowledge Graphs Prompts Papers
• Paper - Iterative Research Idea Generation
• Open Deep Search
• LangChain Hub

🌟 Star this repo so it will grow new features!