IntentForge v2

IntentForge v2 is a high-performance, intent-first discovery engine designed to identify and rank web content based on deep user intent signals. It combines a high-speed local index with a real-time meta-search discovery layer to provide comprehensive, relevant results even for complex, long-tail queries.

🚀 Key Features

• Intent-First Ranking: Beyond keyword matching, IntentForge parses the underlying intent (Tutorial, Academic, DevOps, etc.) to weigh sources and results dynamically.
• Hybrid Search Architecture: Blends BM25 keyword matching with ONNX-powered dense vector semantic search (384-dimensional embeddings with 8x binary quantization).
• Synchronous Discovery Fallback: When the local index has insufficient quality, the engine automatically fans out parallel meta-searches through Tor to bridge the gap in real-time (with a sub-3s total latency target).
• Autonomous Self-Improvement: Search gaps trigger background crawling and indexing tasks, ensuring the engine "learns" from every query it cannot initially satisfy.
• Privacy-Preserving Meta-Search: Meta-discovery is routed through the Tor network using Snowflake 2.10+ bridges with SQS rendezvous for maximum anonymity and bypass of exit-node blocking.
• Smart Attribute Extraction: Automatically identifies skill levels (beginner/advanced), temporal constraints (years), content types (news/papers), and domain filters (site:) from natural language.

📈 Current Status & Roadmap

For a detailed look at the system's evolution, latest performance benchmarks, and future plans, please see:

• Investigation Report: Recent stabilization results and latency breakthroughs.
• Development Roadmap: Future phases and upcoming features.

🛠 Tech Stack

• Core: Rust (Edition 2021), Tokio, Axum
• Indexing: Meilisearch (v1.13+)
• Storage & Caching: Redis (Redis Stack 7.2) with SimHash deduplication
• Inference: ONNX Runtime (all-MiniLM-L6-v2 for embeddings, rerank-MiniLM-L6-v2 for cross-encoding)
• Extraction: Python/Trafilatura microservice
• Network: Tor with Snowflake pluggable transports

🚦 Getting Started

Prerequisites

• Docker & Docker Compose
• Rust 1.75+ (for local development)

Deployment

# Start the full stack (Infrastructure + Microservices + API)
docker-compose -f docker-compose.dev.yml up -d --build

# Initialize the Meilisearch index with optimized vector settings
./scripts/init_meilisearch.sh

Usage

The API listens on port 9100.

Search (Hybrid Discovery):

curl "http://localhost:9100/search?q=beginner+rust+tutorial+2024"

Dedicated Endpoints:

• GET /news: News aggregation
• GET /images: Semantic image discovery
• GET /videos: Intent-weighted video search

📖 API Documentation

For detailed endpoint definitions, request/response models, and configuration options, please refer to the interaction context defined in GEMINI.md.

---

Built with ❤️ by Likhith Sai Seemala