Merge pull request #2 from qdequele/qdequele/backend-rebuild-rails-82f5a8

Backend rebuild: Rails SaaS control plane + Rust crawl engine (SCR-85 / SCR-87)

Author: qdequele

.env.example

@@ -17,9 +17,54 @@ ENABLE_CORS=true
 # JWT secret for auth tokens
 JWT_SECRET=dev-jwt-secret-change-in-production
 
+# Auth brute-force limit (attempts/min/IP), read by the Rails app's
+# Rack::Attack throttle. Default 5; raise for contract-test runs (contracts/
+# suite signs up many throwaway users). Leave unset in prod.
+# AUTH_RATE_LIMIT=1000
+
 # PostgreSQL connection URL
 DATABASE_URL=postgres://scrapix:scrapix@localhost:5433/scrapix
 
+# -----------------------------------------------------------------------------
+# SaaS Control Plane (Rails) — SCR-85 backend split
+# -----------------------------------------------------------------------------
+# Port for the Rails app (saas/)
+SAAS_PORT=8081
+
+# Console proxy routing: path prefixes listed in SAAS_PREFIXES (comma-
+# separated) are proxied to the Rails app; everything else goes to the Rust
+# crawl engine. Since phase 9 the Rust API no longer serves any SaaS route,
+# so SAAS_API_URL + the full prefix list below are REQUIRED for the console
+# to work. Set inline on the console entry in Procfile.dev for native dev.
+# SAAS_EXCLUDE_PREFIXES keeps deeper paths on Rust when a route group is
+# split mid-migration (empty since phase 7).
+SAAS_API_URL=http://localhost:8081
+SAAS_PREFIXES=analytics,configs,engines,auth,account,webhooks,oauth,.well-known,mcp
+
+# MCP tool generation (Rails /mcp): path to the frozen OpenAPI snapshot and
+# the base URL the MCP server uses to call back into the SaaS app itself.
+# Defaults work for native dev; docker-compose sets both explicitly.
+# MCP_OPENAPI_SPEC_PATH=contracts/openapi.json
+# MCP_SAAS_BASE_URL=http://localhost:8081
+
+# Transactional email (Rails ActionMailer via Resend SMTP; SolidQueue drains
+# the shared scheduled_emails queue the Rust engine writes into). Without the
+# key, dev deliveries go to ActionMailer's :test backend.
+# RESEND_API_KEY=re_...
+# Some hosts (Scaleway) block egress on 465/587; Resend also listens on 2465.
+# RESEND_SMTP_PORT=465
+# CONSOLE_PUBLIC_URL=https://scrapix.meilisearch.com
+
+# Social login (Rodauth omniauth in the Rails app). Register the callback
+# URLs /auth/google/callback and /auth/github/callback with the providers.
+# GOOGLE_CLIENT_ID=...
+# GOOGLE_CLIENT_SECRET=...
+# GITHUB_CLIENT_ID=...
+# GITHUB_CLIENT_SECRET=...
+
+# Rust crawl engine URL, used by the Rails app to submit crawls (config trigger)
+# SCRAPIX_ENGINE_URL=http://localhost:8080
+
 # -----------------------------------------------------------------------------
 # Kafka/Redpanda Configuration
 # -----------------------------------------------------------------------------
@@ -89,6 +134,8 @@ BATCH_TIMEOUT_SECS=5
 # Stripe Payments (Optional)
 # -----------------------------------------------------------------------------
 # Enables credit purchases via Stripe. Without these, topups are free (dev mode).
+# Purchases/webhooks are handled by the Rails app; the Rust engine reads only
+# STRIPE_SECRET_KEY (for auto-topup charges triggered by usage debits).
 # STRIPE_SECRET_KEY=sk_test_...
 # STRIPE_WEBHOOK_SECRET=whsec_...
 

CLAUDE.md

@@ -169,9 +169,9 @@ scrapix health
 
 ## Analytics API (Tinybird-style)
 
-When ClickHouse is configured (`CLICKHOUSE_URL` environment variable), the API server automatically:
-1. **Persists crawl events to ClickHouse** - PageCrawled and PageFailed events are batched (100 events) and flushed every 5 seconds
-2. **Enables the analytics API** at `/analytics/v0/pipes/`
+When ClickHouse is configured (`CLICKHOUSE_URL` environment variable):
+1. The **Rust engine persists crawl events to ClickHouse** - PageCrawled and PageFailed events are batched (100 events) and flushed every 5 seconds
+2. The **Rails SaaS app serves the analytics API** at `/analytics/v0/pipes/` (port 8081, or via the console proxy)
 
 This provides long-term analytics storage beyond the in-memory diagnostics.
 
@@ -243,6 +243,26 @@ curl "http://localhost:8080/analytics/v0/pipes/kpis.json?hours=24"
 
 ## Architecture
 
+### Two Backends (SCR-85 split)
+
+The backend is deliberately split into two services sharing one Postgres:
+
+- **Rails SaaS control plane (`saas/`, port 8081)** — auth + sessions, social
+  login, account/team/invites, API keys, billing + Stripe, saved crawl
+  configs + engines CRUD, analytics pipes, the OAuth 2.1 provider, and the
+  MCP server at `/mcp`. Contract-tested by `contracts/`.
+- **Rust crawl engine (`bins/scrapix-api`, port 8080)** — /scrape, /map,
+  /search, /crawl*, jobs, WebSockets, diagnostics, plus the background tasks
+  that belong to the data plane: cron scheduler (fires saved configs), email
+  scheduler (delivers the shared `scheduled_emails` queue via Resend),
+  OAuth token cleanup, and Stripe auto-topup on usage debits. It validates
+  credentials (API keys, Bearer tokens, session JWTs) but issues none.
+
+The console proxy (`console/src/app/api/scrapix/[...path]/route.ts`) routes
+by path prefix via `SAAS_API_URL` + `SAAS_PREFIXES`; the frozen full-platform
+spec is `contracts/openapi.json`, the engine-only spec is
+`contracts/openapi.engine.json`.
+
 ### Workspace Structure
 
 The project is organized as a Cargo workspace with two main directories:

Cargo.lock

@@ -2,4 +2,5 @@ api:      set -a && source .env && set +a && cargo watch -w crates -w bins -x 'r
 frontier: set -a && source .env && set +a && cargo watch -w crates -w bins -x 'run -p scrapix --bin scrapix -- frontier'
 crawler:  set -a && source .env && set +a && cargo watch -w crates -w bins -x 'run -p scrapix --bin scrapix -- crawler'
 content:  set -a && source .env && set +a && cargo watch -w crates -w bins -x 'run -p scrapix --bin scrapix -- content'
-console:  cd console && npm install && npm run dev
+console:  cd console && npm install && SAAS_API_URL=http://localhost:8081 SAAS_PREFIXES=analytics,configs,engines,auth,account,webhooks,oauth,.well-known,mcp npm run dev
+saas:     set -a && source .env && set +a && cd saas && bin/rails db:prepare && SOLID_QUEUE_IN_PUMA=1 OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES bin/rails server

Procfile.dev

@@ -60,34 +60,19 @@ tracing-subscriber = { workspace = true }
 # Async traits
 async-trait = { workspace = true }
 
-# Auth
+# Credential validation (API keys, OAuth Bearer tokens, session JWTs)
 sqlx = { version = "0.8", features = ["runtime-tokio", "postgres", "uuid", "chrono", "tls-rustls"] }
 sha2 = "0.10"
 hex = "0.4"
-argon2 = "0.5"
-jsonwebtoken = "9"
 axum-extra = { version = "0.10", features = ["cookie"] }
-rand = { workspace = true }
-
-# OAuth
-base64 = { workspace = true }
-
-# Rate limiting
-redis = { workspace = true }
-dashmap = { workspace = true }
-
-# URL encoding for OAuth redirects
-urlencoding = "2.1"
-
-# MCP HTTP transport
-rmcp = { version = "1.2", features = ["server", "transport-streamable-http-server"] }
-rmcp-openapi = "0.25"
 
 # Cron scheduling
 croner = "3"
 
 # Stripe payments
-async-stripe = { version = "0.38", default-features = false, features = ["runtime-tokio-hyper", "checkout", "billing", "connect", "webhook-events"] }
+# "billing" alone doesn't compile in async-stripe 0.38 (its quote resource
+# references checkout types), so checkout stays enabled alongside it.
+async-stripe = { version = "0.38", default-features = false, features = ["runtime-tokio-hyper", "checkout", "billing"] }
 hmac = "0.12"
 
 # OpenAPI