Recommendation Engine

A custom ML-based product recommendation engine built from raw clickstream in ClickHouse. Three use cases, one pipeline:

1. Product page — "Similar items" — ALS item-factor cosine similarity
2. Cart / product page — "Frequently viewed together" — 90-day session co-occurrence
3. Homepage — "Recommended for you" — personalised ALS predictions per user

No external recommender service. No prebuilt event matrices. Everything is computed from raw events with implicit's ALS, materialised back into ClickHouse, and served by a thin FastAPI layer.

Why this design

• ClickHouse for the heavy lifting — joining 700M-row events with 800M-row item denormalisation in Python is a non-starter. Aggregation runs as ClickHouse SQL; Python only loads the resulting interaction matrix (≈50–100M rows).
• ALS via implicit — fast, well-supported, handles implicit feedback (views/carts/purchases) natively. 128-factor model, ~2 minutes to train on 500k users × 100k items.
• Pre-compute, don't predict at request time — the API just reads pre-computed top-N lists out of ClickHouse. Latency is microseconds, and we filter inactive/OOS items at read time.
• Cold-start — falls back to category-aware popularity for new users / unknown items.

Architecture

         clickstream events (ClickHouse)
                    │
     ┌──────────────┼──────────────┐
     ▼                  ▼                  ▼
 materialize.py     train.py          precompute.py
  (nightly)        (weekly)           (weekly, post-train)
                          │                  │
                  als_model.pkl    user_recs / item_similar / item_pairs
                                          │
                                          ▼
                                       api.py
                                  GET /recommend/...

Event weighting

Interaction score per (user, item) over a rolling 180-day window:

Event	Weight
view_item	1
add_to_wishlist	3
add_to_cart	3
begin_checkout	4
purchase	5

Tune in materialize.py if your domain calls for different weights (e.g. a high-AOV catalogue may want to lean harder on purchase).

Pipeline

Step 1 — Materialise interaction data (nightly)

Aggregates 180 days of raw events into a user-item interaction matrix in ClickHouse. Also rebuilds 90-day session co-occurrence pairs and 30-day popularity.

python3 materialize.py

Duration: ~5–10 min on a typical ClickHouse cluster.

Step 2 — Train ALS (weekly)

Trains a matrix factorization model with implicit. Saves to als_model.pkl.

python3 train.py

Duration: ~2–3 min for 500k users × 100k items, 128 factors, 20 iterations.

Tunable via env:

Var	Default	What
ALS_FACTORS	128	Embedding dimensions
ALS_ITERATIONS	20	Training iterations
ALS_REGULARIZATION	0.01	L2 regularisation
MODEL_PATH	als_model.pkl	Where to save the model

Step 3 — Precompute (weekly, after training)

Writes top-50 personalised recs per user and top-20 similar items per product back into ClickHouse.

python3 precompute.py

Duration: ~5–7 min depending on catalogue size.

Recommended schedule

0 2 * * *   python3 materialize.py >> logs/materialize.log 2>&1
0 4 * * 0   python3 train.py       >> logs/train.log 2>&1
30 4 * * 0  python3 precompute.py  >> logs/precompute.log 2>&1

Serving

uvicorn api:app --host 0.0.0.0 --port 8003
# or
docker compose up

Endpoints

Endpoint	Source table	Use case
GET /recommend/user/{user_id}?n=20	recommendations.user_recs	Personalised homepage feed
GET /recommend/similar/{item_id}?n=10	recommendations.item_similar	"Similar items" on PDP
GET /recommend/together/{item_id}?n=10	recommendations.item_pairs	"Frequently viewed together"
GET /health	—	Liveness

All endpoints filter out inactive / OOS items at read time and fall back to popularity for unknown users/items. Results are Redis-cached (1h for user feeds, 6h for item-level).

curl http://localhost:8003/recommend/user/1466796?n=10
curl http://localhost:8003/recommend/similar/46301?n=10
curl http://localhost:8003/recommend/together/46301?n=10

Schema assumptions

The pipeline expects three logical sources in ClickHouse — rename the table identifiers in materialize.py / precompute.py / api.py to match yours:

• events table with columns: event_id, user_id, event_type, event_date, platform
• event→items denorm table with: event_id, item_id, event_date, optional item_category_name
• product catalog with: id_product (or equivalent), active, quantity

The shipped SQL uses placeholder identifiers (events.events, events.event_items_denorm, catalog.products) that you swap for your actual table names.

One-time setup

pip install -r requirements.txt
cp .env.example .env   # fill in ClickHouse credentials

# Create the output tables
python3 -c "
import sys, re
sys.path.insert(0, '.')
from db import get_client
client = get_client()
sql = open('setup_tables.sql').read()
for stmt in [s.strip() for s in re.sub(r'--[^\n]*', '', sql).split(';') if s.strip()]:
    client.command(stmt)
    print('OK:', stmt[:60])
"

Configuration

Variable	Default	Description
CLICKHOUSE_HOST	—	ClickHouse host (required)
CLICKHOUSE_PORT	8443	Native HTTPS port
CLICKHOUSE_USER	default
CLICKHOUSE_PASSWORD	—
CLICKHOUSE_SECURE	true	Use TLS
MODEL_PATH	als_model.pkl	Where train.py saves the model
USER_RECS_N	50	Top-N user recs to precompute
ITEM_SIMILAR_N	20	Top-N similar items to precompute
REDIS_URL	redis://localhost:6379	Optional Redis for response caching

License

MIT — see LICENSE.