Skip to content

Commit e41c966

Browse files
author
crawlee-cloud-docs-bot
committed
docs: sync from platform@b12b619
Mirrored from crawlee-cloud/crawlee-cloud@b12b619 Triggered by push on main
1 parent cad1a68 commit e41c966

1 file changed

Lines changed: 78 additions & 23 deletions

File tree

‎src/docs/roadmap.md‎

Lines changed: 78 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,15 @@
22

33
A CLI-first platform for running large-scale scrapers on your own infrastructure.
44

5-
## Current Version: v0.6.x
5+
## Current Version: v0.9.1
66

7-
- Everything from v0.5.0 plus:
8-
- DigitalOcean App Platform deployment (API + Dashboard)
9-
- Single-command DO setup script (PG, Redis, App Platform, Runner Droplet)
10-
- One-click deploy templates for Railway, Render, DigitalOcean
11-
- Platform-agnostic runner auto-scaler (DO provider, extensible)
12-
- Runner heartbeat system (CPU, memory, disk metrics via Redis)
13-
- Dashboard path-routing for reverse proxy deployments
14-
- Scraper deployment tooling (deploy any scraper from a monorepo)
7+
- Apify-compatible webhook payload templating engine (quoted, unquoted, interpolated, dot notation)
8+
- Server-side `?q=` substring search on every list endpoint
9+
- Reaper uses `GREATEST(accessed_at, modified_at)` so actively-written-to unnamed resources don't get reaped
10+
- Dashboard counter tiles read real `COUNT(*)` instead of capped `items.length`
11+
- Stress-fixture script in tree for at-scale QA
12+
13+
The platform survives months of operation unattended (retention reaper from v0.9.0), runs on real production volume (race-free dataset push from v0.8.0+, real pagination from v0.9.0), and is now Apify-template-compatible end-to-end.
1514

1615
---
1716

@@ -34,7 +33,7 @@ A CLI-first platform for running large-scale scrapers on your own infrastructure
3433

3534
---
3635

37-
## v0.3.0 ✅ - Production Scraping at Scale
36+
## v0.3.0 ✅ — Production Scraping at Scale
3837

3938
| Feature | Status |
4039
| --------------- | ------------------------------------------------- |
@@ -44,7 +43,7 @@ A CLI-first platform for running large-scale scrapers on your own infrastructure
4443
| Webhooks | ✅ Shipped — webhook deliveries with retry logic |
4544
| Resource limits | ✅ Shipped — memory limits per container |
4645

47-
## v0.4.0 ✅ - Reliability & Operations
46+
## v0.4.0 ✅ — Reliability & Operations
4847

4948
| Feature | Status |
5049
| ----------------- | --------------------------------------------------------------- |
@@ -53,7 +52,7 @@ A CLI-first platform for running large-scale scrapers on your own infrastructure
5352
| Graceful shutdown | ✅ Shipped — SIGTERM/SIGINT handlers in API and runner |
5453
| Backup & restore | ✅ Shipped — backup:create, backup:restore, backup:list scripts |
5554

56-
## v0.5.0 ✅ - Security & Polish
55+
## v0.5.0 ✅ — Security & Polish
5756

5857
| Feature | Status |
5958
| -------------------------- | --------------------------------------------------- |
@@ -63,7 +62,7 @@ A CLI-first platform for running large-scale scrapers on your own infrastructure
6362
| SSRF protection | ✅ Shipped — webhook URL validation |
6463
| Security config validation | ✅ Shipped — blocks insecure defaults in production |
6564

66-
## v0.6.0 ✅ - Cloud Deployment & Auto-Scaling
65+
## v0.6.0 ✅ — Cloud Deployment & Auto-Scaling
6766

6867
| Feature | Status |
6968
| ----------------------- | ---------------------------------------------------------------------- |
@@ -76,18 +75,74 @@ A CLI-first platform for running large-scale scrapers on your own infrastructure
7675
| SSL for managed DBs | ✅ Shipped — auto-enables for production Postgres |
7776
| Auto-migrations | ✅ Shipped — `run_command` runs migrations before API starts |
7877

79-
## v0.7.0 - Scale & Operate
78+
## v0.7.0 ✅ — Image Registry & Runner Cleanup
79+
80+
| Feature | Status |
81+
| ----------------------- | --------------------------------------------------------------- |
82+
| GHCR image registry | ✅ Shipped — push actor images to GitHub Container Registry |
83+
| DO Spaces integration | ✅ Shipped — managed S3-compatible storage in addition to MinIO |
84+
| Runner image cleanup | ✅ Shipped — runners prune old images so disks don't fill up |
85+
| Custom registry support | ✅ Shipped — `CRAWLEE_CLOUD_REGISTRY_URL` env var |
86+
87+
## v0.8.0 → v0.8.6 ✅ — Production-Hardening Cycle
88+
89+
A rapid patch cycle driven by what showed up under real load.
90+
91+
| Version | Headline |
92+
| ------- | ------------------------------------------------------------------------------------------------------------ |
93+
| v0.8.0 | Race-free dataset push (`UPDATE … RETURNING`); queue dedup hot path; pool-ceiling backpressure |
94+
| v0.8.1 | Scaler security & reliability patch (clamping, idempotent provider calls, SIGTERM grace) |
95+
| v0.8.2 | macOS dev reachability (host.docker.internal); scaler heartbeat resilience under network blips |
96+
| v0.8.3 | Webhook test/debug UI; real Settings panel; rich payload shape |
97+
| v0.8.4 | `/v2/system/info` execution defaults reflect runner config per-provider, not API env |
98+
| v0.8.5 | Honest API version on `/health` and dashboard (read from `package.json` at module-load instead of stale env) |
99+
| v0.8.6 | CLI `push` header shows the registry-qualified image; `publish-cli.yml` upgrades npm before publishing |
100+
101+
## v0.9.0 ✅ — Retention Reaper + Pagination at Scale
102+
103+
The "platform survives months of operation" gate.
104+
105+
| Feature | Status |
106+
| ----------------------------- | --------------------------------------------------------------------------------------------------- |
107+
| Retention reaper | ✅ Shipped — periodic cleanup of unnamed datasets/KV/queues + finished runs past TTL |
108+
| `retention_tombstones` audit | ✅ Shipped — every reaped row leaves a tombstone before being pruned at `RETENTION_TOMBSTONE_DAYS` |
109+
| Admin retention status | ✅ Shipped — `GET /v2/system/retention/status` + dashboard `/retention` page (auto-refresh 30s) |
110+
| Real list-endpoint pagination | ✅ Shipped — `total` from parallel `COUNT(*)`, stable `created_at DESC, id DESC` tiebreaker |
111+
| Dashboard pagination UI | ✅ Shipped — URL-driven `?page=N`, "Page X / Y" with editable page input, out-of-range error |
112+
| Constants centralized | ✅ Shipped — `packages/dashboard/src/lib/constants.ts`; sidebar version sourced from `package.json` |
113+
| Deployment recipes | ✅ Shipped — hobby docker-compose + DigitalOcean Spaces |
114+
115+
## v0.9.1 ✅ — Webhook Templating + Server-Side Search
116+
117+
| Feature | Status |
118+
| ------------------------------------------------ | ------------------------------------------------------------------------------------------ |
119+
| Apify-compatible webhook engine | ✅ Shipped — both `"{{x}}"` quoted and `{{x}}` unquoted forms; interpolation; dot notation |
120+
| Test webhook applies template | ✅ Shipped — `POST /v2/webhooks/:id/test` runs the same engine as production deliveries |
121+
| Server-side `?q=` search | ✅ Shipped — ILIKE substring on every list endpoint, LIKE-metachar escape |
122+
| Reaper uses `GREATEST(accessed_at, modified_at)` | ✅ Shipped — actively-written-to unnamed resources no longer false-reaped |
123+
| Dashboard counts real total | ✅ Shipped — counter tiles read `COUNT(*)`, not `items.length` capped at 1000 |
124+
| Stress-fixture script | ✅ Shipped — `scripts/seed-stress-fixtures.ts` for at-scale QA |
125+
126+
## v0.9.x patch line
127+
128+
Anything additive or bug-fix-shaped lands here as `0.9.<n+1>` until v1.0 cuts.
129+
130+
---
131+
132+
## v1.0.0 — Stability commitment
133+
134+
**The next major after v0.9.x is `1.0.0`, not `0.10.0`.** Going to 1.0 carries semver weight: breaking changes after 1.0 cost a major bump or a deprecation window, so the version is also a commitment to the contract.
80135

81-
**Priority:** Production operations for 100+ scrapers.
136+
What goes in:
82137

83-
| Feature | Description |
84-
| ----------------------- | ------------------------------------------------------------------- |
85-
| GHCR image registry | Push actor images to GitHub Container Registry for fast runner pull |
86-
| DO Spaces integration | Managed S3 storage instead of self-hosted MinIO |
87-
| Scheduling UI | Create and manage cron schedules from the dashboard |
88-
| Actor versioning | Complete the versioning workflow (table exists, needs UI) |
89-
| API key scopes | Read-only vs full access keys |
90-
| Hetzner scaler provider | Scale runners on Hetzner Cloud (cheaper than DO) |
138+
| Area | Why it's a v1.0 task |
139+
| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
140+
| `packages/shared` workspace | The `applyWebhookTemplate` engine is duplicated in api+runner with KEEP-IN-SYNC headers. Promote to a real workspace. |
141+
| Apify v2 API drift audit | Any gap between our v2 surface and current Apify is cheaper to close before the contract is "supported." |
142+
| DB migration policy | Forward-only, additive. No breaking column drops without a deprecation window. Documented. |
143+
| CLI command stability | No rename/repurpose without a `crawlee-cloud migrate` style helper. |
144+
| Webhook `payload_template` examples in dashboard editor | So operators don't have to consult Apify docs to discover `{{eventData}}` syntax. |
145+
| Auth/role surface tightening | Admin scopes, API key TTLs — anything role-related lands here so post-1.0 doesn't break tokens. |
91146

92147
---
93148

0 commit comments

Comments
 (0)