0.29.0

[MohamedBassem]

0.29.0

Welcome to the 0.29.0 release of Karakeep! This release ships some of our most awaited features. Collaborative lists, automated bookmark backups, search auto complete, highlighs are getting notes and search, and the mobile app is getting some more love. As usual thanks to @aa-ko, @fivestones, and everyone who shipped code, triaged bugs, or shared feedback for this release.

If you enjoy using Karakeep, consider supporting the project here ☕️ or via GitHub here.

And in case you missed it, we now have a ☁️ managed offering ☁️ for those who don't want to self-host. We're still in private beta (you can signup for access here) and gradually letting more and more users in. We're in public beta now and you can signup here 🎉.

New Features 🚀

• Collaborative lists are here! (feat: Add collaborative lists #2146, feat: Add invitation approval for shared lists #2152)
  • You can now invite collaborators to your lists and manage their access levels between viewers and editors.
  • This was the most requested feature on the roadmap, and it's now here!
• Automated bookmark backups you can schedule once and forget (feat: Add automated bookmark backup feature #2182)
  • Currently it only captures non-asset bookmarks, but I'm planning to include lists, tags, and other metadata in the future.
• Search gets autocomplete so you can find the right filters and terms faster (refactor(web): split search autocomplete logic #2178)
• Highlights overhaul: notes + search bar on web, plus a dedicated highlights page on mobile (feat: Add notes feature to highlights #2154, feat: Add search bar to highlights page #2155, feat(mobile): Add highlights page to mobile app #2156, feat: Add AI summary field to mobile bookmark info #2157)
• Mobile catches up with smart list creation and an all-tags screen (feat(mobile): Add smart list creation in mobile app #2153, feat: Add tags screen to mobile app #2163)
• Crawler domain rate limiting to avoid getting throttled by external sites (feat: add crawler domain rate limiting #2115)
  • Configure it with CRAWLER_DOMAIN_RATE_LIMIT_WINDOW_MS and CRAWLER_DOMAIN_RATE_LIMIT_MAX_REQUESTS.
• Import from MyMind (feat: import from mymind #2138)

UX Improvements ✨

• Sidebar typography and colors should feel nicer (specially in dark mode).
• Page titles are now correctly displayed in the browser tabs.
• We have a friendlier 404 page for bookmarks/lists that don't exist.
• You can now see stats about the source of your bookmarks in the usage stats page (extension, web app, mobile app, etc).

Fixes 🔧

• Prompts lazily load js-tiktoken which should cut between 70-150MB of karakeep's memory usage (fix: lazy load js-tiktoken in prompts module #2176)
• The edit dialog wasn't correctly showing the extracted text from assets, this is now fixed (OCR works but doesn't populate the "extracted content" textbox #2181).
• IP validation allowlisting now allows bypassing all domains by setting CRAWLER_ALLOWED_INTERNAL_HOSTNAMES to ..
• Fix a worker crash when hitting invalid URLs with proxy enabled.

For Developers 🛠️

• GET /api/version endpoint for getting server version (feat: add GET /api/version endpoint #2167)
• More visibility: HTTP status Prometheus counters, failed_permanent worker metric, and system metrics on web/worker containers (feat: add Prometheus counter for HTTP status codes #2117, feat: add last failure timestamp metric for worker monitoring #2107)
• Documentation updates for LOG_LEVEL and Raycast links (Add LOG_LEVEL to configuration documentation #2166, Update 13-community-projects.md #1923) by @aa-ko and @fivestones

Screenshots 📸

Collaborative Lists

Automated Backups

Search Autocomplete

Upgrading 📦

To upgrade:

• If you're using KARAKEEP_VERSION=release, run docker compose pull && docker compose up -d.
• If you're pinning it to a specific version, bump the version and then run docker compose pull && docker compose up -d.

All Commits

• i18n: fix en_US translation - @MohamedBassem in f01d96f
• i18n: Sync weblate translations - @HosteD Weblate in e1ad2cf
• feat: autocomplete search terms (refactor(web): split search autocomplete logic #2178) - @MohamedBassem in ebafbe5
• build: switch npm to trusted publishing - @MohamedBassem in 335a84b
• feat: Add automated bookmark backup feature (feat: Add automated bookmark backup feature #2182) - @MohamedBassem in 86a4b39
• fix: making serverConfig readonly - @MohamedBassem in e67c33e
• fix: fix react errors in signin and signup forms - @MohamedBassem in 6ab7984
• fix: separate shared lists in the sidebar (fix: separate shared lists in the sidebar #2180) - @MohamedBassem in 2619f4c
• fix: correctly render asset extracted text in the edit bookmark dialog. fixes OCR works but doesn't populate the "extracted content" textbox #2181 - @MohamedBassem in 9ed338f
• fix: lazy load js-tiktoken in prompts module (fix: lazy load js-tiktoken in prompts module #2176) - @MohamedBassem in e2877b4
• fix: fix colors in invitation form - @MohamedBassem in a13a227
• fix: hide archived checkbox in shared lists - @MohamedBassem in adde809
• feat: improve font and colors of sidebar items - @MohamedBassem in 5bea5d3
• fix: Propagate group ids in queue calls (fix: Propagate group ids #2177) - @MohamedBassem in 6821257
• feat: Introduce groupId in restate queue (feat: Introduce groupId in restate queue #2168) - @MohamedBassem in 5426875
• fix: support invocation cancellation while awaiting sempahore - @MohamedBassem in 38842f7
• docs: Add LOG_LEVEL to configuration documentation (Add LOG_LEVEL to configuration documentation #2166) - @aa-ko in 6912d0d
• docs: fix link to raycast extension (Update 13-community-projects.md #1923) - @fivestones in 9fedfc1
• tests: Add a test for listing lists - @MohamedBassem in e16ae2a
• feat: add GET /api/version endpoint (feat: add GET /api/version endpoint #2167) - @MohamedBassem in 472adec
• fix(mcp): propagate parent id to createList call. fixes: MCP create-list tool ignores parentId parameter #2144 - @MohamedBassem in 0d14130
• feat(mobile): proper handling for shared list permissions (feat(mobile): proper handling for shared list permissions #2165) - @MohamedBassem in c5c71ba
• feat(mobile): Add highlights page to mobile app (feat(mobile): Add highlights page to mobile app #2156) - @MohamedBassem in 8a5a109
• feat: A better looking 404 page - @MohamedBassem in 7f555f5
• fix: hide manage collaborators option for smart lists - @MohamedBassem in 2b38c00
• fix: Hide shared lists where user is a viewer in Manage Lists dialog (fix: Hide viewer-only shared lists in Manage Lists #2164) - @MohamedBassem in e4db9bf
• feat(mobile): Add AI summary field to mobile bookmark info (feat: Add AI summary field to mobile bookmark info #2157) - @MohamedBassem in 45081dc
• feat(mobile): Add tags screen to mobile app (feat: Add tags screen to mobile app #2163) - @MohamedBassem in ad66f78
• feat: Add notes feature to highlights (feat: Add notes feature to highlights #2154) - @MohamedBassem in de5ebbc
• feat(mobile): Add smart list creation in mobile app (feat(mobile): Add smart list creation in mobile app #2153) - @MohamedBassem in 48ab8a1
• feat: Add search bar to highlights page (feat: Add search bar to highlights page #2155) - @MohamedBassem in ed6a3bf
• fix: hide collaborator emails from non-owners (fix: hide collaborator emails from non-owners #2160) - @MohamedBassem in 8ab5df6
• feat: Add invitation approval for shared lists (feat: Add invitation approval for shared lists #2152) - @MohamedBassem in 5f0934a
• deps: upgrade oxlint - @MohamedBassem in daee8e7
• fix: add a way to allowlist all domains from ip validation - @MohamedBassem in 67b8a3c
• fix: use kbd for editor card - @MohamedBassem in 3345377
• fix: drop journal retention for sempahore and id providers - @MohamedBassem in 1b44eaf
• refactor: remove the PrivacyAware interface - @MohamedBassem in 815e196
• feat: Add collaborative lists (feat: Add collaborative lists #2146) - @MohamedBassem in 88c73e2
• deps: upgrade hono and playwright - @MohamedBassem in cc8fee0
• deps: Upgrade typescript to 5.9 - @MohamedBassem in 391af8a
• build: Improve docker caching (build: Improve docker caching #2140) - @MohamedBassem in 6cccb9f
• fix: fix hydration error in admin user list - @MohamedBassem in 12d09a7
• feat: import from mymind (feat: import from mymind #2138) - @MohamedBassem in 0c80f51
• feat: add Prometheus counter for HTTP status codes (feat: add Prometheus counter for HTTP status codes #2117) - @MohamedBassem in 4350666
• fix(mobile): upgrade react-native-pdf to v7 to fix page alignment - @MohamedBassem in 4c6ef25
• fix(mobile): fix app memory page size compatibility (fix(mobile): fix app memory page size compatibility #2135) - @MohamedBassem in fbd12ea
• release(mobile): Bump mobile version to 1.8.2 - @MohamedBassem in 76c291a
• fix: remove incorrect array destructuring in mobile search (fix: remove incorrect array destructuring in mobile search #2124) - @MohamedBassem in 07390ae
• feat: correct default prom metrics from web and worker containers - @MohamedBassem in c34f70d
• fix: stop retrying indefinitely in restate queues - @MohamedBassem in d4b7b89
• fix: fix crash in crawler on invalid URL in matchesNoProxy - @MohamedBassem in d0f71a4
• feat: add crawler domain rate limiting (feat: add crawler domain rate limiting #2115) - @MohamedBassem in 4cf0856
• refactor: Allow runner functions to return results to onComplete - @MohamedBassem in b28cd03
• refactor: Extract ratelimiter into separate plugin (refactor: Extract ratelimiter into separate plugin #2112) - @MohamedBassem in 0316148
• feat(extension): Add custom header support for extension (feat(extension): Add custom header support for extension #2111) - @MohamedBassem in ec87813
• feat: Add bookmark sources statistics section (feat: Add bookmark sources statistics section #2110) - @MohamedBassem in 725b521
• feat: Add page titles (feat: Add page titles #2109) - @MohamedBassem in 3083be0
• feat: add failed_permanent metric for worker monitoring (feat: add last failure timestamp metric for worker monitoring #2107) - @MohamedBassem in 1b8129a
• release(docs): release the 0.28 docs - @MohamedBassem in d9ef832
• release(extension): Release version 1.7.1 - @MohamedBassem in 7339d1d
• release(mobile): Bump mobile version to 1.8.1 - @MohamedBassem in f06b8ea

---

This discussion was created from the release 0.29.0.

[sprior]

Any suggested starting values for CRAWLER_DOMAIN_RATE_LIMIT_WINDOW_MS and CRAWLER_DOMAIN_RATE_LIMIT_MAX_REQUESTS?

[MohamedBassem]

Maybe 5 crawls / minute is a safe default? Something that you wouldn't hit normally unless you're doing a large import.

CRAWLER_DOMAIN_RATE_LIMIT_WINDOW_MS=60000
CRAWLER_DOMAIN_RATE_LIMIT_MAX_REQUESTS=5