Skip to content

Lowered the default value of CLUSTER_CACHE_SIZE#1038

Merged
kelson42 merged 1 commit intomainfrom
smaller_default_cluster_cache_size
Feb 3, 2026
Merged

Lowered the default value of CLUSTER_CACHE_SIZE#1038
kelson42 merged 1 commit intomainfrom
smaller_default_cluster_cache_size

Conversation

@veloman-yunkan
Copy link
Collaborator

!!! IMPORTANT !!!

For this change to take effect when rebuilding/recompiling the project in an existing build directory, the following command must be explicitly executed:

meson configure -D CLUSTER_CACHE_SIZE=16777216

The default value of CLUSTER_CACHE_SIZE was made low-end-device-friendly - the cluster cache max memory size was lowered from 512MiB to 16MiB. For mono-ZIM-file use cases (when a single ZIM-file is open at a time) this approximately returns the memory usage by the cluster cache to how it used to be before the cluster cache was made global (shared across all open ZIM-files).

Note however that there is some ambiguity with respect to what memory usage means - there is virtual memory vs physical memory. Current implementation of cluster cache memory management targets physical memory usage on systems supporting mmap, where allocation of virtual memory by the cluster cache can exceed the actual physical memory usage by a factor of ~5x-9x.

On systems not supporting mmap (or with an implementation of malloc not utilizing mmap for large allocations) the consumption of physical memory should equal the size of allocated virtual memory, and the actual memory usage will exceed the cache memory limit by the said factor, i.e. as much as 144MiB can be consumed by the cluster cache at the default setting.

!!! IMPORTANT !!!

For this change to take effect when rebuilding/recompiling the project in an
existing build directory, the following command must be explicitly
executed:

```
meson configure -D CLUSTER_CACHE_SIZE=16777216
```

---

The default value of CLUSTER_CACHE_SIZE was made low-end-device-friendly
- the cluster cache max memory size was lowered from 512MiB to 16MiB.
For mono-ZIM-file use cases (when a single ZIM-file is open at a time)
this approximately returns the memory usage by the cluster cache to
how it used to be before the cluster cache was made global (shared
across all open ZIM-files).

Note however that there is some ambiguity with respect to what memory
usage means - there is virtual memory vs physical memory. Current
implementation of cluster cache memory management targets physical
memory usage on systems supporting mmap, where allocation of virtual
memory by the cluster cache can exceed the actual physical memory usage
by a factor of ~5x-9x.

On systems not supporting mmap (or with an implementation of malloc not
utilizing mmap for large allocations) the consumption of physical memory
should equal the size of allocated virtual memory, and the actual memory
usage will exceed the cache memory limit by the said factor, i.e. as
much as 144MiB can be consumed by the cluster cache at the default
setting.
@codecov
Copy link

codecov bot commented Feb 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.25%. Comparing base (388010e) to head (7466066).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1038   +/-   ##
=======================================
  Coverage   56.25%   56.25%           
=======================================
  Files         101      101           
  Lines        5016     5016           
  Branches     2185     2185           
=======================================
  Hits         2822     2822           
  Misses        738      738           
  Partials     1456     1456           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kelson42 kelson42 added the bug label Feb 3, 2026
@kelson42 kelson42 merged commit 1998b31 into main Feb 3, 2026
27 checks passed
@kelson42 kelson42 deleted the smaller_default_cluster_cache_size branch February 3, 2026 16:16
@kelson42 kelson42 added this to the 9.5.0 milestone Feb 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants