A fast, persistent, process-safe key-value cache for Python (with a C++20 core).
Think of it as a tiny local database for expensive-to-recompute things: you store Python objects under string keys, and any thread or process on the machine can read them back — safely, even at the same time. It stays flat from 100 to 1M+ entries, is 2x faster than diskcache on writes and up to 6x faster in batched transactions, and it never corrupts your data when processes crash, fork, or race each other.
pip install pysciqlop-cache📖 Documentation: quickstart, guides for threads and processes, big numpy arrays and diskcache migration, API reference, C++ guide and internals.
from pysciqlop_cache import Cache
cache = Cache("/tmp/my-cache")
cache["sensor/temperature"] = {"ts": 1710000000, "values": [21.3, 21.5, 21.4]}
print(cache["sensor/temperature"])
# {'ts': 1710000000, 'values': [21.3, 21.5, 21.4]}Any picklable Python object works out of the box.
- Persistent — data survives process restarts; open the same directory and it's all there.
- Safe to share — multiple threads and multiple processes (including forked worker pools) can hit the same cache concurrently. No corruption, no lock files, no setup.
- Bounded — optional size limit with automatic LRU eviction, optional per-key expiration, and tag-based bulk eviction.
- Fast — small values live inside SQLite; large values are memory-mapped files, so reading a 100 MB array costs ~zero copies.
- Built for big arrays — the
PickleOOBSerializermoves numpy array bytes in C++ without holding the GIL, and compresses them when it pays off, so threads loading data don't block each other (details). - Self-healing — crashes, races and interrupted writes are detected and repaired automatically or via
cache.check(fix=True).
| Quickstart | dict API, expiration and tags, size limits, memoization, transactions |
| Choosing a store | Cache, Index, FanoutCache, FanoutIndex |
| Threads, processes and transactions | what is safe, what runs in parallel, locks |
| Serializers | pickle, msgspec, and PickleOOBSerializer for big numpy arrays |
| Coming from diskcache | one-command migration, API differences |
| C++ guide | the same engine from C++20 |
| Performance | benchmarks against diskcache |
| How it works | storage, crash and fork safety, self-healing |
| API reference | every class and method |
Measured against diskcache, both with their default settings, on a RAM filesystem, on Linux (x86-64) and macOS (Apple M2). Numpy measurement arrays from 100 KB to 100 MB, and 1 to 16 threads, on Linux:
More charts and the methodology are in the performance page.
pip install . # or, for development:
meson setup build -Dwith_tests=true
meson compile -C build
meson test -C buildSee installation for the build options.
MIT