-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Open
Milestone
Description
We're seeing read latency spikes (very high read latency p95, up to the ~300 ms range) in our Scylla backend.
These seem to be correlated to blob read operations like contains_blob
.
We suspect that what's happening is that:
- blob queries are relatively slow, because blobs are big
- all blobs are assigned the same
root_key
, which we use as a Scylla partition key - being on the same partition means that all blob queries are handled on one CPU, where they block all other queries to that CPU
- this results in the queries at the tail of the query queue being serviced very late
As a solution, we suggest setting the partition key of blobs to be just the blob ID.
This is a different behaviour from that expected by linera-views
(which correlates things under the same root_key
, e.g. chain ID, for fast multi-key queries on related objects), so we'd like to pull out the ‘key/value store abstraction’ layer (linera_views::backend
) from the ‘persistent views’ layer of linera-views
and use that directly for blobs.
Metadata
Metadata
Assignees
Labels
No labels