Skip to content

All blobs live on the same Scylla partition #4807

@Twey

Description

@Twey

We're seeing read latency spikes (very high read latency p95, up to the ~300 ms range) in our Scylla backend.

These seem to be correlated to blob read operations like contains_blob.

We suspect that what's happening is that:

  • blob queries are relatively slow, because blobs are big
  • all blobs are assigned the same root_key, which we use as a Scylla partition key
  • being on the same partition means that all blob queries are handled on one CPU, where they block all other queries to that CPU
  • this results in the queries at the tail of the query queue being serviced very late

As a solution, we suggest setting the partition key of blobs to be just the blob ID.

This is a different behaviour from that expected by linera-views (which correlates things under the same root_key, e.g. chain ID, for fast multi-key queries on related objects), so we'd like to pull out the ‘key/value store abstraction’ layer (linera_views::backend) from the ‘persistent views’ layer of linera-views and use that directly for blobs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions