overlap: use overlapcache - #3639
Conversation
This commit implements a new overlap cache data structure which will be embedded into `FileMetadata`. The cache works by remembering a handful of data regions and whether the spaces in-between are known to be empty. The goal is to make repeated overlap checks in the same area of a file much cheaper. This will make it feasible to have an optimistic overlap check that can be repeated on a slightly changed version without redoing most of the work.
|
This change is |
We embed an overlap cache in `FileMetadata` and use it with the overlap checker. When we have to open a table, we now try to find a maximal empty region and the two surrounding data regions so that we can populate the cache with this information. The cache will be very useful for optimistic overlap checks which happen without blocking compactions, and might need to be retried.
|
Friendly ping. |
jbowens
left a comment
jbowens
left a comment
There was a problem hiding this comment.
Is the memory overhead of maintaining per-FileMetadata cache sustainable? It's not uncommon to have ~500k-1m sstables in a LSM version
Reviewable status: 0 of 8 files reviewed, all discussions resolved (waiting on @itsbilal and @sumeerbhola)
RaduBerinde
left a comment
RaduBerinde
left a comment
There was a problem hiding this comment.
Is the memory overhead of maintaining per-FileMetadata cache sustainable? It's not uncommon to have ~500k-1m sstables in a LSM version
In the common case of mostly points, we will have at most 6 keys per FileMetadata. I assumed that's not a huge problem given that we have ~4 keys already for the smallest/largest point/range keys. I can reduce the cache to 4 and it should still be very beneficial.
Or maybe I should at moving towards a separate LRU cache for a limited number of tables?
Reviewable status: 0 of 8 files reviewed, all discussions resolved (waiting on @itsbilal and @sumeerbhola)
|
We can also greatly reduce the |
|
Some higher level thoughts from a discussion with @sumeerbhola - I might hold off on this and get more data to establish how effective/necessary this cache is. |
3 participants
overlap: add overlapcache subpackage
This commit implements a new overlap cache data structure which will
be embedded into
FileMetadata.The cache works by remembering a handful of data regions and whether
the spaces in-between are known to be empty. The goal is to make
repeated overlap checks in the same area of a file much cheaper. This
will make it feasible to have an optimistic overlap check that can be
repeated on a slightly changed version without redoing most of the
work.
overlap: use overlapcache
We embed an overlap cache in
FileMetadataand use it with theoverlap checker.
When we have to open a table, we now try to find a maximal empty
region and the two surrounding data regions so that we can populate
the cache with this information.
The cache will be very useful for optimistic overlap checks which
happen without blocking compactions, and might need to be retried.