RFC: Key-Value Separation for Mini-LSM#169
Closed
ben1009 wants to merge 98 commits intoskyzh:mainfrom
Closed
Conversation
* chore: typos * merge starter code
* w1d4 refactor
* chore: typos & refine comments (#65) * typo in week2-01-compaction.md * chroe: typos & add comments * chore: more typos * Update week2-01-compaction.md --------- Co-authored-by: Alex Chi Z <iskyzh@gmail.com> * Fix typos in W3D5 writeup and code (#67) * Fix minor mistake in W3D6 writeup (#69) --------- Co-authored-by: Alex Chi Z <iskyzh@gmail.com> Co-authored-by: Yue Yin <41224888+yyin-dev@users.noreply.github.com>
* chore: fix lint
* chore: optimize overlap_len
* chore: add comment about blockcache * chore: fix ci
feature: add writebatch
Bumps [crossbeam-channel](https://github.com/crossbeam-rs/crossbeam) from 0.5.14 to 0.5.15. - [Release notes](https://github.com/crossbeam-rs/crossbeam/releases) - [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md) - [Commits](crossbeam-rs/crossbeam@crossbeam-channel-0.5.14...crossbeam-channel-0.5.15) --- updated-dependencies: - dependency-name: crossbeam-channel dependency-version: 0.5.15 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
chore: fix lints
…l-0.5.15 chore(deps): bump crossbeam-channel from 0.5.14 to 0.5.15
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
feat: fix lint
Reverts all changes to mini-lsm-starter from PR #69 to keep starter code clean for course participants.
Added explicit lifetime annotation to KeySlice return type
deps: sync with upstream skyzh/mini-lsm and restore starter templates
deps: bump bytes from 1.10.1 to 1.11.1
This RFC proposes implementing WiscKey-style key-value separation for Mini-LSM, which stores large values in dedicated Value Log (vLog) files while keeping keys and value pointers in the LSM tree. Key benefits: - 5-10x reduction in write amplification for large-value workloads - Faster range scans (no need to scan through large values) - Better block cache efficiency The RFC includes: - Detailed architecture design with ValuePointer, ValueLog, and GC components - File format specifications for vLog entries - 4-phase implementation plan - Testing strategy and compatibility considerations - References to WiscKey, BadgerDB, RocksDB BlobDB, and Titan
Contributor
Author
|
Closing - opened to wrong repository. The correct PR is at ben1009#71 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This RFC proposes implementing WiscKey-style key-value separation for Mini-LSM, inspired by production systems like BadgerDB, RocksDB's BlobDB, and Titan.
Motivation
The current Mini-LSM architecture stores keys and values together in SSTable blocks, which leads to:
Key Benefits
Design Highlights
Implementation Plan
4-phase approach over ~4 weeks:
See the full RFC in for detailed design, API changes, and references.
Related Work:
cc: @skyzh