Repository navigation
feat: Persistent storage layer using RocksDB - #12
Merged
Merged
Conversation
- Mark ROCKS-001 as 100% complete - Set up ROCKS-002 ColumnFamily enum task - Update progress tracking metrics 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implements persistent storage using RocksDB with 6 column families for system and data Raft logs, state, and key-value storage. - CRUD operations per column family - Atomic batch writes across multiple CFs - Raft log operations (append, range, truncate) - OpenRaft 0.9 trait integration - Snapshot creation/restoration via checkpoints - Storage metrics tracking Targets <1ms p99 latency for single key ops.
- Fix unused imports across all test files - Replace bool assert_eq with assert! macro (clippy::bool-assert-comparison) - Use struct init syntax instead of field reassign with default - Remove clone() on Copy types (ColumnFamily) - Add #[allow(dead_code)] for unused test strategies - Rename iterator.next/prev to step_forward/step_backward (deprecation) - Add parking_lot RwLock instead of std::sync::RwLock - Add mise tasks for check/fmt/lint/test per crate and workspace - Update AGENTS.md with skill loading prompts and mise tasks reference
- Add parameterized RaftGroup trait for SystemRaft and DataRaft - Implement RocksDBLogStorage<G> with RaftLogStorage trait - Implement RocksDBStateMachine<G> with RaftStateMachine trait - Add protobuf serialization for Vote, LogId, and Membership - Remove config field from Storage to enable Send+Sync - Expose update_cached_last_log_index as public method - Add unit tests for serialization roundtrips
Critical fixes: - Fix N+1 query in get_log_range using iterator range scan - Add 16MB size limits to protobuf decode calls - Fix Operation deserialization using proper bincode (not 32-byte slice) - Implement delete operations in state machine apply() High priority fixes: - Add voted_for field to VoteMessage for complete vote persistence - Preserve log metadata (term, node_id) instead of using zeros - Add bincode size limits to prevent OOM from large operations Medium priority fixes: - Add snapshot path validation to prevent path traversal - Extract StorageBackend trait for testing abstraction - Fix CFOptions Clone to preserve prefix length Tests: 41 passing, clippy clean
- Fix entry type marker for membership serialization (CRITICAL) - Add apply() error propagation instead of silent failure (CRITICAL) - Fix cache invalidation after truncate operations (HIGH) - Add iterator safety checks to prevent panic on compaction (HIGH) - Use batch writes in RaftLogStorage::append for atomicity (HIGH) - Validate snapshot checkpoints before install (MEDIUM) - Add TOCTOU mitigation with directory sync (MEDIUM) - Implement find_last_log_entry fallback (MEDIUM) - Add batch size limit to prevent memory exhaustion (MEDIUM) - Cap max_open_files to 1024 (LOW) Closes code review findings from PR #14
- Add Clone derive to RocksDBLogStorage and RocksDBStateMachine - Export LogIdMessage for test usage - Add integration tests for RaftLogStorage and RaftStateMachine traits - Test vote, log entries, and state persistence across storage restart - Test Set/Del operations and state machine apply Closes seshat-5dw
Replace N individual get() calls with single iterator-based range scan. O(n) RocksDB reads → O(1) seek + O(n) in-memory iteration. For 100K entry log compaction this reduces ~100K syscalls to ~1.
- Add type alias MockData to fix clippy type_complexity warning - Run rustfmt on openraft_rocksdb.rs and openraft_integration_tests.rs - Improve code readability with consistent formatting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements a persistent storage layer using RocksDB, enabling cluster recovery after restarts and maintaining consistency across nodes.
Key Changes:
thiserrorand rich context propagation