fix: reduce virtual address space usage to support 32-bit (386) architectures - #2321
Open
adityeah8969 wants to merge 2 commits into
Open
fix: reduce virtual address space usage to support 32-bit (386) architectures#2321adityeah8969 wants to merge 2 commits into
adityeah8969 wants to merge 2 commits into
Conversation
|
|
matthewmcneely
force-pushed
the
fix/memory-leak-32-bit-machine
branch
from
August 12, 2026 20:48
ff53ba7 to
ef1c40d
Compare
On 32-bit systems each memtable WAL consumes 128 MB of virtual address space (2× MemTableSize). During write-heavy workloads multiple memtables can queue in db.imm before the background flusher processes them, exhausting the ~3 GB user-space limit. Add releaseWAL() which munmaps the WAL data and closes the fd after the memtable becomes immutable — all reads go through the skiplist, so the mmap'd region is dead weight. The OnClose callback is replaced with a simple os.Remove since the file descriptor is already closed. Called at four sites where memtables transition to immutable: - openMemTables (recovery path) - ensureRoomForWrite (write-path rotation) - memory flush handler - DropPrefix Co-Authored-By: Claude <noreply@anthropic.com>
Two changes to prevent virtual address space exhaustion on 32-bit: 1. Reduce vlog file mmap from 2× to 1× ValueLogFileSize. The 2× multiplier was dead headroom — each 4 MB vlog file consumed 8 MB of address space, and with hundreds of files the total exceeded the 3 GB 32-bit limit. 2. Add a pre-emptive rotation check before each vlog entry write. When an entry would straddle the file boundary, rotate to a fresh file first so the entry lands cleanly. This avoids the need for dynamic mmap growth via Truncate/mremap, which can fail on 32-bit under address pressure. The rotation logic is extracted into a shared rotateVlog helper used by both the pre-rotation check and the existing toDisk(). Giant entries that exceed the file size still fall through to the existing Truncate path. Co-Authored-By: Claude <noreply@anthropic.com>
adityeah8969
force-pushed
the
fix/memory-leak-32-bit-machine
branch
from
August 23, 2026 20:08
ef1c40d to
b12f3c0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes: #2287
Badger's default configuration consumes >4 GB of virtual address space during write-heavy workloads, exceeding the ~3 GB user-space limit on 32-bit systems. This causes mmap failures with ENOMEM despite abundant physical RAM.
This PR makes two targeted changes to bring the working set under the 32-bit ceiling without affecting 64-bit behavior.
Each memtable WAL is mmap'd at 2× MemTableSize (128 MB). During writes, multiple memtables queue in db.imm before the background flusher drains them — all holding their WAL mmap simultaneously.
Vlog files were mmap'd at 2× their configured size. With a 4 MB vlog file, hundreds of files accumulate during a 2 GB workload, consuming 8 MB each (4 GB total).
Checklist