Skip to content

fix(bufferedwrites): prevent memory leak & race in UploadHandler.Destroy - #5026

Open
meet2mky wants to merge 1 commit into
masterfrom
fix-uploadhandler-destroy-race
Open

fix(bufferedwrites): prevent memory leak & race in UploadHandler.Destroy#5026
meet2mky wants to merge 1 commit into
masterfrom
fix-uploadhandler-destroy-race

Conversation

@meet2mky

@meet2mky meet2mky commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Description

When a file handle is destroyed (e.g., after a failed upload or unlink), UploadHandler.Destroy did not cancel the ongoing GCS upload context or wait for the background uploader goroutine to finish processing its current chunk.

This caused the uploader goroutine to return its in-flight block back to the block pool after ClearFreeBlockChannel had already executed and finished teardown. In production, this resulted in severe consequences:

  1. Memory Leaks: The late-arriving block was abandoned in the pool, meaning Deallocate() was never called and the underlying mmap memory was never freed.
  2. Global Write Degredation: The global semaphore permit (globalMaxBlocksSem) tied to the abandoned block was never released. Over time, these leaked permits would exhaust the semaphore capacity, causing all new writes across the GCSFuse mount to use staged writes.

This PR fixes the teardown sequence and synchronization:

  • Cancels the GCS upload context inside Destroy() to quickly abort the in-flight upload.
  • Adds uh.wg.Wait() after draining uploadCh to ensure the uploader goroutine fully completes and releases its block before teardown continues.
  • Introduces TestDestroyWithUploaderGoroutineInProgress to assert strict synchronization between Destroy and the uploader.

How Has This Been Tested?

  • Ran go test -v -count=100 -run TestBufferedWriteTestSuite/TestDestroyShouldClearFreeBlockChannel ./internal/bufferedwrites/... and verified it passes 100% of the time.
  • Implemented and verified the new TestDestroyWithUploaderGoroutineInProgress unit test to directly trigger and assert this race condition.
  • Confirmed that the new test fails without the fix and succeeds with it.

Link to the issue in case of a bug fix.

b/546299575

Testing details

  1. Manual - NA
  2. Unit tests - NA
  3. Integration tests - NA

Any backward incompatible change? If so, please explain.

NA

@meet2mky
meet2mky requested a review from a team as a code owner August 14, 2026 03:40
@meet2mky meet2mky added execute-integration-tests Run only integration tests execute-integration-tests-on-zb To run E2E tests on zonal bucket. labels Aug 14, 2026
@github-actions github-actions Bot added the remind-reviewers Auto remind reviewers in attention set for review post 24hrs of inactivity on PR. label Aug 14, 2026
@meet2mky
meet2mky force-pushed the fix-uploadhandler-destroy-race branch from 90601ec to 66b3bac Compare August 14, 2026 03:40

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Destroy method in UploadHandler to cancel ongoing GCS uploads and wait for the uploader goroutine to finish processing in-flight blocks. It also adds a unit test to verify this behavior. The review feedback points out that using time.Sleep in the new unit test to synchronize goroutines is an anti-pattern that can cause flakiness, and suggests using a channel-based synchronization approach instead.

Comment thread internal/bufferedwrites/upload_handler_test.go Outdated
@codecov

codecov Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.91%. Comparing base (3eb9f35) to head (976c7f0).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5026      +/-   ##
==========================================
+ Coverage   83.83%   83.91%   +0.08%     
==========================================
  Files         174      174              
  Lines       21378    21441      +63     
==========================================
+ Hits        17923    17993      +70     
+ Misses       2776     2758      -18     
- Partials      679      690      +11     
Flag Coverage Δ
unittests 83.91% <100.00%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@meet2mky
meet2mky force-pushed the fix-uploadhandler-destroy-race branch from 66b3bac to 73f4253 Compare August 14, 2026 04:29
…ndler.Destroy

When a file handle is destroyed (e.g., after a failed upload or unlink), UploadHandler.Destroy did not cancel the ongoing GCS upload context or wait for the background uploader goroutine to finish.

This caused the uploader goroutine to return its in-flight block to the block pool *after* ClearFreeBlockChannel had already executed. In production, this resulted in:
1. Memory Leaks: The late-arriving block was never munmap'ed.
2. Write Stalls: The global semaphore permit (globalMaxBlocksSem) for the block was never released, eventually exhausting the semaphore and causing all new writes on the mount to hang indefinitely.
3. Wasted Bandwidth: GCS uploads continued running in the background for closed/aborted files.

This commit fixes the teardown sequence:
- Cancels the GCS upload context inside Destroy() to quickly terminate the in-flight upload.
- Adds uh.wg.Wait() to ensure the uploader goroutine completes and releases its block before the block pool is cleared.
- Introduces TestDestroyWithUploaderGoroutineInProgress to assert correct synchronization.
@meet2mky
meet2mky force-pushed the fix-uploadhandler-destroy-race branch from 73f4253 to 976c7f0 Compare August 14, 2026 04:30
@meet2mky meet2mky changed the title fix(bufferedwrites): prevent memory leak and write stalls in UploadHandler.Destroy fix(bufferedwrites): prevent memory leak & race in UploadHandler.Destroy Aug 14, 2026
@github-actions

Copy link
Copy Markdown

Hi @vadlakondaswetha, @vipnydav, @abhishek10004, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

1 similar comment
@github-actions

Copy link
Copy Markdown

Hi @vadlakondaswetha, @vipnydav, @abhishek10004, your feedback is needed to move this pull request forward. This automated reminder was triggered because there has been no activity for over 24 hours. Please provide your input when you have a moment. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

execute-integration-tests Run only integration tests execute-integration-tests-on-zb To run E2E tests on zonal bucket. remind-reviewers Auto remind reviewers in attention set for review post 24hrs of inactivity on PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant