Skip to content

Add bounded concurrent syncing with --max-workers#126

Draft
pierreadorni wants to merge 6 commits intoklieret:mainfrom
pierreadorni:main
Draft

Add bounded concurrent syncing with --max-workers#126
pierreadorni wants to merge 6 commits intoklieret:mainfrom
pierreadorni:main

Conversation

@pierreadorni
Copy link
Copy Markdown

I often have multiple runs at the same time, and found that one long wandb sync would be blocking the sync of my other (shorter) runs. I therefore added a concurrency feature to allow for syncing multiple runs at the same time.

Summary

  • Add bounded concurrent syncing in the watcher to avoid one long wandb sync blocking other runs.
  • Expose --max-workers CLI option (default 1, preserves current behavior).
  • Document the new option and add a changelog entry.
  • Add tests for concurrency scheduling and invalid max_workers.

Possible improvement

The new concurrency kind of messes up the display when multiple wandb sync processes write to stdout at the same time. This could be solved by capturing the output of the wandb sync command and printing our own info in a more concurrency-friendly manner, but I chose to not do it as to limit code changes to the minimum.

@semanticdiff-com
Copy link
Copy Markdown

semanticdiff-com Bot commented Feb 6, 2026

Review changes with  SemanticDiff

Changed Files
File Status
  src/wandb_osh/syncer.py  48% smaller
  CHANGELOG.md Unsupported file format
  README.md Unsupported file format
  src/wandb_osh/cli.py  0% smaller
  tests/test_syncer.py  0% smaller

@pierreadorni pierreadorni marked this pull request as draft February 15, 2026 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant