Skip to content

[WIP][Feature][Bugfix] WandB setup/finish simplified + callback. - #381

Closed
sami-bg wants to merge 2 commits into
galilai-group:mainfrom
sami-bg:wandb-finish-setup-consistency
Closed

[WIP][Feature][Bugfix] WandB setup/finish simplified + callback.#381
sami-bg wants to merge 2 commits into
galilai-group:mainfrom
sami-bg:wandb-finish-setup-consistency

Conversation

@sami-bg

@sami-bg sami-bg commented Feb 14, 2026

Copy link
Copy Markdown
Contributor

Description

Extracts all wandb lifecycle logic (init, config sync, offline data dump, wandb.finish()) from Manager into a new WandbCallback that uses Lightning's setup()/teardown() hooks, so wandb.finish() is called automatically after each trainer stage. This fixes run leaks in multirun sweeps (fit then test) without requiring manual cleanup

Moves WandbCheckpoint out of checkpoint_sklearn.py into wandb_lifecycle.py alongside WandbCallback, and fixes its docstring which was copy-pasted from the sklearn checkpoint

Adds unit tests covering config flattening, offline file I/O, SLURM requeue directory discovery, setup/teardown state management, on_exception crash recovery, and checkpoint run-ID continuity

Checklist

  • I have read the Contributing document.
  • The documentation is up-to-date with the changes I made (check build artifacts).
  • All tests passed, and additional code has been covered with new tests.
  • I have added the PR to the RELEASES.rst file.

@RandallBalestriero

Copy link
Copy Markdown
Contributor

Nice, thanks @sami-bg ! did you try running with DDP and preemption/requeue to make sure everything is safely dumped/loaded re-init every time?

@sami-bg sami-bg changed the title [Feature][Bugfix] WandB setup/finish simplified + callback. [WIP][Feature][Bugfix] WandB setup/finish simplified + callback. Feb 14, 2026
@sami-bg

sami-bg commented Feb 14, 2026

Copy link
Copy Markdown
Contributor Author

Not yet! Sorry forgot to add [WIP]. Will test this soon

@RandallBalestriero

Copy link
Copy Markdown
Contributor

Hi @sami-bg !

I did an update on wandb callbacks in the last few merge, could you please double check and if spt now has all the functionalities you were implementing, please close this PR, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants