Add save_on_exception option to ModelCheckpoint

[vsey]

What does this PR do?
This PR adds a save_on_exception option to the ModelCheckpoint callback. This some of this functionality is already implemented in the OnExceptionCheckpoint checkpoint, but I believe that bundling all checkpoint options in the ModelCheckpoint is more intuitive. Additionally, this leads to the same naming conventions and directory paths used for the exception checkpoint as for all the others.
When enabled, this option serves as a contingency in case of any disruption during training, allowing one to continue from the last step before the exception occurs without losing too much progress. By printing the exception type and message, this also alleviates issue #20187.

Fixes #19686

Before submitting

• Was this discussed/agreed via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

---

📚 Documentation preview 📚: https://pytorch-lightning--20916.org.readthedocs.build/en/20916/

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79%. Comparing base (3138305) to head (7dfdbc5).
Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (3138305) and HEAD (7dfdbc5). Click for more details.

HEAD has 352 uploads less than BASE

Flag	BASE (3138305)	HEAD (7dfdbc5)
gpu	3	0
pytest	55	0
lightning_fabric	24	0
lightning	61	15
cpu	106	27
python3.12	11	3
python3.12.7	35	9
python3.11	24	6
python	12	3
python3.10	24	6
pytorch2.1	12	6
pytorch_lightning	24	12
pytest-full	54	27
pytorch2.3	6	3
pytorch2.5	6	3
pytorch2.2.2	6	3
pytorch2.4.1	6	3
pytorch2.7	6	3
pytorch2.5.1	6	3
pytorch2.6	6	3

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #20916     +/-   ##
=========================================
- Coverage      87%      79%     -8%     
=========================================
  Files         268      265      -3     
  Lines       23442    23399     -43     
=========================================
- Hits        20394    18398   -1996     
- Misses       3048     5001   +1953     

[Borda]

Suggested change

	def test_model_checkpoint_on_exception_run_condition(tmp_path):
	"""Test that the checkpoint is saved when an exception is raised in a lightning module."""
	# Don't save checkpoint if sanity check fails
	class TroubledModelSanityCheck(BoringModel):
	def on_validation_start(self) -> None:
	if self.trainer.sanity_checking:
	print("Trouble!")
	raise RuntimeError("Trouble!")
	model = TroubledModelSanityCheck()
	checkpoint_callback = ModelCheckpoint(dirpath=tmp_path, filename="sanity_check", save_on_exception=True)
	trainer = Trainer(
	default_root_dir=tmp_path,
	num_sanity_val_steps=4,
	limit_train_batches=2,
	callbacks=[checkpoint_callback],
	max_epochs=2,
	logger=False,
	)
	with pytest.raises(RuntimeError, match="Trouble!"):
	trainer.fit(model)
	assert not os.path.isfile(tmp_path / "exception-sanity_check.ckpt")
	# Don't save checkpoint if fast dev run fails
	class TroubledModelFastDevRun(BoringModel):
	def on_train_batch_start(self, batch, batch_idx) -> None:
	if self.trainer.fast_dev_run and batch_idx == 1:
	raise RuntimeError("Trouble!")
	model = TroubledModelFastDevRun()
	checkpoint_callback = ModelCheckpoint(dirpath=tmp_path, filename="fast_dev_run", save_on_exception=True)
	trainer = Trainer(
	default_root_dir=tmp_path,
	fast_dev_run=2,
	limit_train_batches=2,
	callbacks=[checkpoint_callback],
	max_epochs=2,
	logger=False,
	)
	with pytest.raises(RuntimeError, match="Trouble!"):
	trainer.fit(model)
	assert not os.path.isfile(tmp_path / "exception-fast_dev_run.ckpt")
	# Don't save checkpoint if already saved a checkpoint
	class TroubledModelAlreadySavedCheckpoint(BoringModel):
	def on_train_batch_start(self, batch, batch_idx) -> None:
	if self.trainer.global_step == 1:
	raise RuntimeError("Trouble!")
	model = TroubledModelAlreadySavedCheckpoint()
	checkpoint_callback = ModelCheckpoint(
	dirpath=tmp_path, filename="already_saved", save_on_exception=True, every_n_train_steps=1
	)
	trainer = Trainer(
	default_root_dir=tmp_path, limit_train_batches=2, callbacks=[checkpoint_callback], max_epochs=2, logger=False
	)
	with pytest.raises(RuntimeError, match="Trouble!"):
	trainer.fit(model)
	assert not os.path.isfile(tmp_path / "exception-already_saved.ckpt")
	assert os.path.isfile(tmp_path / "already_saved.ckpt")

This seems to be the same as the parametrization bellow