Jules was unable to complete the task in time. Please review the work… #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… done so far and provide feedback for Jules to continue.
This commit addresses potential memory issues and ensures the ParameterPredictor is trained correctly in the adaptive superpixels feature.
Changes include:
1. **Engine (`engine.py`):**
* Removed `torch.no_grad()` from the `ParameterPredictor` call during training in `train_one_epoch`. This allows the predictor to receive gradients and learn from the main task loss, as originally intended for joint optimization.
* Moved the `ParameterPredictor` forward pass into the `torch.cuda.amp.autocast()` context to enable mixed-precision training for the predictor, potentially reducing its memory footprint and speeding up its execution.
* Added explicit `del` statements for `pred_k`, `pred_m`, and `spix_id` tensors at the end of each training iteration where they are dynamically generated. This encourages more prompt garbage collection.
2. **Datasets (`datasets.py`):**
* In `generate_superpixels`, added explicit `del` statements for intermediate CPU-bound NumPy arrays (`img_for_spix_normalized_cpu`, `img_for_spix_denormalized`, `img_for_spix_numpy`, `img_for_spix_rescaled`, `assignment`) and the `slic_engine` object at the end of each iteration within the batch processing loop. This is a precautionary measure to help CPU-side memory management.
These changes aim to make the training process more robust, particularly when the adaptive superpixels feature is enabled and the `ParameterPredictor` is being trained. The primary cause of previously reported memory hangs was likely the increased memory demand from enabling `ParameterPredictor` training, and these modifications ensure it's done correctly within typical PyTorch paradigms like AMP, along with more explicit memory handling.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… done so far and provide feedback for Jules to continue.