How to release ram cache of CacheDataset?

[cvlearn913]

I need to use cachedataset in turn in my program，But just using the del method cannot release the memory occupied by this object

del self.train_fold

This will result in the following error

  File "/home/wanyi/anaconda3/envs/monai/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 226264) is killed by signal: Killed. 

[Nic-Ma]

Hi @cvlearn913 ,

Thanks for your interest here.
Could you please help share your full test program? I don't quite understand your expected behavior.

Thanks.

[cvlearn913]

When I perform cross-validation, I need to reset the training set and validation set, and using CacheDataset will cause insufficient memory error like above. It seemd that the occupied memory by self.train_fold cannot be released by the del method

    def setup_fold_index(self, fold_index: int) -> None:
        train_indices, val_indices = self.splits[fold_index]
        del self.train_fold
        del self.val_fold
        self.train_fold = CacheDataset(data=np.array(self.train_files)[train_indices],transform=self.train_transforms,cache_rate=1.0)#Not enough memory
        #self.train_fold = Dataset(data=np.array(self.train_files)[train_indices],transform=self.train_transforms) #it works
        self.val_fold = Dataset(data=np.array(self.train_files)[val_indices],transform=self.val_transforms)

[Nic-Ma]

I think every time calling this self.train_fold = CacheDataset() will replace the previous object with a new one, so maybe no need to care about the python memory?

Thanks.

[cvlearn913]

Yes, it should be like this in theory, but it will report an out of memory error, which is very strange

[Nic-Ma]

Could you please help provide the full program for further analysis if necessary?

Thanks.

[cvlearn913]

We are grateful to provide it if possible, but the entire project involves sensitive information, and it is not possible to provide the full program for the time being. Thanks for your reply ！

[hooman-b]

I have also the same problem with CacheDataset. I do not know how can I free the RAM memory after each cross validation iteration!