Run inference script crashes

[JMBokhorst]

Hi all,

I have tried to run the Pytorch version after I initially tried with the Tensorflow version. I tried to run the inference script in wsi mode with a ndpi image. It start correct but mid-way through the process I got this error:

Process Chunk 48/99:  61%|#############5        | 35/57 [02:19<01:11,  3.23s/it]|2021-01-06|13:06:15.182| [ERROR] Crash
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 779, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/local/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/local/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 282, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/local/lib/python3.7/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/local/lib/python3.7/multiprocessing/reduction.py", line 185, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/local/lib/python3.7/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 803, in _try_get_data
    fs = [tempfile.NamedTemporaryFile() for i in range(fds_limit_margin)]
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 803, in <listcomp>
    fs = [tempfile.NamedTemporaryFile() for i in range(fds_limit_margin)]
  File "/usr/local/lib/python3.7/tempfile.py", line 547, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/local/lib/python3.7/tempfile.py", line 258, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
OSError: [Errno 24] Too many open files: '/tmp/tmpxrmts9vn'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hover_net-master/infer/wsi.py", line 746, in process_wsi_list
    self.process_single_file(wsi_path, msk_path, self.output_dir)
  File "/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hover_net-master/infer/wsi.py", line 550, in process_single_file
    self.__get_raw_prediction(chunk_info_list, patch_info_list)
  File "/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hover_net-master/infer/wsi.py", line 374, in __get_raw_prediction
    chunk_patch_info_list[:, 0, 0], pbar_desc
  File "/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hover_net-master/infer/wsi.py", line 287, in __run_model
    for batch_idx, batch_data in enumerate(dataloader):
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 974, in _next_data
    idx, data = self._get_data()
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 941, in _get_data
    success, data = self._try_get_data()
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 807, in _try_get_data
    "Too many open files. Communication with the"
RuntimeError: Too many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code
Process Chunk 48/99:  61%|#############5        | 35/57 [02:19<01:27,  4.00s/it]
/usr/local/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))

Do you know why this error might occur?

Running on an Ubuntu 20 machine that has a conda env with the requirements.

[vqdang]

@JMBokhorst This is certainly very new to me. Can you detail the running settings ? At the moment, can you also check the number of files opened by the processes, or #hanging processes, as it seems the error propagated from the OS.

OSError: [Errno 24] Too many open files: '/tmp/tmpxrmts9vn'

There is an output log file too, so please also attach for reference.

[simongraham]

@JMBokhorst

Can you confirm that you are running one script or are you processing multiple WSIs in parallel?

[vqdang]

@JMBokhorst could you pull down the PR and check if that fixes the issue ?

[JMBokhorst]

@simongraham and @vqdang,

Thanks for the quick response. I will check out the PR now and see if it fixes the issue.

I'm trying to run the script on a folder containing a single ndpi image. I use this command (based on the run_wsi.sh script):

python3.7 run_infer.py --gpu='0,1' --nr_types=6 --type_info_path=type_info.json --batch_size=64 --model_mode=fast --model_path=/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hovernet_fast_pannuke_type_tf2pytorch.tar --nr_inference_workers=8 --nr_post_proc_workers=16 wsi --input_dir=/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/test_image/ --output_dir=/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/result/ --save_thumb --save_mask

Below is the output of the debug log, is this the log file you are referring to?

|2021-01-06|11:46:06.636| [INFO] ................ Process: TB_S02_P005_C0001_L15_A15
|2021-01-06|11:46:11.858| [INFO] ................ WARNING: No mask found, generating mask via thresholding at 1.25x!
|2021-01-06|11:46:23.762| [INFO] ........ Preparing Input Output Placement: 17.12366568017751
|2021-01-06|13:06:15.182| [ERROR] Crash
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 779, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/local/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/local/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 282, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/local/lib/python3.7/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/local/lib/python3.7/multiprocessing/reduction.py", line 185, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/local/lib/python3.7/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 803, in _try_get_data
    fs = [tempfile.NamedTemporaryFile() for i in range(fds_limit_margin)]
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 803, in <listcomp>
    fs = [tempfile.NamedTemporaryFile() for i in range(fds_limit_margin)]
  File "/usr/local/lib/python3.7/tempfile.py", line 547, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/local/lib/python3.7/tempfile.py", line 258, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
OSError: [Errno 24] Too many open files: '/tmp/tmpxrmts9vn'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hover_net-master/infer/wsi.py", line 746, in process_wsi_list
    self.process_single_file(wsi_path, msk_path, self.output_dir)
  File "/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hover_net-master/infer/wsi.py", line 550, in process_single_file
    self.__get_raw_prediction(chunk_info_list, patch_info_list)
  File "/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hover_net-master/infer/wsi.py", line 374, in __get_raw_prediction
    chunk_patch_info_list[:, 0, 0], pbar_desc
  File "/mnt/netcache/pathology/projects/colon-budding-he/nuclei_detection/hover_pytorch/hover_net-master/infer/wsi.py", line 287, in __run_model
    for batch_idx, batch_data in enumerate(dataloader):
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 974, in _next_data
    idx, data = self._get_data()
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 941, in _get_data
    success, data = self._try_get_data()
  File "/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 807, in _try_get_data
    "Too many open files. Communication with the"
RuntimeError: Too many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code

[JMBokhorst]

Unfortunately, I get the same error with the PR. Since it is happening at the same point, I have put a breakpoint at the point it crashes. I will let you know when I have more information.

[vqdang]

@JMBokhorst
Could you try this ? Also, what is the size of your WSI ?

import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')

[JMBokhorst]

@vqdang, yes going to try it now.

The image has these dimensions (x, y): 86016, 105728

[JMBokhorst]

That also didn't fix the issue, I'm not trying with 0 inference workers to disable multiple processes, see if I get a more clear error

[JMBokhorst]

I have set the inference workers to 0, I don't get an error during processing the slide. However, the script hangs as soon as it is done processing the final chunk; no error, no process, nothing.

The error does seem to be related to the inference workers. I will play a bit with the settings and report back :)

[vqdang]

Are you on the windows or linux ? Could you check also check with --nr_post_proc_workers 0 to turn off multithread for post proc ?

hover_net/infer/wsi.py

Lines 529 to 534 in 4978aa5

	self.wsi_pred_map = np.lib.format.open_memmap(
	"%s/pred_map.npy" % self.cache_path,
	mode="w+",
	shape=tuple(self.wsi_proc_shape) + (out_ch,),
	dtype=np.float32,
	)

You can replace this portion to read the cached memmap and comment this line out to prevent rerunning the prediction
to speed up the process.

hover_net/infer/wsi.py

Line 550 in 4978aa5

self.__get_raw_prediction(chunk_info_list, patch_info_list)

[JMBokhorst]

I'm running on linux. I re-created the conda env. Now with --nr_post_proc_workers set to 0 it seems to be working when I just have a very small mask (only 1 tile big). When I use the entire tissue mask I'm running in to some memory issue's. It is always fine until the final part of phase 1. Could you guys tell me how much memory it should use +/-? I have 32GB of memory but maybe that isn't enough.

[vqdang]

It is always fine until the final part of phase 1.

Rather than memory issue, this may due to some dead thread making the processes hanging. Can you supply the log ?

And for the memory, because you are using linux, you can adding more swap to avoid OOM problem. Our internal testing done upto 100k x 100k on a system with 128GB ram and 128GB swap. The memory usage will ofc scale with the WSI size, but it should not affect much at the post proc phase. The memory at the post proc scale with the #worker mostly because we copy a tile (should be small, by default its 2048)

hover_net/run_infer.py

Line 66 in 0845678

--tile_shape=<n> Shape of tiles for processing. [default: 2048]

from mmap back to ram per worker. (1 worker keep memory for 1 tile, 8 workers keep 8). Could you use a TCGA sample for testing so that our side can replicate any further problems?

The inference phase should use more memory, and also may consume lots of hard drive space depending on WSI sizes.

[JMBokhorst]

I will try with a slide from TCGA, if you want I can try with one of the slide that you used as well?

I have added the debug log below, it is killed during runtime and at that point I saw only 200MB memory left. I see the memory starts with <8GB but it is steadily increasing when the first phase is almost complete. I will try to increase the SWAP area. The image is relative big 100K X 200K, so that might be an issue.

|2021-01-15|15:13:38.395| [INFO] ........ Preparing Input Output Placement: 7.505539770999803
|2021-01-15|15:15:57.582| [ERROR] Crash
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 872, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 107, in get
    if not self._poll(timeout):
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 424, in _poll
    r = wait([self], timeout)
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/local/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 1022) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/maindisk/phd/hover/hover_net/infer/wsi.py", line 746, in process_wsi_list
    self.process_single_file(wsi_path, msk_path, self.output_dir)
  File "/mnt/maindisk/phd/hover/hover_net/infer/wsi.py", line 550, in process_single_file
    self.__get_raw_prediction(chunk_info_list, patch_info_list)
  File "/mnt/maindisk/phd/hover/hover_net/infer/wsi.py", line 373, in __get_raw_prediction
    patch_output_list = self.__run_model(
  File "/mnt/maindisk/phd/hover/hover_net/infer/wsi.py", line 287, in __run_model
    for batch_idx, batch_data in enumerate(dataloader):
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1068, in _next_data
    idx, data = self._get_data()
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1034, in _get_data
    success, data = self._try_get_data()
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 885, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 1022) exited unexpectedly

[simongraham]

Just to confirm, can you also say whether you are using the exact environment instructions that we specify on the README?

We will send a couple of links to TCGA slides to test. One big and one small.

Thanks for hanging in there - I’m sure we will sort it ASAP 😊