Error running simulated data

[mxw010]

Hi, I run into the following error while trying to run some simulated dataset. We have several settings, and two settings gave us the same error regardless of parameters settings.

100%|██████████| 166314/166314 [00:00<00:00, 309719.43it/s]
generating start/end dict for chromosome
extracting from data.txt
generating contact maps for baseline
data loaded
166314 False
creating matrices tasks: 100%|██████████| 1/1 [00:00<00:00,  2.69it/s]
100%|██████████| 1/1 [00:00<00:00, 20.34it/s]
0.608: 100%|██████████| 300/300 [00:00<00:00, 300.39it/s]
100%|██████████| 1/1 [00:00<00:00, 46091.25it/s]
100%|██████████| 1/1 [00:00<00:00, 54471.48it/s]
total_feats_size 80
cpu_num 1
setting to gpu:0
training on data from: ['chr1']
total_sparsity_cell 0.8946236559139785
contractive loss
batch_size 256
Node type num [100  61] [100 161]
start making attribute
loss 0.6080093383789062 loss best 0.6089616417884827 epochs 299

initializing data generator
initializing data generator
First stage training
[ Epoch 0 of 60 ]
 - (Training) :   0%|          | 0/1000 [00:00<?, ?it/s]concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/.conda/envs/higashi/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/.conda/envs/higashi/lib/python3.9/site-packages/higashi-0.1.0a0-py3.9.egg/higashi/Higashi_wrapper.py", line 248, in one_thread_generate_neg
    raise EOFError
EOFError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Data/run_higashi.py", line 10, in <module>
    higashi_model.train_for_embeddings()
  File "/.conda/envs/higashi/lib/python3.9/site-packages/higashi-0.1.0a0-py3.9.egg/higashi/Higashi_wrapper.py", line 1336, in train_for_embeddings
    self.train(
  File "/.conda/envs/higashi/lib/python3.9/site-packages/higashi-0.1.0a0-py3.9.egg/higashi/Higashi_wrapper.py", line 1113, in train
    bce_loss, mse_loss, train_accu, auc1, auc2, str1, str2, train_pool, train_p_list = self.train_epoch(
  File "/.conda/envs/higashi/lib/python3.9/site-packages/higashi-0.1.0a0-py3.9.egg/higashi/Higashi_wrapper.py", line 914, in train_epoch
    batch_edge_big, batch_y_big, batch_edge_weight_big, batch_chrom_big, batch_to_neighs_big, chroms_in_batch = p.result()
  File "/.conda/envs/higashi/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/.conda/envs/higashi/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
EOFError

The JSON input:

{
    "config_name": "test run",
    "input_format": "higashi_v1",
    "data_dir": "/Datasimba1_7k",
    "temp_dir": "/Data/simba1_7k/temp",
    "genome_reference_path": "/Data/test_run/simulation/chrom.size",
    "cytoband_path": "/Data/test_run/simulation/cytoband.txt",
    "chrom_list": [
        "chr1"
    ],
    "resolution": 1000000,
    "resolution_cell": 1000000,
    "minimum_distance": 2000000,
    "maximum_distance": -1,
    "local_transfer_range": 1,
    "dimensions": 30,
    "impute_list": [
        "chr1"
    ],
    "minimum_impute_distance": 0,
    "maximum_impute_distance": -1,
    "neighbor_num": 5,
    "call_tads": false,
    "embedding_name": "exp_zinb3",
    "cpu_num": 1,
    "gpu_num": 1,
    "optional_smooth": false,
    "optional_quantile": false,
    "rank_thres": 1,
    "loss_mode": "zinb",
    "random_walk": false,
    "UMAP_params": {
        "n_neighbors": 20
    }
}

I did a little debugging myself, and for the runs that failed (very early into training), len(neg_list) (line 248 ish in Higashi_wrapper.py) usually is <5. In runs that didn't fail, it's usually around 100.