[BUG] difficulty with training custom model on GUI #1377
Description
Before you fill out this form:
Did you review the FAQ?
- [ X ] Yes, I reviewed the FAQ on the cellpose ReadTheDocs
Did you look through previous (open AND closed) issues posted on GH?
- [ X ] Yes, I searched for related previous GH issues.
Now fill this form out completely:
Describe the bug
I have been trying to train a custom model for cell segmentation of RNAscope images using the Cellpose GUI on MacBook, but it seems to continually over-segment the images after training, thus massively underperforming even the CP-SAM model. All images for training had been converted into RGB image formatted TIFF files (which we and our colleagues found seemed to work better.) I’ve tried many different things at this point and would appreciate any guidance you can offer! I’m also happy to include any other images/files/screenshots that may be helpful!
To Reproduce
Steps to reproduce the behavior:
- Load 6 images as seg.npy files (see screenshots attached below) - these have already been manually corrected after running CP-SAM with the same settings (diameter = 0, flow threshold = 0.4, cell prob threshold = 0.40, norm percentiles: 1.0-99.0, niter dynamics = 200)
- Then click on Models -> Train new model with image+masks in folder
- The settings were used as follows: learning_rate: 1e-05, weight_decay: 0.1, n_epochs: 1000
- The images were confirmed to be listed in the filenames section of the train settings window. With all 6 images, we would be training with a total of 1613 masks with about 200-400 masks per image.
- Then click OK and let the model training run.
- Then I see the final result of the model running on a training image and this result shows the over-segmentation I mentioned in the summary of the bug.
Run log
GUI_INFO: name of new model: RNascope_cells_12–01-25
2025-12-01 11:41:12,050 [INFO] training with ['MAX_CAG-Rep1-Calca,Sstr2,EGFP,Tubb3_2025-10-10_12.09.53-0005.tif', 'MAX_CAG-Rep2-Calca,Sstr2,EGFP,Tubb3_2025-10-10_10.46.16-0005.tif', 'NP9-3_Rep_1_egfpMrgprd568Tubb3dapi-0004.tif', 'NP9-3_Rep_1_egfpMrgprd568Tubb3dapi-0004_02.tif', 'NP9-3_Rep_2_egfpMrgprd568Tubb3dapi-0004.tif', 'Opal780-MAX_PEP1-17-Rep3-Calca,Sstr2,EGFP,Tubb3_2025-10-17_10.53.15-0005.tif']
2025-12-01 11:41:12,051 [INFO] training new model starting at model cpsam
2025-12-01 11:41:12,051 [WARNING] model_type argument is not used in v4.0.1+. Ignoring this argument...
2025-12-01 11:41:12,085 [INFO] ** TORCH MPS version installed and working. **
2025-12-01 11:41:12,085 [INFO] >>>> using GPU (MPS)
2025-12-01 11:41:39,603 [INFO] >>>> loading model /Users/lsh27/.cellpose/models/cpsam
GUI_INFO: name of new model: RNascope_cells_12-01-25
2025-12-01 11:41:41,059 [WARNING] Training with bfloat16 on MPS is not supported, using float32 network instead
2025-12-01 11:41:41,155 [INFO] computing flows for labels
100%|█████████████████████████████████████████████████| 6/6 [00:20<00:00, 3.39s/it]
2025-12-01 11:42:01,533 [INFO] >>> computing diameters
100%|████████████████████████████████████████████████| 6/6 [00:00<00:00, 123.29it/s]
2025-12-01 11:42:01,585 [INFO] >>> normalizing {'lowhigh': None, 'percentile': [1.0, 99.0], 'normalize': True, 'norm3D': True, 'sharpen_radius': 0.0, 'smooth_radius': 0.0, 'tile_norm_blocksize': 0.0, 'tile_norm_smooth3D': 1.0, 'invert': False}
2025-12-01 11:42:01,806 [INFO] >>> n_epochs=1000, n_train=6, n_test=None
2025-12-01 11:42:01,806 [INFO] >>> AdamW, learning_rate=0.00001, weight_decay=0.10000
2025-12-01 11:42:01,808 [INFO] >>> saving model to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 11:42:07,845 [INFO] 0, train_loss=1.7423, test_loss=0.0000, LR=0.000000, time 6.04s
2025-12-01 11:42:30,111 [INFO] 5, train_loss=1.3042, test_loss=0.0000, LR=0.000006, time 28.30s
2025-12-01 11:42:52,674 [INFO] 10, train_loss=0.5546, test_loss=0.0000, LR=0.000010, time 50.87s
2025-12-01 11:43:37,944 [INFO] 20, train_loss=0.3645, test_loss=0.0000, LR=0.000010, time 96.14s
2025-12-01 11:44:24,636 [INFO] 30, train_loss=0.3213, test_loss=0.0000, LR=0.000010, time 142.83s
2025-12-01 11:45:11,788 [INFO] 40, train_loss=0.3172, test_loss=0.0000, LR=0.000010, time 189.98s
2025-12-01 11:45:58,490 [INFO] 50, train_loss=0.2907, test_loss=0.0000, LR=0.000010, time 236.68s
2025-12-01 11:46:46,914 [INFO] 60, train_loss=0.2821, test_loss=0.0000, LR=0.000010, time 285.11s
2025-12-01 11:47:36,722 [INFO] 70, train_loss=0.2685, test_loss=0.0000, LR=0.000010, time 334.91s
2025-12-01 11:48:29,695 [INFO] 80, train_loss=0.2721, test_loss=0.0000, LR=0.000010, time 387.89s
2025-12-01 11:49:23,239 [INFO] 90, train_loss=0.2574, test_loss=0.0000, LR=0.000010, time 441.43s
2025-12-01 11:50:16,204 [INFO] 100, train_loss=0.2811, test_loss=0.0000, LR=0.000010, time 494.40s
2025-12-01 11:50:16,205 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 11:51:08,833 [INFO] 110, train_loss=0.2514, test_loss=0.0000, LR=0.000010, time 547.03s
2025-12-01 11:51:59,254 [INFO] 120, train_loss=0.3011, test_loss=0.0000, LR=0.000010, time 597.45s
2025-12-01 11:52:49,296 [INFO] 130, train_loss=0.2353, test_loss=0.0000, LR=0.000010, time 647.49s
2025-12-01 11:53:40,132 [INFO] 140, train_loss=0.2583, test_loss=0.0000, LR=0.000010, time 698.32s
2025-12-01 11:54:30,275 [INFO] 150, train_loss=0.2337, test_loss=0.0000, LR=0.000010, time 748.47s
2025-12-01 11:55:20,892 [INFO] 160, train_loss=0.2737, test_loss=0.0000, LR=0.000010, time 799.08s
2025-12-01 11:56:11,295 [INFO] 170, train_loss=0.2548, test_loss=0.0000, LR=0.000010, time 849.49s
2025-12-01 11:57:00,426 [INFO] 180, train_loss=0.2520, test_loss=0.0000, LR=0.000010, time 898.62s
2025-12-01 11:57:49,868 [INFO] 190, train_loss=0.2539, test_loss=0.0000, LR=0.000010, time 948.06s
2025-12-01 11:58:38,717 [INFO] 200, train_loss=0.2394, test_loss=0.0000, LR=0.000010, time 996.91s
2025-12-01 11:58:38,718 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 11:59:28,521 [INFO] 210, train_loss=0.2559, test_loss=0.0000, LR=0.000010, time 1046.71s
2025-12-01 12:00:17,038 [INFO] 220, train_loss=0.2499, test_loss=0.0000, LR=0.000010, time 1095.23s
2025-12-01 12:01:05,494 [INFO] 230, train_loss=0.2194, test_loss=0.0000, LR=0.000010, time 1143.69s
2025-12-01 12:01:53,758 [INFO] 240, train_loss=0.2517, test_loss=0.0000, LR=0.000010, time 1191.95s
2025-12-01 12:02:41,983 [INFO] 250, train_loss=0.2424, test_loss=0.0000, LR=0.000010, time 1240.18s
2025-12-01 12:03:30,477 [INFO] 260, train_loss=0.2195, test_loss=0.0000, LR=0.000010, time 1288.67s
2025-12-01 12:04:18,896 [INFO] 270, train_loss=0.2339, test_loss=0.0000, LR=0.000010, time 1337.09s
2025-12-01 12:05:11,443 [INFO] 280, train_loss=0.2384, test_loss=0.0000, LR=0.000010, time 1389.63s
2025-12-01 12:06:00,136 [INFO] 290, train_loss=0.2397, test_loss=0.0000, LR=0.000010, time 1438.33s
2025-12-01 12:06:49,579 [INFO] 300, train_loss=0.2334, test_loss=0.0000, LR=0.000010, time 1487.77s
2025-12-01 12:06:49,581 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 12:07:39,969 [INFO] 310, train_loss=0.2578, test_loss=0.0000, LR=0.000010, time 1538.16s
2025-12-01 12:08:29,601 [INFO] 320, train_loss=0.2342, test_loss=0.0000, LR=0.000010, time 1587.79s
2025-12-01 12:09:19,214 [INFO] 330, train_loss=0.2562, test_loss=0.0000, LR=0.000010, time 1637.41s
2025-12-01 12:10:08,593 [INFO] 340, train_loss=0.2555, test_loss=0.0000, LR=0.000010, time 1686.79s
2025-12-01 12:10:57,884 [INFO] 350, train_loss=0.2587, test_loss=0.0000, LR=0.000010, time 1736.08s
2025-12-01 12:11:48,260 [INFO] 360, train_loss=0.2193, test_loss=0.0000, LR=0.000010, time 1786.45s
2025-12-01 12:12:39,913 [INFO] 370, train_loss=0.2278, test_loss=0.0000, LR=0.000010, time 1838.11s
2025-12-01 12:13:31,828 [INFO] 380, train_loss=0.2541, test_loss=0.0000, LR=0.000010, time 1890.02s
2025-12-01 12:14:22,588 [INFO] 390, train_loss=0.2363, test_loss=0.0000, LR=0.000010, time 1940.78s
2025-12-01 12:15:12,484 [INFO] 400, train_loss=0.2465, test_loss=0.0000, LR=0.000010, time 1990.68s
2025-12-01 12:15:12,486 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 12:16:03,036 [INFO] 410, train_loss=0.2427, test_loss=0.0000, LR=0.000010, time 2041.23s
2025-12-01 12:16:52,860 [INFO] 420, train_loss=0.2323, test_loss=0.0000, LR=0.000010, time 2091.05s
2025-12-01 12:17:42,961 [INFO] 430, train_loss=0.2169, test_loss=0.0000, LR=0.000010, time 2141.15s
2025-12-01 12:18:33,350 [INFO] 440, train_loss=0.2465, test_loss=0.0000, LR=0.000010, time 2191.54s
2025-12-01 12:19:24,287 [INFO] 450, train_loss=0.2342, test_loss=0.0000, LR=0.000010, time 2242.48s
2025-12-01 12:20:15,105 [INFO] 460, train_loss=0.2539, test_loss=0.0000, LR=0.000010, time 2293.30s
2025-12-01 12:21:05,693 [INFO] 470, train_loss=0.2340, test_loss=0.0000, LR=0.000010, time 2343.89s
2025-12-01 12:21:55,540 [INFO] 480, train_loss=0.2473, test_loss=0.0000, LR=0.000010, time 2393.73s
2025-12-01 12:22:44,943 [INFO] 490, train_loss=0.2342, test_loss=0.0000, LR=0.000010, time 2443.14s
2025-12-01 12:23:34,068 [INFO] 500, train_loss=0.2125, test_loss=0.0000, LR=0.000010, time 2492.26s
2025-12-01 12:23:34,070 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 12:24:24,758 [INFO] 510, train_loss=0.2262, test_loss=0.0000, LR=0.000010, time 2542.95s
2025-12-01 12:25:15,733 [INFO] 520, train_loss=0.2407, test_loss=0.0000, LR=0.000010, time 2593.93s
2025-12-01 12:26:05,969 [INFO] 530, train_loss=0.2055, test_loss=0.0000, LR=0.000010, time 2644.16s
2025-12-01 12:26:56,923 [INFO] 540, train_loss=0.2314, test_loss=0.0000, LR=0.000010, time 2695.12s
2025-12-01 12:27:47,424 [INFO] 550, train_loss=0.2190, test_loss=0.0000, LR=0.000010, time 2745.62s
2025-12-01 12:28:38,461 [INFO] 560, train_loss=0.2107, test_loss=0.0000, LR=0.000010, time 2796.65s
2025-12-01 12:29:29,493 [INFO] 570, train_loss=0.2530, test_loss=0.0000, LR=0.000010, time 2847.69s
2025-12-01 12:30:21,685 [INFO] 580, train_loss=0.2226, test_loss=0.0000, LR=0.000010, time 2899.88s
2025-12-01 12:31:13,442 [INFO] 590, train_loss=0.2181, test_loss=0.0000, LR=0.000010, time 2951.63s
2025-12-01 12:32:05,568 [INFO] 600, train_loss=0.2357, test_loss=0.0000, LR=0.000010, time 3003.76s
2025-12-01 12:32:05,570 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 12:32:58,280 [INFO] 610, train_loss=0.2077, test_loss=0.0000, LR=0.000010, time 3056.47s
2025-12-01 12:33:49,581 [INFO] 620, train_loss=0.2021, test_loss=0.0000, LR=0.000010, time 3107.77s
2025-12-01 12:34:42,140 [INFO] 630, train_loss=0.2367, test_loss=0.0000, LR=0.000010, time 3160.33s
2025-12-01 12:35:34,505 [INFO] 640, train_loss=0.2100, test_loss=0.0000, LR=0.000010, time 3212.70s
2025-12-01 12:36:25,307 [INFO] 650, train_loss=0.2265, test_loss=0.0000, LR=0.000010, time 3263.50s
2025-12-01 12:37:16,416 [INFO] 660, train_loss=0.2035, test_loss=0.0000, LR=0.000010, time 3314.61s
2025-12-01 12:38:07,395 [INFO] 670, train_loss=0.2280, test_loss=0.0000, LR=0.000010, time 3365.59s
2025-12-01 12:38:58,755 [INFO] 680, train_loss=0.2316, test_loss=0.0000, LR=0.000010, time 3416.95s
2025-12-01 12:39:49,850 [INFO] 690, train_loss=0.2247, test_loss=0.0000, LR=0.000010, time 3468.04s
2025-12-01 12:40:43,829 [INFO] 700, train_loss=0.2219, test_loss=0.0000, LR=0.000010, time 3522.02s
2025-12-01 12:40:43,830 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 12:41:37,534 [INFO] 710, train_loss=0.2274, test_loss=0.0000, LR=0.000010, time 3575.73s
2025-12-01 12:42:29,975 [INFO] 720, train_loss=0.2210, test_loss=0.0000, LR=0.000010, time 3628.17s
2025-12-01 12:43:22,821 [INFO] 730, train_loss=0.2237, test_loss=0.0000, LR=0.000010, time 3681.01s
2025-12-01 12:44:14,949 [INFO] 740, train_loss=0.2129, test_loss=0.0000, LR=0.000010, time 3733.14s
2025-12-01 12:45:08,831 [INFO] 750, train_loss=0.2187, test_loss=0.0000, LR=0.000010, time 3787.02s
2025-12-01 12:46:00,354 [INFO] 760, train_loss=0.2042, test_loss=0.0000, LR=0.000010, time 3838.55s
2025-12-01 12:46:52,499 [INFO] 770, train_loss=0.1960, test_loss=0.0000, LR=0.000010, time 3890.69s
2025-12-01 12:47:44,545 [INFO] 780, train_loss=0.2088, test_loss=0.0000, LR=0.000010, time 3942.74s
2025-12-01 12:48:35,337 [INFO] 790, train_loss=0.2095, test_loss=0.0000, LR=0.000010, time 3993.53s
2025-12-01 12:49:25,206 [INFO] 800, train_loss=0.2082, test_loss=0.0000, LR=0.000010, time 4043.40s
2025-12-01 12:49:25,208 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 12:50:15,871 [INFO] 810, train_loss=0.2133, test_loss=0.0000, LR=0.000010, time 4094.06s
2025-12-01 12:51:07,354 [INFO] 820, train_loss=0.2113, test_loss=0.0000, LR=0.000010, time 4145.55s
2025-12-01 12:51:58,340 [INFO] 830, train_loss=0.2310, test_loss=0.0000, LR=0.000010, time 4196.53s
2025-12-01 12:52:49,628 [INFO] 840, train_loss=0.2428, test_loss=0.0000, LR=0.000010, time 4247.82s
2025-12-01 12:53:41,057 [INFO] 850, train_loss=0.2140, test_loss=0.0000, LR=0.000010, time 4299.25s
2025-12-01 12:54:32,645 [INFO] 860, train_loss=0.1994, test_loss=0.0000, LR=0.000010, time 4350.84s
2025-12-01 12:55:24,162 [INFO] 870, train_loss=0.2105, test_loss=0.0000, LR=0.000010, time 4402.35s
2025-12-01 12:56:15,499 [INFO] 880, train_loss=0.2353, test_loss=0.0000, LR=0.000010, time 4453.69s
2025-12-01 12:57:06,840 [INFO] 890, train_loss=0.1934, test_loss=0.0000, LR=0.000010, time 4505.03s
2025-12-01 12:57:56,507 [INFO] 900, train_loss=0.2192, test_loss=0.0000, LR=0.000005, time 4554.70s
2025-12-01 12:57:56,508 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
2025-12-01 12:58:47,800 [INFO] 910, train_loss=0.1955, test_loss=0.0000, LR=0.000003, time 4605.99s
2025-12-01 12:59:38,184 [INFO] 920, train_loss=0.2169, test_loss=0.0000, LR=0.000001, time 4656.38s
2025-12-01 13:00:28,229 [INFO] 930, train_loss=0.2255, test_loss=0.0000, LR=0.000001, time 4706.42s
2025-12-01 13:01:18,436 [INFO] 940, train_loss=0.2184, test_loss=0.0000, LR=0.000000, time 4756.63s
2025-12-01 13:02:08,132 [INFO] 950, train_loss=0.1892, test_loss=0.0000, LR=0.000000, time 4806.32s
2025-12-01 13:02:58,075 [INFO] 960, train_loss=0.2124, test_loss=0.0000, LR=0.000000, time 4856.27s
2025-12-01 13:03:57,572 [INFO] 970, train_loss=0.2344, test_loss=0.0000, LR=0.000000, time 4915.76s
2025-12-01 13:04:53,137 [INFO] 980, train_loss=0.2067, test_loss=0.0000, LR=0.000000, time 4971.33s
2025-12-01 13:05:48,357 [INFO] 990, train_loss=0.2098, test_loss=0.0000, LR=0.000000, time 5026.55s
2025-12-01 13:06:38,255 [INFO] saving network parameters to /Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25
/Users/lsh27/Downloads/Cellpose RGB Training Images/models/RNascope_cells_12-01-25 copied to models folder /Users/lsh27/.cellpose/models
GUI_INFO: selected model RNascope_cells_12-01-25, loading now
2025-12-01 13:06:40,995 [INFO] ** TORCH MPS version installed and working. **
2025-12-01 13:06:40,998 [INFO] >>>> using GPU (MPS)
2025-12-01 13:06:42,148 [INFO] >>>> loading model /Users/lsh27/.cellpose/models/RNascope_cells_12-01-25
GUI_INFO: loading image: /Users/lsh27/Downloads/Cellpose RGB Training Images/NP9-3_Rep_1_egfpMrgprd568Tubb3dapi-0004_02.tif
GUI_INFO: image shape: (1024, 1024, 3)
GUI_INFO: normalization checked: computing saturation levels (and optionally filtered image)
{'lowhigh': None, 'percentile': [1.0, 99.0], 'normalize': True, 'norm3D': True, 'sharpen_radius': 0.0, 'smooth_radius': 0.0, 'tile_norm_blocksize': 0.0, 'tile_norm_smooth3D': 1.0, 'invert': False}
[np.float32(0.0), np.float32(106.0)]
2025-12-01 13:06:43,241 [INFO] ** TORCH MPS version installed and working. **
2025-12-01 13:06:43,241 [INFO] >>>> using GPU (MPS)
2025-12-01 13:06:44,341 [INFO] >>>> loading model /Users/lsh27/.cellpose/models/RNascope_cells_12-01-25
{'lowhigh': None, 'percentile': [1.0, 99.0], 'normalize': True, 'norm3D': True, 'sharpen_radius': 0.0, 'smooth_radius': 0.0, 'tile_norm_blocksize': 0.0, 'tile_norm_smooth3D': 1.0, 'invert': False}
2025-12-01 13:06:54,238 [INFO] 2462 cells found with model in 11.004 sec
GUI_INFO: 2462 masks found
GUI_INFO: creating cellcolors and drawing masks
2025-12-01 13:06:54,282 [INFO] !!! computed masks for NP9-3_Rep_1_egfpMrgprd568Tubb3dapi-0004_02.tif from new model !!!
Screenshots
If applicable, add screenshots to help explain your problem.
