RuntimeError: CUDA driver error: invalid resource handle #100
Description
When I am trying to run bash dev_scripts/dtu_test_inf/inftest_scan1.sh , i got this
training images = 16
dtu_ft train id [25, 21, 33, 22, 14, 15, 26, 30, 31, 35, 34, 43, 46, 29, 16, 36]
dtu_ft test id [32, 24, 23, 44]
self.plane_ind 0
/home/won147/miniconda3/envs/pointnerf/lib/python3.8/site-packages/numpy/core/shape_base.py:420: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
arrays = [asanyarray(arr) for arr in arrays]
dataset total: test 4
dataset [DtuDataset] was created
Traceback (most recent call last):
File "train_ft.py", line 1081, in
main()
File "train_ft.py", line 791, in main
bg_ray_train_lst = create_all_bg(train_dataset, model, img_lst, c2ws_lst, w2cs_lst, intrinsics_all, HDWD_lst)
File "train_ft.py", line 572, in create_all_bg
bg_ray, _ = model.set_bg(xyz_world_sect_plane, img_lst, c2ws_lst, w2cs_lst, intrinsics_all, HDWD_lst, data["plane_color"])
File "/home/won147/Desktop/pointnerf/run/../models/mvs_points_volumetric_model.py", line 284, in set_bg
src_grid, mask, hard_id_xy = warp(c2w, w2c, intrinsics, xyz_world_sect_plane, HD, WD, filter=False, tolerate=0.1)
File "/home/won147/Desktop/pointnerf/run/../models/mvs/mvs_utils.py", line 308, in homo_warp_nongrid
mask = torch.prod(torch.cat([torch.ge(src_grid, torch.zeros([1,1,2], device=src_grid.device)), torch.le(src_grid, torch.tensor([[[WD-1,HD-1]]], device=src_grid.device))],dim=-1), dim=-1, keepdim=True, dtype=torch.int8) > 0
RuntimeError: CUDA driver error: invalid resource handle
any suggestion is appreciated!