Nan to zero frames

[tayllatheodoro]

convert nan frames to zero, as nan frames causes errors on virtual stain step

[talonchandler]

Hi @tayllatheodoro, can you provide more context here. Are these nan frames appearing in the raw data?

I quickly profiled np.nan_to_num, and I'm getting ~0.5 seconds for 50 x 2048 x 2048 arrays. Not a blocker, but worth considering if we can handle this elsewhere or in a faster way.

[tayllatheodoro]

Hi @talonchandler, so the empty frames of the raw data cames as nan frames instead of zero frames. When we run VS on a FOV with nan frames it aftes the model inference (because of the statistics mesurements) and all frames are not predicted. The nan to zero check was done previously on stabilization step, then VS used to be run after stabilization, but it introduces artifacts to VS. Now we are runing VS on the recontructed phase. And Ivan attempted to correct this here https://github.com/mehta-lab/waveorder/blob/11f977ae2245fb5041d2f03369b4dc9e4ff2357c/waveorder/cli/utils.py but the nan frames are still there. Maybe a solution would be to create the output zarr with zeros?

[srivarra]

@tayllatheodoro @talonchandler

I just benchmarked NumPy's nan_to_num and Dasks'. Dask's nan_to_num is about 2x faster on a 50 x 2048 x 2048 Array. I suggest we use Dask's implementation.

NaN %	NumPy Time	Dask Time	Speedup
20%	0.621 s	0.271 s	2.29x
40%	0.841 s	0.292 s	2.88x
60%	0.856 s	0.298 s	2.87x

[talonchandler]

@srivarra let's hold off on this. @tayllatheodoro is going to look closer into the root cause of this issue, and we'll decide what's next.

[srivarra]

@srivarra let's hold off on this.

🫡