test_from_cache_or_calc failing seemingly randomly

[kerberizer]

One of the tests seems to fail in what appears to be a random manner—or at least I haven't found a definitive trigger. Since it would pass on a subsequent run—and there's no caching between runs—I wonder if it isn't some not completely deterministic results changing because of, e.g., rounding, or perhaps even something related to clock time (though that would be, admittedly, unusual).

To be clear, it fails on different versions of Python, too. In one run, it may fail, e.g., on 3.13, but pass on 3.10. In another run it would fail on 3.10, yet pass on 3.13. This adds to the impression of the problem being related to non-deterministic results and rounding issues or perhaps clock/racing conditions.

Here's an excerpt from a failed run:

  tests/spectrum/test_random_generator_seeds.py ....                       [100%]
  
  =================================== FAILURES ===================================
  ___________________________ test_from_cache_or_calc ____________________________
  
      def test_from_cache_or_calc():
          with tempfile.TemporaryDirectory() as tmpdir:
              steps = [{'proc': 'from_delta_lines',
                        'args': [],
                        'kwargs': {'nbins': 3000, 'deltas': {200: 100, 600: 50, 1000: 150, 1500: 70}}},
                       {'proc': 'normalize', 'args': [], 'kwargs': {}},
                       {'proc': 'add_gaussian_noise_drift', 'args': [], 'kwargs': {'sigma': 1, 'coef': .1}}]
      
              cachefile = None
              spe1 = rc2.spectrum.from_cache_or_calc(cachefile=cachefile, required_steps=steps)
              spe2 = rc2.spectrum.from_cache_or_calc(cachefile=cachefile, required_steps=steps)
              assert not np.allclose(spe1.y, spe2.y)  # spe2 is distict from spe1. spe2 not coming from cache
      
              cachefile = ''
              spe1 = rc2.spectrum.from_cache_or_calc(cachefile=cachefile, required_steps=steps)
              spe2 = rc2.spectrum.from_cache_or_calc(cachefile=cachefile, required_steps=steps)
              assert not np.allclose(spe1.y, spe2.y)  # spe2 is distict from spe1. spe2 not coming from cache
      
              cachefile = os.path.join(tmpdir, 'test.cha')
              spe1 = rc2.spectrum.from_cache_or_calc(cachefile=cachefile, required_steps=steps)
              spe2 = rc2.spectrum.from_cache_or_calc(cachefile=cachefile, required_steps=steps)
              assert np.allclose(spe1.y, spe2.y)  # spe2 is equal to spe1. spe2 is coming from cache
      
              spe_deltas = rc2.spectrum.from_delta_lines(deltas={200: 100, 600: 50, 1000: 150, 1500: 70},
                                                         nbins=3000,
                                                         cachefile=cachefile)
              spe_gaus = spe_deltas.convolve('gaussian', sigma=20).normalize()
              spe_baseline = spe_gaus.add_baseline(n_freq=20, pedestal=.02, amplitude=.15)
              spe_noise = spe_baseline.add_gaussian_noise(sigma=.05).normalize()
              spe_cand = spe_noise.find_peak_multipeak_filter(prominence=.3, width=25)
              spe_fit = spe_cand.fit_peaks_filter(profile='Gaussian')
      
  >           assert len(spe_cand.result) == 4
  E           AssertionError: assert 3 == 4
  E            +  where 3 = len([{'base_intercept': 0.004057548812964315, 'base_slope': 0.0003405586148865484, 'boundaries': (150.74733333333333, 235....itude': 0.4844449331009718, 'position': 1498.8083333333334, 'sigma': 8.501441397520601, 'skew': 0.13893415021937666}]}])
  E            +    where [{'base_intercept': 0.004057548812964315, 'base_slope': 0.0003405586148865484, 'boundaries': (150.74733333333333, 235....itude': 0.4844449331009718, 'position': 1498.8083333333334, 'sigma': 8.501441397520601, 'skew': 0.13893415021937666}]}] = from_delta_lines(deltas={200: 100, 600: 50, 1000: 150, 1500: 70}, nbins=3000, cachefile='/tmp/tmpp9s7s1b1/test.cha').c...l=0.02, amplitude=0.15).add_gaussian_noise(sigma=0.05).normalize().find_peak_multipeak_filter(prominence=0.3, width=25).result
  
  tests/end_to_end/test_from_cache_or_calc.py:42: AssertionError

[georgievgeorgi]

It looks like the find_peak algorithm is not able to find all 4 peaks in the synthetic spectrum. The non deterministic behavior is most probably related to the fact that the seed for the random noise is not fixed. Are you aware of any recent changes to the algorithms related to peak finding?

[kerberizer]

It looks like the find_peak algorithm is not able to find all 4 peaks in the synthetic spectrum. The non deterministic behavior is most probably related to the fact that the seed for the random noise is not fixed.

Thanks! That makes sense indeed.

Are you aware of any recent changes to the algorithms related to peak finding?

Not that I'm aware of, indeed. But with setuptools we don't have any kind of dependency pinning, like with poetry or uv (and we dropped the requirements.txt files, perhaps for good reasons). So the actual dependency versions might have changed, indeed, at least within the requirements in setup.py.

On a side note, we've discussed with Nina transitioning everything Python that we have to uv for ease of use and maintenance, but that's not necessarily going to alleviate such problems. Unlike applications, libraries like rc2 can be pulled in vastly different environments, where our lockfile wouldn't matter anyway, and only the setup.py/pyproject.toml restrictions would be honored.

[georgievgeorgi]

As a quick fix i would suggest decreasing by a little bit the amplitude of the noise. In both add_gaussian_noise_drift and add_gaussian_noise the parameter name is sigma.