Pipeline issue

[arnodelorme]

I wrote this script to test the pipelines side by side.
It seems that it does not meet pass the tolerance. The difference is quite large.
Please review my last 2 commits as well. I added some code to debug.

ee493e5
ee493e5

Thanks,

Arno

INFO (numexpr.utils) NumExpr defaulting to 10 threads.
/Users/arno/Python/eegprep/tests/../data/eeglab_data_with_ica_tmp.set
INFO (eegprep.clean_artifacts) Detecting flat line channels...
INFO (eegprep.clean_artifacts) Applying high‑pass filter...
INFO (eegprep.clean_channels) Scanning for bad channels...
INFO (eegprep.clean_channels) 1/47 blocks, 0.0 minutes remaining.
ERROR (eegprep.clean_channels) Apparently you do not have EEGLAB's pop_select() on the path.
INFO (eegprep.clean_channels) Removing 2 channels and dropping signal meta-data.
INFO (eegprep.clean_artifacts) Final post‑processing – removing irrecoverable windows...
INFO (eegprep.clean_windows) Determining time window rejection thresholds...
INFO (eegprep.clean_windows) done.
INFO (eegprep.clean_windows) Keeping 100.0% (240 seconds) of the data.
ERROR (eegprep.clean_windows) Apparently you do not have EEGLAB's pop_select() on the path.
INFO (eegprep.clean_windows) Falling back to a basic substitute and dropping signal meta-data.
INFO (eegprep.clean_artifacts) Use vis_artifacts to compare the cleaned data to the original.
Loading MAT runtime...This is the path2eeglab: /Users/arno/Python/eegprep/src/eegprep/eeglab
eeglab: options file is ~/eeg_options.m
Retrieving plugin versions from server...
Retrieving download statistics...
EEGLAB: adding "EEG-BIDS" v? (see >> help eegplugin_eegbids) - new version 10.2 available
EEGLAB: adding "ICLabel" v? (see >> help eegplugin_iclabel) - new version 1.6 available (critical bug fix)
EEGLAB: adding "clean_rawdata" v? (see >> help eegplugin_clean_rawdata) - new version 2.10 available
EEGLAB: adding "dipfit" v? (see >> help eegplugin_dipfit) - new version 5.5 available
EEGLAB: adding "firfilt" v? (see >> help eegplugin_firfilt) - new version 2.8 available
done.
pop_loadset(): loading file /Users/arno/Python/eegprep/temp/tmpy3q40c74.set ...
Running in MATLAB: EEG = clean_artifacts(EEG,'BurstCriterion','off','ChannelCriterion',0.8);
Detecting flat line...
Applying highpass filter...
Scanning for bad channels...
clean_channel: 1/47 blocks, 0.0 minutes remaining.
clean_channel: 2/47 blocks, 0.0 minutes remaining.
clean_channel: 3/47 blocks, 0.0 minutes remaining.
clean_channel: 4/47 blocks, 0.0 minutes remaining.
clean_channel: 5/47 blocks, 0.0 minutes remaining.
clean_channel: 6/47 blocks, 0.0 minutes remaining.
clean_channel: 7/47 blocks, 0.0 minutes remaining.
clean_channel: 8/47 blocks, 0.0 minutes remaining.
clean_channel: 9/47 blocks, 0.0 minutes remaining.
clean_channel: 10/47 blocks, 0.0 minutes remaining.
clean_channel: 11/47 blocks, 0.0 minutes remaining.
clean_channel: 12/47 blocks, 0.0 minutes remaining.
clean_channel: 13/47 blocks, 0.0 minutes remaining.
clean_channel: 14/47 blocks, 0.0 minutes remaining.
clean_channel: 15/47 blocks, 0.0 minutes remaining.
clean_channel: 16/47 blocks, 0.0 minutes remaining.
clean_channel: 17/47 blocks, 0.0 minutes remaining.
clean_channel: 18/47 blocks, 0.0 minutes remaining.
clean_channel: 19/47 blocks, 0.0 minutes remaining.
clean_channel: 20/47 blocks, 0.0 minutes remaining.
clean_channel: 21/47 blocks, 0.0 minutes remaining.
clean_channel: 22/47 blocks, 0.0 minutes remaining.
clean_channel: 23/47 blocks, 0.0 minutes remaining.
clean_channel: 24/47 blocks, 0.0 minutes remaining.
clean_channel: 25/47 blocks, 0.0 minutes remaining.
clean_channel: 26/47 blocks, 0.0 minutes remaining.
clean_channel: 27/47 blocks, 0.0 minutes remaining.
clean_channel: 28/47 blocks, 0.0 minutes remaining.
clean_channel: 29/47 blocks, 0.0 minutes remaining.
clean_channel: 30/47 blocks, 0.0 minutes remaining.
clean_channel: 31/47 blocks, 0.0 minutes remaining.
clean_channel: 32/47 blocks, 0.0 minutes remaining.
clean_channel: 33/47 blocks, 0.0 minutes remaining.
clean_channel: 34/47 blocks, 0.0 minutes remaining.
clean_channel: 35/47 blocks, 0.0 minutes remaining.
clean_channel: 36/47 blocks, 0.0 minutes remaining.
clean_channel: 37/47 blocks, 0.0 minutes remaining.
clean_channel: 38/47 blocks, 0.0 minutes remaining.
clean_channel: 39/47 blocks, 0.0 minutes remaining.
clean_channel: 40/47 blocks, 0.0 minutes remaining.
clean_channel: 41/47 blocks, 0.0 minutes remaining.
clean_channel: 42/47 blocks, 0.0 minutes remaining.
clean_channel: 43/47 blocks, 0.0 minutes remaining.
clean_channel: 44/47 blocks, 0.0 minutes remaining.
clean_channel: 45/47 blocks, 0.0 minutes remaining.
clean_channel: 46/47 blocks, 0.0 minutes remaining.
clean_channel: 47/47 blocks, 0.0 minutes remaining.
Now doing final post-cleanup of the output.
Determining time window rejection thresholds...done.
Keeping 100.0% (240 seconds) of the data.
Use vis_artifacts to compare the cleaned data to the original.
Saving dataset...
/Users/arno/Python/eegprep/temp/tmpy3q40c74.set.result.set
Actual differences: rtol: 1.0, atol: 24.164947509765625
Traceback (most recent call last):
File "/Users/arno/Python/eegprep/tests/test_pipeline.py", line 54, in
test_pipeline()
File "/Users/arno/Python/eegprep/tests/test_pipeline.py", line 21, in test_pipeline
compare_eeg(EEG_py_ch['data'], EEG_mat_ch['data'], rtol=0.005, atol=1e-5, err_msg='clean_artifacts() channel cleaning Python vs MATLAB failed')
File "/Users/arno/Python/eegprep/src/eegprep/utils/testing.py", line 41, in compare_eeg
np.testing.assert_allclose(a, b, rtol=rtol, atol=atol, err_msg=err_msg)
File "/Users/arno/Python/eegprep/.venv/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 1592, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/local/Cellar/python@3.10/3.10.18/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/Users/arno/Python/eegprep/.venv/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 862, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.005, atol=1e-05
clean_artifacts() channel cleaning Python vs MATLAB failed
Mismatched elements: 882061 / 921600 (95.7%)
Max absolute difference: 24.164948
Max relative difference: 30366.475
x: array([ 3.532915e-05, 1.453137e+01, 9.636181e+00, ..., 2.230953e+00,
8.263576e+00, -1.832808e-03], dtype=float32)
y: array([ 4.164393e-09, 1.447033e+01, 9.514220e+00, ..., 2.351895e+00,
8.324990e+00, -2.160398e-07], dtype=float32)

[chkothe]

@arnodelorme do you have the .fdt file on hand that goes with the test file referenced in that script? It looks like that was not committed (and perhaps should not, and rather go on the sccntestdatasets S3 bucket).

[arnodelorme]

It is only 3Mb. For the sake of simplicity, I have committed it. We can remove it later if needed and store it on s3 (although I think it is nice to have a local file you can rely on). It is cluttering the Git history a little bit, even if we remove it, but it is small enough.

[chkothe]

Thanks; I've turned that into a unit test. The test passes over here. The equivalent MATLAB command was:

EEG = clean_artifacts(EEG,'FlatlineCriterion',5,'ChannelCriterion',0.87,'LineNoiseCriterion',4,'Highpass',[0.25, 0.75],'BurstCriterion',20,'WindowCriterion',0.25,'WindowCriterionTolerances',[-inf, 7]);

Maybe a single/double precision conversion issue again?

[arnodelorme]

Let me check. Thanks Christian, Arno

…

On Jul 25, 2025, at 18:13, Christian Kothe ***@***.***> wrote: chkothe left a comment (sccn/eegprep#5) Thanks; I've turned that into a unit test. The test passes over here. The equivalent MATLAB command was: EEG = clean_artifacts(EEG,'FlatlineCriterion',5,'ChannelCriterion',0.87,'LineNoiseCriterion',4,'Highpass',[0.25, 0.75],'BurstCriterion',20,'WindowCriterion',0.25,'WindowCriterionTolerances',[-inf, 7]); Maybe a single/double precision conversion issue again? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[arnodelorme]

What script were you running exactly? The test_clean_rawdata.py
I just pushed tests/test_pipeline.py (sorry, I forgot to do that earlier).

After pulling the latest changes (develop branch), on my MAC, I still get

Now doing final post-cleanup of the output.
Determining time window rejection thresholds...done.
Keeping 100.0% (240 seconds) of the data.
Use vis_artifacts to compare the cleaned data to the original.
Saving dataset...
/Users/arno/Python/eegprep/temp/tmpyreub6em.set.result.set
Actual differences: rtol: 1.0, atol: 24.164947509765625
Traceback (most recent call last):
  File "/Users/arno/Python/eegprep/tests/test_pipeline.py", line 54, in <module>
    test_pipeline()
  File "/Users/arno/Python/eegprep/tests/test_pipeline.py", line 21, in test_pipeline
    compare_eeg(EEG_py_ch['data'], EEG_mat_ch['data'], rtol=0.005, atol=1e-5, err_msg='clean_artifacts() channel cleaning Python vs MATLAB failed')
  File "/Users/arno/Python/eegprep/src/eegprep/utils/testing.py", line 41, in compare_eeg
    np.testing.assert_allclose(a, b, rtol=rtol, atol=atol, err_msg=err_msg)
  File "/Users/arno/Python/eegprep/.venv/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 1592, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/local/Cellar/python@3.10/3.10.18/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/Users/arno/Python/eegprep/.venv/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 862, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.005, atol=1e-05
clean_artifacts() channel cleaning Python vs MATLAB failed
Mismatched elements: 882061 / 921600 (95.7%)
Max absolute difference: 24.164948
Max relative difference: 30366.475
 x: array([ 3.532915e-05,  1.453137e+01,  9.636181e+00, ...,  2.230953e+00,
        8.263576e+00, -1.832808e-03], dtype=float32)
 y: array([ 4.164393e-09,  1.447033e+01,  9.514220e+00, ...,  2.351895e+00,
        8.324990e+00, -2.160398e-07], dtype=float32)