memory error

[aileenfisher]

Could probably get this to run on a box with higher RAM; I have only like 8G and 8G SWAP on this box local. But (and I have very little knowledge of processing) is there a way to make less memory-heavy? It was like 90% memory and 3% of my 4 CPU and then crashed out.

(infrapy_env) bash-4.2$ pwd
/data/studies/PyGS/git/infrapy/examples
(infrapy_env) bash-4.2$ infrapy run_sd --local-wvfrms 'data/YJ.BRP1..EDF.SAC' --cpu-cnt 4 --spectral-option cwt --cluster-min-samples 500 --cluster-eps 5

#####################################

InfraPy

Spectral Detection Analyses

#####################################

Data parameters:
local_wvfrms: data/YJ.BRP1..EDF.SAC
local_latlon: None
local_detect_label: None
cpu_cnt: 4

Algorithm parameters:
spectral_option: cwt
morlet_omega0: 12.0
freq_min: 1.0
freq_max: 20.0
signal_start: None
signal_end: None
window_len: 900.0
window_step: 450.0
p_value: 0.01
freq_tm_factor: 35.0
cluster_eps: 5.0
cluster_min_samples: 500
cpu_cnt: 4

Loading local data from data/YJ.BRP1..EDF.SAC

Data summary:
YJ.BRP1..EDF 2012-04-09T18:00:00.008300Z - 2012-04-09T18:19:59.998300Z

Running spectral detection (sd) analysis...
Progress: [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>]
Clustering into detections...
Traceback (most recent call last):
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/bin/infrapy", line 33, in
sys.exit(load_entry_point('infrapy', 'console_scripts', 'infrapy')())
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/data/studies/PyGS/git/infrapy/infrapy/cli/cli_detection.py", line 917, in run_sd
det_list = spectral.run_sd(stream[0], spectral_option, morlet_omega0, [freq_min, freq_max], 0.8, p_value, window_len, window_step, freq_tm_factor, cluster_eps, cluster_min_samples, pl)
File "/data/studies/PyGS/git/infrapy/infrapy/detection/spectral.py", line 217, in run_sd
clustering = DBSCAN(eps=clustering_eps, min_samples=clustering_min_samples).fit(spec_dets_logf)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/sklearn/base.py", line 1151, in wrapper
return fit_method(estimator, *args, **kwargs)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/sklearn/cluster/_dbscan.py", line 410, in fit
dbscan_inner(core_samples, neighborhoods, labels)
File "sklearn/cluster/_dbscan_inner.pyx", line 34, in sklearn.cluster._dbscan_inner.dbscan_inner
MemoryError: std::bad_alloc
(infrapy_env) bash-4.2$ infrapy run_sd --local-wvfrms 'data/YJ.BRP1..EDF.SAC' --spectral-option cwt --cluster-min-samples 500 --cluster-eps 5

#####################################

InfraPy

Spectral Detection Analyses

#####################################

Data parameters:
local_wvfrms: data/YJ.BRP1..EDF.SAC
local_latlon: None
local_detect_label: None

Algorithm parameters:
spectral_option: cwt
morlet_omega0: 12.0
freq_min: 1.0
freq_max: 20.0
signal_start: None
signal_end: None
window_len: 900.0
window_step: 450.0
p_value: 0.01
freq_tm_factor: 35.0
cluster_eps: 5.0
cluster_min_samples: 500

Loading local data from data/YJ.BRP1..EDF.SAC

Data summary:
YJ.BRP1..EDF 2012-04-09T18:00:00.008300Z - 2012-04-09T18:19:59.998300Z

Running spectral detection (sd) analysis...
Progress: [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>]
Clustering into detections...
Traceback (most recent call last):
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/bin/infrapy", line 33, in
sys.exit(load_entry_point('infrapy', 'console_scripts', 'infrapy')())
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/data/studies/PyGS/git/infrapy/infrapy/cli/cli_detection.py", line 917, in run_sd
det_list = spectral.run_sd(stream[0], spectral_option, morlet_omega0, [freq_min, freq_max], 0.8, p_value, window_len, window_step, freq_tm_factor, cluster_eps, cluster_min_samples, pl)
File "/data/studies/PyGS/git/infrapy/infrapy/detection/spectral.py", line 217, in run_sd
clustering = DBSCAN(eps=clustering_eps, min_samples=clustering_min_samples).fit(spec_dets_logf)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/sklearn/base.py", line 1151, in wrapper
return fit_method(estimator, *args, **kwargs)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/sklearn/cluster/_dbscan.py", line 410, in fit
dbscan_inner(core_samples, neighborhoods, labels)
File "sklearn/cluster/_dbscan_inner.pyx", line 34, in sklearn.cluster._dbscan_inner.dbscan_inner
MemoryError: std::bad_alloc
(infrapy_env) bash-4.2$

[aileenfisher]

Also:

(infrapy_env) bash-4.2$ infrapy run_sd --local-wvfrms 'data/YJ.BRP1..EDF.SAC' --cpu-cnt 4

#####################################

InfraPy

Spectral Detection Analyses

#####################################

Data parameters:
local_wvfrms: data/YJ.BRP1..EDF.SAC
local_latlon: None
local_detect_label: None
cpu_cnt: 4

Algorithm parameters:
spectral_option: spectrogram
morlet_omega0: 12.0
freq_min: 1.0
freq_max: 20.0
signal_start: None
signal_end: None
window_len: 900.0
window_step: 450.0
p_value: 0.01
freq_tm_factor: 35.0
cluster_eps: 10.0
cluster_min_samples: 40
cpu_cnt: 4

Loading local data from data/YJ.BRP1..EDF.SAC

Data summary:
YJ.BRP1..EDF 2012-04-09T18:00:00.008300Z - 2012-04-09T18:19:59.998300Z

Running spectral detection (sd) analysis...
Progress: [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>]
Clustering into detections...
Identified 5 detections.

Writing detection results using label: data/YJ.BRP1_2012.04.09_18.00.00-18.19.59
(infrapy_env) bash-4.2$ infrapy plot sd --local-wvfrms 'data/YJ.BRP1..EDF.SAC'

#####################################

InfraPy

Spectral Detection (sd)

Visualization

#####################################

Data parameters:
local_wvfrms: data/YJ.BRP1..EDF.SAC
local_latlon: None
local_detect_label: None

Loading local data from data/YJ.BRP1..EDF.SAC
Plotting spectrogram with detection info...

Traceback (most recent call last):
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/bin/infrapy", line 33, in
sys.exit(load_entry_point('infrapy', 'console_scripts', 'infrapy')())
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/data/studies/PyGS/git/infrapy/infrapy/cli/cli_visualization.py", line 511, in sd
det_vis.plot_sd(stream[0], det_list, [freq_min, freq_max], spec_option=spectral_option, morlet_omega0=morlet_omega0, output_path=figure_out, show_fig=show_figure)
File "/data/studies/PyGS/git/infrapy/infrapy/detection/visualization.py", line 127, in plot_sd
a[2].get_shared_y_axes().join(a[2], a[1])
AttributeError: 'GrouperView' object has no attribute 'join'
(infrapy_env) bash-4.2$

[aileenfisher]

Works with --single-det-index added, but I'm pulling straight from the --help examples. I'm assuming there's some default stuff it wants but it's hard to track all that down. Would request either forcing inputs from the user or better error messaging, documentation and template config files with every option and explanations if there needs to be a certain combination of options.