Out of Memory force-higher-level tsa

[sulmonieri]

I am using Force 3.6.5.
When conducting the time series analysis with the function: force-higher-level and the parameterfile below, I always get an Out-of-Memory error. I am only processing Sentinel-2 Images.
tsa_swiss.txt
My available RAM is 1.9TB and 128 cores, so it should be plenty.
The setting of the parallel processing is:
NTHREAD_READ = 1
NTHREAD_COMPUTE = 1
NTHREAD_WRITE = 1
I tried higher numbers, but the Out-of-Memory error just happens sooner.
The datacube is the following:
PROJCS["CH1903+ / LV95",GEOGCS["CH1903+",DATUM["CH1903+",SPHEROID["Bessel 1841",6377397.155,299.1528128,AUTHORITY["EPSG","7004"]],TOWGS84[674.374,15.056,405.346,0,0,0,0],AUTHORITY["EPSG","6150"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4150"]],PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],PARAMETER["latitude_of_center",46.95240555555556],PARAMETER["longitude_of_center",7.439583333333333],PARAMETER["azimuth",90],PARAMETER["rectified_grid_angle",90],PARAMETER["scale_factor",1],PARAMETER["false_easting",2600000],PARAMETER["false_northing",1200000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Y",EAST],AXIS["X",NORTH],AUTHORITY["EPSG","2056"]]
5.650000
47.810000
2466029.933524
1297027.212477
110000.000000
110000.000000

It worked before using smaller Tiles of 10km but I would like to use the 110km.

[davidfrantz]

Hi @sulmonieri,

can you please post the exact error message?

and just out of curiosity: 1.9TB and 15GB per core? That`s a really nice setup :) What is the main purpose of that machine?

[sulmonieri]

Here is the exact message:

/storage/sulmonie/./tsa_swiss.sh: line 3: 122216 Killed singularity exec /storage/sulmonie/singularity/force.sif force-higher-level /storage/sulmonie/sentinel2/tsa/tsa_swiss.prm
slurmstepd: error: Detected 1 oom-kill event(s) in step 5371451.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
srun: error: node57: task 0: Out Of Memory
slurmstepd: error: Detected 1 oom-kill event(s) in step 5371451.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

This Machine is part of a cluster at our Swiss Federal Institute for Forest, Snow and Landscape Research WSL. It is one of two 'Big-memory' nodes each with 1.9TB RAM and 128 cores. There are also 56 other compute nodes each with 64GB RAM and 20 cores as well as three GPU nodes. The main purpose of the cluster is high-memory demanding simulation (e.g. global-climate coupled species disdribution models, phylogenetics) and deep learning.

[davidfrantz]

Hmm, I am not entirely sure whether this error is related to FORCE as it seems to be emitted by SLURM.

Did you define memory constraints in SLURM?
Are you able/allowd to bypass SLURM for a test run?

[sulmonieri]

This error is by SLURM, I should have written the question more precisely. My question is, where did I make a mistake with the parameter file since the analysis always worked with tile-size 10km and now with 110km, it doesn't? Or could it be with the previous level2 processing? I have T31TGM T32TLR T32TLS T32TLT T32TMR T32TMS T32TMT T32TNS T32TNT T32TPS T32TPT with 1920 original images. In the level2-processing, I got this error file level2_swiss_5357630.txt. The tile selection: X0000_Y0000 X0001_Y0000 X0002_Y0000 X0003_Y0000 X0000_Y0001 X0001_Y0001 X0002_Y0001 X0003_Y0001 (datacube, see my first question).

The memory constraint I set in SLURM is 1.7TB and I tried various mininum cpus (3-96). All trials gave the same error after different times.

[sulmonieri]

swiss_analysis.txt is the parameter file of the level 2 processing.

[davidfrantz]

I took a closer look at the details you provided, the parameter file and the datacube settings.

You indeed might be exceeding your memory...

You generated the L2 data with a tile size AND block size of 110km. This results in a reading of the entire tile in the higher level processing.
You can adjust with the BLOCK_SIZE parameter.

A small calculation (which does not cover the complete memory footprint that is used):

11000 columns x 11000 lines x 2 byte x T images x 12 indices * 3 blocks of data / (1024*1024*1024) [byte-to-GB]

This gives 4TB for T = 500

If you substantially reduce BLOCK_SIZE = 1000, e.g. to 1000m = 100 lines, this reduces to 36GB

11000 columns x 100 lines x 2 byte x T images x 12 indices * 3 blocks of data / (1024*1024*1024) [byte-to-GB]

PS: for the errors in the L2 processing, we should take this to another thread