force-level2 stops with "/home/docker/bin/force-l2ps_: line 161: 123 Killed"

[davedavemckay]

Hi, I'm attempting a test run with the docker command:

docker run -v /data/Sentinel-2/L1C/2023/04/01:/level1 -v /data/Copernicus/copernicus-dem-30m-egm2008:/dem -v /home/dm/force:/force --user "$(id -u):$(id -g)" davidfrantz/force force-level2 /force/param/l2ps.prm

The output looks like this:

1 images enqueued. Start processing with 4 CPUs

Computers / CPU cores / Max jobs to run
1:local / 4 / 1

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s /home/docker/bin/force-l2ps_: line 161:   123 Killed                  $EXE $TODO $PRM >> $LOGFILE
ETA: 0s Left: 0 AVG: 0.00s  local:0/1/100%/442.0s

l2ps.prm is the automatically generated param file with changes to directories, NPROC=4 and NTHREAD=1, otherwise default.

I have a filelist.txt containing a single line:
/level1/S2A_MSIL1C_20230401T110621_N0509_R137_T30UWA_20230401T144922.SAFE QUEUED

I get a single log file, S2A_MSIL1C_20230401T110621_N0509_R137_T30UWA_20230401T144922.SAFE.log, contents:

/level1/S2A_MSIL1C_20230401T110621_N0509_R137_T30UWA_20230401T144922.SAFE
:

I'm running on a fairly modest VM (8 GB RAM, 4 CPUs). The /data filesystem is an nfs-mounted 1TB disk, so no issues there with just one S2 L1C file.

I also tried NPROC=1 just in case it was being killed because I was using all of the cores, but I get the same behaviour.

Any ideas?

Thanks,
Dave

[davedavemckay]

Forgot to add, a "europe" folder is added to the level2 dir with a datacube-definition.prj file. Contents below.

PROJCS["BU MEaSUREs Lambert Azimuthal Equal Area - EU - V01",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["degree",0.0174532925199433]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["false_easting",0.0],PARAMETER["false_northing",0.0],PARAMETER["longitude_of_center",20],PARAMETER["latitude_of_center",55],UNIT["meter",1.0]]
-76.852360
41.477561
-5505560.000000
3346245.000000
150000.000000
1500.000000

No other output as far as I can see.

[davedavemckay]

Just seen this is probably the RAM filling up - I'm trying now with 1 PROC and 4 threads.

[davedavemckay]

Threw a bigger VM at it and got up and running with NPROC=16, NTHREAD=2 with 896 GB RAM.

[davidfrantz]

I am glad that it worked and that you figured it out by yourself. Tough 896GB RAM is probably an overkill ;) I believe it would be safe to reduce to 128GB with the 16 processes used.

Cheers,
David

[davedavemckay]

Haha - yes, I did go from the smallest to the largest VM I have access to!
Thanks for the tip!
Cheers,
Dave