Trouble generating the Sentinel 2 Level 2 Product

[JonnyReGIF]

First of all, I would like to wish everyone a happy new year :)

I am currently working on Sentinel 2 and Sentinel 3 data for the year 2023 and stumbled across this nice piece of open source code. My idea is to (pre)process the Sentinel 2 data via FORCE and benefit from the data cubing when applying a linear regression to fill data gaps caused by clouds.

Over the holidays I worked my way through setting up the docker image, mounting directories and exchanging credentials between my windows system and the docker image to get the force-level1-csd up and running. Because i am new to linux and docker this was quite a hassle.

Nevertheless here is my problem, I am working on the tiles T32UPB, T32UPC, T32UQB, T32UQC and I have prepped. the SRTM elevation model (units in metres, nodata value != 0 and it covers my entire study area).

I am using the following parameter file:
l2params.txt

The corresponding file queue is:
l1_pool.txt

And the corresponding DEM is:
gdalinfo_dgm.txt

But after running:
docker run --rm -ti -v /c/force/data_force/:/opt/data davidfrantz/force force-level2 /opt/data/param/l2ps.prm

The only output I get is:

My force version is FORCE v. 3.7.11

I appreciate any suggestions and help.
Thanks in advance :)

[ernstste]

Hi JonnyReGif,

could you clarify whether this is the output you are receiving for all of your processed tiles or just for some of them?
I don't see anything wrong with the screenshot, this is what you would expect for some tiles, depending on the corresponding Sentinel-2 tile and orbit.

Best,
Stefan

[JonnyReGIF]

Hi Stefan,

thanks for the reply :)

Yes the screenshot above is the only output i am receiving for all my processed tiles.
The complete output from the docker command with the param-file shown in the question is:

docker run --rm -ti -v /c/force/data_force/:/opt/data davidfrantz/force force-level2 /opt/data/param/l2ps.prm

+---europe
¦ ¦ datacube-definition.prj
¦ ¦ equi7-tile-dictionary.txt
¦ ¦
¦ +---X0049_Y0036
¦ 20231010_LEVEL2_SEN2A_BOA.tif
¦ 20231010_LEVEL2_SEN2A_OVV.jpg
¦ 20231010_LEVEL2_SEN2A_QAI.tif

When i load this raster in R, I get:

> library(terra)
terra 1.7.46
> rast("C:\\force\\data_force\\level2\\europe\\X0049_Y0036\\20231010_LEVEL2_SEN2A_BOA.tif")
class       : SpatRaster 
dimensions  : 5000, 5000, 10  (nrow, ncol, nlyr)
resolution  : 20, 20  (x, y)
extent      : 4900000, 5e+06, 1900000, 2e+06  (xmin, xmax, ymin, ymax)
coord. ref. : Azimuthal_Equidistant 
source      : 20231010_LEVEL2_SEN2A_BOA.tif 
names       :   BLUE,  GREEN,    RED, REDEDGE1, REDEDGE2, REDEDGE3, ... 
min values  : -32768, -25438, -32768,   -32768,   -32768,   -32768, ... 
max values  :  32767,  32767,  32767,    32767,    32767,    32767, ... 

Also, what I should have included in the question are the errors I get for most of the scenes:
lvl2_processing_error.txt

the corrosponding log files are empty except for the path to the Scene directory, like /opt/data/level1/T32UQB/S2A_MSIL1C_20230403T100551_N0509_R022_T32UQB_20230403T135016.SAFE:

Sorry for the drippy droppy of information, please let me know if I can aid with any further clarification and many thanks for your time.

Best,
Jonny

[davidfrantz]

Hi @JonnyReGIF,

it seems most of your processes were killed by the kernel. This is usually related to memory limitations of your rig, or using too many processes at once.

As the only successful image is covering only a small area, this supports this theory.

How much RAM, and how many CPUs do you have?

My recommendation: 8GB per process (NPROC in parameter file).

Cheers,
David

[JonnyReGIF]

Hi David,

thanks for the reply.

I am currently testing FORCE on my personal laptop with 16GB RAM and 6 CPU cores.

But unfortunately the following settings are crashing my system too:

NPROC = 8
NTHREAD = 2
PARALLEL_READS = FALSE
DELAY = 20
TIMEOUT_ZIP = 30

I will try to switch to a more powerful machine and run it again.
Or is there any way to access CODE:DE FORCE products without a user account?

Thanks to you and @ernstste for the support.
Really awesome.

Best,
Jonny

[ernstste]

Hi Jonny,

try reducing the processes (NPROC) according to David's recommendation.
NPROC = 1 --> min. 8GB RAM, NPROC = 2 --> min. 16 GB RAM. Running 8 processes in parallel on a mere 16GB RAM is a bit daring... ;)

Stefan