How to create tile allow list for the parameter file for level 2 processing? Also, recommendations on multiprocessing and multi-threading for the level 2 processing.

[paulvpop]

Hello. I am currently using the very useful FORCE software for radiometric corrections of LANDSAT 8 OLI imagery for land cover - land use classification of a region in north-east India. I read through the documentation of FORCE, and from my understanding the tile allow-list is automatically created once I run the FORCE L2PS module. But the parameter file for the FORCE L2PS module itself has a parameter called "FILE_TILE" which is for the tile allow-list. How do I generate the tile-allow list before the level 2 processing? This will be useful as I am using a single CPU (32 GB RAM) for the processing and any way to cut down on the processing load and time will be highly appreciated. The tile-allow list may reduce the load and time required for processing.

Also, how many processes and threads is good for a single CPU with a 32 GB RAM? The CPU processor is a 13th Gen Intel(R) Core(TM) i9-13900 2.00 GHz (24 cores and 32 logical processors) and GPU is a NVIDIA T1000 8 GB along with an Intel UHD Graphics 770.

[davidfrantz]

Hi @paulvpop,

this parameter is only meant to suppress output for specific tiles. It won't affect processing time.

If you want to generate a tile allow list in advance, you might want to use force-tile-extent.

With 32GB RAM you will be able to process 5 Landsat images in parallel. Thus, you could use 6 threads per process.

NPROC = 5
NTHREAD = 6

[paulvpop]

Hello @davidfrantz Thanks for your quick response.

Got it.

I went to the documentation for force-tile-extent (it seems to be old. More on this below), and for this a datacube-definition.prj was needed (as shown here). So, I made one by writing the following and saving it using unicode style file:

PROJCS["WGS 84 / India NSF LCC",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["latitude_of_origin",24],PARAMETER["central_meridian",80],PARAMETER["standard_parallel_1",12.472955],PARAMETER["standard_parallel_2",35.1728044444444],PARAMETER["false_easting",4000000],PARAMETER["false_northing",4000000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","7755"]]
93.9574520
29.3477290
49.2859027
-14.0282153
30000.000000
3000.0000000

I have faced a few issues. I will explain them in detail:

The projection I chose was EPSG:7755.

When I run the command davidfrantz/force gdalsrsinfo -e EPSG:7755 I get the details correctly and when I run davidfrantz/force gdalsrsinfo -V -e EPSG:7755 , I get the "Validate Succeeds" message with the details.

However, when I Copy Text the wkt from https://epsg.io/7755 to get it in one line, and run the davidfrantz/force gdalsrsinfo -V 'PROJCS["WGS 84 / India NSF LCC",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["latitude_of_origin",24],PARAMETER["central_meridian",80],PARAMETER["standard_parallel_1",12.472955],PARAMETER["standard_parallel_2",35.1728044444444],PARAMETER["false_easting",4000000],PARAMETER["false_northing",4000000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","7755"]]' I get the following error:

"ERROR 1: missing [
ERROR 1: ERROR - failed to load SRS definition from PROJCS[WGS 84 / India NSF LCC,GEOGCS[WGS 84,DATUM[WGS_1984,SPHEROID[WGS 84,6378137,298.257223563]],PRIMEM[Greenwich,0],UNIT[degree,0.0174532925199433,AUTHORITY[EPSG,9122]],AUTHORITY[EPSG,4326]],PROJECTION[Lambert_Conformal_Conic_2SP],PARAMETER[latitude_of_origin,24],PARAMETER[central_meridian,80],PARAMETER[standard_parallel_1,12.472955],PARAMETER[standard_parallel_2,35.1728044444444],PARAMETER[false_easting,4000000],PARAMETER[false_northing,4000000],UNIT[metre,1],AXIS[Easting,EAST],AXIS[Northing,NORTH],AUTHORITY[EPSG,7755]]"

But that doesn't make sense. I validated the wkt in QGIS too. Nothing's missing. Do you know why this happens?

I had created a bounding box for my study area to extract the north-west point's geographic longitude and latitude (in WGS84 format - 93.9574520
29.3477290 respectively) and the projected longitude (X) and latitude (Y) (in EPSG:7755 format - 49.2859027
-14.0282153 respectively). Is this correct? I am asking because I am a little confused by the custom projection you had used in this example. 2456026.250000,
4574919.500000 are longitude and latitude of the projected coordinate system?

3. The documentation for force-tile-extent seems old, but the documentation in program itself is updated. I get the following when I run davidfrantz/force force-tile-extent -h:

"Usage: force-tile-extent [-hvi] [-d datacube-dir] [-a allow-list] [-j jobs] input-vector

optional:
-h = show this help
-v = show version
-i = show program's purpose

-d = directory of existing datacube
defaults to current directory
'datacube-definition.prj' needs to exist in there
-a = tile allow-list to restrict the processing extent
The extent will be computed based on the input AOI
Defaults to './allow-list.txt'
-j = number of jobs, defaults to 100%

Positional arguments:

• input-vector: a polygon vector file for the area of interest (AOI)

---

see https://force-eo.readthedocs.io/en/latest/components/auxilliary/tile-extent.html"

So, I modified the code to force-tile-extent -d /home/docker/force/force/datacube-dir -a /home/docker/force/force/allow-list.txt -j 100% /home/docker/force/force/Siang_SA_Prj/Siang_SA_Prj.shp

and got the following ouput:

Suggested Processing extent parameters:
X_TILE_RANGE = 178 183
Y_TILE_RANGE = -156 -149
FILE_TILE = /home/docker/force/force/allow-list.txt

Maybe, you can give this as an example for your documentation page.

4. I am getting negative tile numbers. Here's the file output of the above command:

48
X0178_Y-149
X0178_Y-150
X0178_Y-151
X0178_Y-152
X0178_Y-153
X0178_Y-154
X0178_Y-155
X0178_Y-156
X0179_Y-149
X0179_Y-150
X0179_Y-151
X0179_Y-152
X0179_Y-153
X0179_Y-154
X0179_Y-155
X0179_Y-156
X0180_Y-149
X0180_Y-150
X0180_Y-151
X0180_Y-152
X0180_Y-153
X0180_Y-154
X0180_Y-155
X0180_Y-156
X0181_Y-149
X0181_Y-150
X0181_Y-151
X0181_Y-152
X0181_Y-153
X0181_Y-154
X0181_Y-155
X0181_Y-156
X0182_Y-149
X0182_Y-150
X0182_Y-151
X0182_Y-152
X0182_Y-153
X0182_Y-154
X0182_Y-155
X0182_Y-156
X0183_Y-149
X0183_Y-150
X0183_Y-151
X0183_Y-152
X0183_Y-153
X0183_Y-154
X0183_Y-155
X0183_Y-156

So, I went through the documentation and it has already been addressed in the "Frequently Asked Questions"

"The tile IDs of the processed Level 2 data have negative numbers, e.g. X-100_Y0100.

Make sure that the origin of the target grid (ORIGIN LAT / ORIGIN LON) is in the North-West of your study area. Potentially, you have accidentally swapped latitude and longitude. Note that a geographic location in the North-West is not necessarily North-West in the output coordinate system, too (for an example see Fig. 14). Although not recommended, higher-level FORCE functions should be able to digest negative tile numbers (note that we did not test this exhaustively)."

I couldn't find Fig. 14. Could you share the link with me? I am trying to figure out why I am getting negative tile numbers even though I selected the north-west points of the bounding box (attaching it here)
Rectangle_bounding_Siang.zip
containing the study area in both geographic and projected coordinate systems, and made sure that the longitude and latitude are in the right place.

Any idea why I am still getting negative tile numbers?

5. Also, a related minor issue I found was there are things like VIII.M and VII.X mentioned in some of the documentation pages. For example "Any gridded data cube (containing a data cube definition file, see VII.M) can be given as 1st argument.". What do these roman numerals refer to?

Thanks in advance for all the help.

With kind regards,
Paul

[davidfrantz]

Hi @paulvpop, can you please use a new thread for this as the original question was already answered?

[paulvpop]

Hello @davidfrantz Sorry for the delay in response. Yes, will do that.