No valid parameter file + directory not existing/readable

[skoenig-eo]

Hi all,

I'm quite sure that this is a problem specific to me, so I rather post this here than open an issue.
I am working on a Ubuntu virtual machine using azure (Ubuntu 20.04, FORCE 3.6.5). I have my Level 1 archive ready and all prerequisies are complete as well. When trying to start force-level2, a number of unpleasant things happen however:

• bash tells me No valid parameter file! Parsing parameter file failed.
• still, the regular force-level2 console output is still visible and the number of completed jobs is increasing. Yet, the image logfiles state /media/storage/bb_comb/L1/192026/LT05_L1GS_192026_19850227_20170219_01_T2: /media/storage/bb_comb/L1/192026/LT05_L1GS_192026_19850227_20170219_01_T2 is not existing/readable or something similar, depending on the image.

I found this quite strange, because I have no reason to suspect that my parameter file is incorrect or corrupted (I actually tried out multiple files in multiple locations, and none worked out). And I think the file is indeed working, because the file queue is read, otherwise the second error wouldn't occur, right?
The second error confuses me as well, because while being on a mounted virtual hdd, the directory /media/storage/bb_comb should be both readable and writable (I hope I made sure with sudo chown -R user:user /media/storage/bb_comb/). Indeed, I can write and delete stuff as I also pointed force-level1-csd to this directory.

Does anybody have an idea what could be the problem with this? Am I overlooking anything? I hope you can help me, I've been trying for days now. My Parameter File is attached.

Thanks a lot!

Best
Simon

++PARAM_LEVEL2_START++

# INPUT/OUTPUT DIRECTORIES
# ------------------------------------------------------------------------
FILE_QUEUE = /media/storage/bb_comb/full.pool
DIR_LEVEL2 = /media/storage/bb_comb/l2
DIR_LOG = /media/storage/bb_comb/aux/log
DIR_TEMP = /media/storage/bb_comb/temp

# DIGITAL ELEVATION MODEL
# ------------------------------------------------------------------------
FILE_DEM = NULL
DEM_NODATA = -32767

# DATA CUBES
# ------------------------------------------------------------------------
DO_REPROJ = TRUE
DO_TILE = TRUE
FILE_TILE = NULL
TILE_SIZE = 30000
BLOCK_SIZE = 3000
RESOLUTION_LANDSAT = 30
RESOLUTION_SENTINEL2 = 10
ORIGIN_LON = 13
ORIGIN_LAT = 50
PROJECTION = GLANCE7
RESAMPLING = CC

# RADIOMETRIC CORRECTION OPTIONS
# ------------------------------------------------------------------------
DO_ATMO = TRUE
DO_TOPO = TRUE
DO_BRDF = TRUE
ADJACENCY_EFFECT = TRUE
MULTI_SCATTERING = TRUE

# WATER VAPOR CORRECTION OPTIONS
# ------------------------------------------------------------------------
DIR_WVPLUT = /media/storage/bb_comb/aux/wvdb
WATER_VAPOR = NULL

# AEROSOL OPTICAL DEPTH OPTIONS
# ------------------------------------------------------------------------
DO_AOD  = TRUE
DIR_AOD  = NULL

# CLOUD DETECTION OPTIONS
# ------------------------------------------------------------------------
ERASE_CLOUDS = FALSE
MAX_CLOUD_COVER_FRAME = 100
MAX_CLOUD_COVER_TILE  = 100
CLOUD_BUFFER  = 300
SHADOW_BUFFER = 90
SNOW_BUFFER   = 30
CLOUD_THRESHOLD  = 0.225
SHADOW_THRESHOLD = 0.02

# RESOLUTION MERGING
# ------------------------------------------------------------------------
RES_MERGE = IMPROPHE

# CO-REGISTRATION OPTIONS
# ------------------------------------------------------------------------
DIR_COREG_BASE = NULL
COREG_BASE_NODATA = -9999

# MISCELLANEOUS OPTIONS
# ------------------------------------------------------------------------
IMPULSE_NOISE = TRUE
BUFFER_NODATA = FALSE

# TIER LEVEL
# ------------------------------------------------------------------------
TIER = 2

# PARALLEL PROCESSING
# ------------------------------------------------------------------------
# Multiprocessing options (NPROC, DELAY) only apply when using the batch
# utility force-level2. They are not used by the core function force-l2ps.
# ------------------------------------------------------------------------
NPROC = 3
NTHREAD = 2
PARALLEL_READS = FALSE
DELAY = 3
TIMEOUT_ZIP = 30

# OUTPUT OPTIONS
# ------------------------------------------------------------------------
OUTPUT_FORMAT = GTiff
OUTPUT_DST = TRUE
OUTPUT_AOD = TRUE
OUTPUT_WVP = TRUE
OUTPUT_VZN = TRUE
OUTPUT_HOT = TRUE
OUTPUT_OVV = TRUE

++PARAM_LEVEL2_END++

[maawoo]

Maybe related to [...] The file can be edited with any text editor. However, make sure that you are using Unix End-of-Line \n. MOST errors are because of parameter files with Windows End-of-Line \r\n. described here?

Did you open/edit the parameter file outside your VM?
I remember something similar happening to me with a completely different software.

[skoenig-eo]

Thanks for the suggestion! I thought of that as well and tried it out with a param file that had never left my VM, but it still didn't work. Also checked the line endings according to this, and it was a regular ASCII file.

[davidfrantz]

Hi Simon,

if I understand correctly, force-level2 keeps running, but the spawned sub-processes are all exiting with some error message.

I am not sure if both errors are happening on the same image. If you get one error, you shouldn't get the other one (parameter file error or image read permission error).

Somehow, this all looks like your file permissions are not correct.

Are you running locally in your VM or with Docker?
Are you running with the same user that has generated the L1 images and the parameter file?

Can you please also post the following output:

whoami
id
umask
ls -l parameter-file
ls -l /media/storage/bb_comb/L1/192026/LT05_L1GS_192026_19850227_20170219_01_T2

Cheers,
David

[skoenig-eo]

Hi David,

thanks for the reply. While trying your suggestions all morning, I noticed that the second error was actually due stupid me renaming a folder after creating the L1 archive 😬. Sorry for wasting your time on that.

Anyway, as this error is solved, things are running now. Regardless, I still get the error No valid parameter file! Parsing parameter file failed. I can sure live with this error as long as it works, but maybe it's still interesting to investigate what causes it.

You can find my ansewers to your suggested commands below. I run the VM locally in the VM and only have one user (whereby I get the errors regardless of including sudo or not. Hope that helps!

Thanks a million!

Cheers
Simon

whoami --> simon
id --> uid=1000(simon) gid=1000(simon) groups=1000(simon),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),50(staff),108(lxd),114(netdev)
umask --> 0002
ls -l /media/storage/bb_comb/l2.prm --> -rw-rw-r-- 1 simon simon 16907 Apr 16 10:22 /media/storage/bb_comb/l2.prm

ls -l /media/storage/bb_comb/L1/192026/LT05_L1GS_192026_19850227_20170219_01_T2 -->

total 86172
-rw-rw-r-- 1 simon simon    34885 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_ANG.txt
-rw-rw-r-- 1 simon simon 10083800 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_B1.TIF
-rw-rw-r-- 1 simon simon 13458316 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_B2.TIF
-rw-rw-r-- 1 simon simon 14762043 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_B3.TIF
-rw-rw-r-- 1 simon simon 13026367 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_B4.TIF
-rw-rw-r-- 1 simon simon 16618564 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_B5.TIF
-rw-rw-r-- 1 simon simon  3658288 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_B6.TIF
-rw-rw-r-- 1 simon simon 14807342 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_B7.TIF
-rw-rw-r-- 1 simon simon  1748527 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_BQA.TIF
-rw-rw-r-- 1 simon simon     6685 Apr  7 09:28 LT05_L1GS_192026_19850227_20170219_01_T2_MTL.txt
-rw-rw-r-- 1 simon simon     8686 Apr  7 09:28 README.GTF

[davidfrantz]

Hi Simon,

this reminds me of an old (unsolved) post in the old Google Group: https://groups.google.com/g/force_eo/c/m1mgYWd1Xwc/m/EU57EmuAAgAJ

I really don't understand this... If this message is printed, the program should actually exit.

Does this happen for each image? If not, can you isolate the image that is causing this?
Does this happen when you call force-l2ps directly?

Cheers,
David

[skoenig-eo]

Hi David,

the error No valid parameter file! Parsing parameter file failed is actually printed with every image, while the outputs look great and are exactly like I want them to be (screenshots of my four-image example plus outputs attached). So I can sure live with this error! ☺️

When calling force-l2ps, I don't get the parameter file error: sudo force-l2ps /media/storage/bb_comb/L1/191026/LT05_L1TP_191026_19840929_20170220_01_T1 /media/storage/bb_comb/l2.prm
dc: 62.81%. wc: 0.42%. sc: 1.93%. cc: 42.05%. AOD: 0.1790. # of targets: 305/4887. 52 product(s) written. Success! Processing time: 06 mins 26 secs

Cheers,
Simon

[davidfrantz]

It eludes me... shouldn't look like this.

But it must be related to the bash script...

Can you potentially verify if this happens in the docker container as well?
Could you try with the latest dev image?

docker pull davidfrantz/force:dev

Cheers,
David

[skoenig-eo]

Hi David,

I can sure try it out once the (now-running) processing is completed in a few days - I hope this is okay! I'll let you know once I have the respective results.

Thanks for your help so far!

Cheers,
Simon

[davidfrantz]

Thanks Simon,

it would be great to get behind this.

Cheers

[davidfrantz]

Hi @skoenig-eo
as this keeps popping up: #102 (comment)
Did you have a chance to test the Docker version?
Cheers,
David

[davidfrantz]

Hi @skoenig-eo ,

when testing @cxxuuuuuuu parameter file (#102), I was indeed able to reproduce this behaviour (by chance)...

For every image, this error message pops up, but processing continues.

No valid parameter file!
Parsing parameter file failed.

Now, the reason for this is really strange. When executing the preprocessing in batch mode (force-level2), there is a test whether the program was compiled in DEBUG mode (because otherwise you likely freeze your system bc of too much parallel debug output that keeps overwriting itself). For this, there is a hidden option in the core module force-l2ps, which either returns 0 when OK, or 1 when compiled in DEBUG mode. The second line of this example prints the exit status:

frantzda@PC-19-0799:~/temp$ force-l2ps ? ?
frantzda@PC-19-0799:~/temp$ echo $?
0

Now comes some weird bash frenzy that I wasn't aware of. If you are currently in your home directory, the ? sign seems to work differently :/ Apparently, I never call the programs directly from my home directory, thus I couldn't reproduce your issue so far....

frantzda@PC-19-0799:~$ force-l2ps ? ?
No valid parameter file!
Parsing parameter file failed.
frantzda@PC-19-0799:~$ echo $?
255

The solutions are straightforward though.

1. Do not use FORCE when in your home directory.
2. upgrade to the development version, where this mechanic was already implemented differently a while ago.

Mystery solved :) I hope it works on your end, too.

Cheers,
David