relative pathnames are not allowed in FILE_QUEUE

[skoenig-eo]

Dear all,

I have some trouble starting the processing of Level-1 Landsat data via force-level2. I have a regular level 2 prm file that links to the file queue. When I start the force-level2 process however, I get the following error:

5 images enqueued. Start processing with 3 CPUs
relative pathnames are not allowed in FILE_QUEUE

This is a bit weird because the file queue looks just as it is supposed to (here is a small version of it, in total I have > 1400 images to process):

/mnt/d/force_dlr/l1_lx/LC08_L1TP_193025_20130420_20200912_02_T1 QUEUED
/mnt/d/force_dlr/l1_lx/LC08_L1TP_193025_20130506_20200912_02_T1 QUEUED
/mnt/d/force_dlr/l1_lx/LC08_L1TP_193025_20130607_20200912_02_T1 QUEUED
/mnt/d/force_dlr/l1_lx/LC08_L1TP_193025_20130709_20200912_02_T1 QUEUED
/mnt/d/force_dlr/l1_lx/LC08_L1TP_193025_20130725_20200912_02_T1 QUEUED

As far as I understand, these are not relative filenames. I can even copy the paths into the console and list the files inside the folders, so that I don't think it's an issue of the force process not having access rights to the mounted disk or something. I looked up the lines of code in FORCE that produce the error and if I'm correct, force-level2 is just checking whether the filenames start with a forward slash (which they do).

Because of my current setup, I'm using the Ubuntu WSL under Windows 11, which might be causing the trouble in the first place I guess. Yet, I've done so for a long time and actually I'm able to process both Sentinel-2 file queues with force-level2 and single Landsat images with force-l2ps, which works like a charm and gives me just the results I desire.

One other thing I should note: the file queue was created manually. Well not exactly, but via a custom script, because not all images downloaded via level1-landsat were included, because I had to run the process multiple times. Yet, both the formatting of the queue as well as the character encoding are identical between my custom file and the one generated by level1-landsat. The only difference is the length of the file, as mine includes all images.

Does anybody have any thoughts on this? Did I miss something? Any help is appreciated; thanks a lot in advance!

Cheers,
Simon

My full queue as well as the level-2 parameter file are attached as txt files.
l1_lx_clean.txt
l2_lx_basic.txt

[ernstste]

Hi Simon,

your queue file uses windows style line endings. Try changing them to unix.

You can use this one liner in the wsl to create a queue find $PWD/.tar | sed 's/$/ QUEUED/' > queue.txt.

[ernstste]

I just saw that you seem to have folders rather than tar archives in your queue. Did you extract the files? force-level1-landsat will create the queue for the tar files, you say the queue files from l1-landsat and your script are identical though?

[skoenig-eo]

Hi Stefan,

thanks for the quick reply and thanks for the valuable thoughts on this!

Indeed, the windows-style file endings were the reason, and changing them to unix did the trick! 😄

---

Some elaboration regarding the folder/tar archive differences: indeed, I forgot to mention that I already untarred all files and hence listed folders in my own queue (and tried the same by changing the lines in the automatically generated queue). This is because for some tar files, I got the following error during testing (from the logfile):

/mnt/d/force_dlr/l1_lx_tar/LC08_L1TP_193025_20130725_20200912_02_T1.tar
: unknown Satellite Mission. Parsing metadata failed.

Indeed, this error was even produced while working wit a unix-style queue. If I change to folders however, the
whereas if I work with folders, I do not get any errors, wit an appropriate logfile (for the exact same scene):

/mnt/d/force_dlr/l1_lx/LC08_L1TP_193025_20130725_20200912_02_T1
: dc:  61.88%. wc:   0.17%. sc:   3.34%. cc:  66.32%. AOD: 0.2790. # of targets: 172/3891. 15 product(s) written. Success! Processing time: 05 mins 50 secs

Inside the WSL console, this is printed:

Couldn't open the directory: Not a directory
Couldn't open the directory: Not a directory
Couldn't open the directory: Not a directory

but it works fine with other images. Any thoughts on this? All data was properly downloaded via force-level1-landsat. This is not a big issue for me, because since I'm now working with the folders instead of the .tar files, I'm hopeful that this error will not come up again, but maybe this is something that you could keep in mind as a potential issue. If you want I can even raise an "official" GitHub issue.

Thanks again and looking forward to your reply!

Cheers,
Simon

[ernstste]

Thanks for the clarification, Simon.

This is interesting, I just checked the same scene and was getting the same errors.
@davidfrantz this is probably something to look into when you find the time. I think I remember talking to someone with a similar issue a while ago. We wouldn't notice any issues that come from extracting files on the fly since our operational processing handles that separately.

[thielfab]

I also vaguely remember that I had this issue and manual extraction fixed it I believe. But the number of affected scenes was minuscule in my case.

[skoenig-eo]

Thanks for the feedback you two!

Too bad that this seems to be an issue, but at least now I know that there is nothing wrong with my processing pipeline, but that this rather relates to certain datasets.

@thielfab maybe the number of problematic issues in this regard is also small and I just got unlucky that I had one in the small queue subset that I used in my test processing.
@ernstste @davidfrantz if you have any idea how to find out which/why scenes cause this error, I'll be happy to test my level-1 data in this regard.

[davidfrantz]

Hi all,

I thought FORCE wouldn't have problems with EOLs anymore, but it seems like force-level2 was still an exception. I just pushed a patch for this in dev.

Regarding the tars... I don't recall that we had this particular error in the past. As far as I remember, there were instances where tar returned non-zero exit codes for some reason, which I believe I fixed at some point.

The issue here is strange. The unknown Satellite Mission. Parsing metadata failed. message comes from force-l2ps. At this point, force-level2, or more specifically force-l2ps_ (with underscore) should have extracted the archive, and should have passed a folder without .tar to force-l2ps... Is this really happening with force-level2 or did you use force-l2ps directly?

[davidfrantz]

can you upload the file for me somewhere?

[ernstste]

Sure
https://box.hu-berlin.de/f/697c1a03ab3f48748872/?dl=1

[davidfrantz]

is this always happening with the last file in a queue if .tar?

[ernstste]

I can't tell you since I just tried the one scene that was mentioned in the post further up.
Any insights @skoenig-eo @thielfab ?

[skoenig-eo]

Hi all, indeed in my case it was the last file in the queue, but I'm honestly not sure if this was always the case during my pre-processing tests. My machine is still processing the dataat the moment. Once it's finished (1-2 days), I can run some tests (e.g. changing the order of the .tar files in the queue) and check again.

[davidfrantz]

Seems like somewhere in the bash script, a newline character is added to the last entry, and the tar command didn't like this.
It was hard to spot since \n is not printed as such.

I have pushed a small fix to dev, which forces a newline substitution... I hope this patches it up. Please test and give feedback ;)

[ernstste]

Just checked with this one file in the queue and it seems okay so far.

[skoenig-eo]

I'll test it ASAP, will let you know how it went!