v6.20.0 container images errors when pulling

[echoix]

Describe the bug
A clear and concise description of what the bug is.
When pulling the latest beta images (as of 2023-02-11) on Gitpod, docker doesn't complete and shows an error when extracting a layer.

To Reproduce
Steps to reproduce the behavior:

1. Go to Gitpod.
2. In the bash terminal, run docker pull oxsecurity/megalinter:beta or docker pull oxsecurity/megalinter-go:beta (was https://hub.docker.com/layers/oxsecurity/megalinter-go/beta/images/sha256-8fab0a400aa67089841912c597863e78a871c14bb6f1b26d57d947ca2b1c2807?context=explore and https://hub.docker.com/layers/oxsecurity/megalinter/beta/images/sha256-43fb5d5d36b4d623be36ea8c46958ed517e230fe17707093454e317d2edae384?context=explore) at time of bug
3. See error when extracting a layer that has node_modules.

Expected behavior
A clear and concise description of what you expected to happen.
Docker images can be pulled and run without any errors.

Screenshots
If applicable, add screenshots to help explain your problem.

failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown /node-deps/node_modules/ast-types-flow/lib/types.js: invalid argument

Additional context
Add any other context about the problem here.
Discovered in #2318

There seems to be a relation to UIDs, lchown, and maybe rootless containers. moby/moby#43576

In the go flavor, the error message was failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown /node-deps/node_modules/character-parser/LICENSE: invalid argument

[echoix]

Oops I pressed enter when writing the title and it sent.

[echoix]

Running the commands:

cat /etc/subuid
cat /etc/subgid

gave:

(.venv) gitpod /workspace/megalinter (dev/remove-apk-go) $ cat /etc/subgid
gitpod:100000:65536
nixbld1:165536:65536
nixbld2:231072:65536
nixbld3:296608:65536
nixbld4:362144:65536
nixbld5:427680:65536
nixbld6:493216:65536
nixbld7:558752:65536
nixbld8:624288:65536
nixbld9:689824:65536
nixbld10:755360:65536
nixbld11:820896:65536
nixbld12:886432:65536
nixbld13:951968:65536
nixbld14:1017504:65536
nixbld15:1083040:65536
nixbld16:1148576:65536
nixbld17:1214112:65536
nixbld18:1279648:65536
nixbld19:1345184:65536
nixbld20:1410720:65536
nixbld21:1476256:65536
nixbld22:1541792:65536
nixbld23:1607328:65536
nixbld24:1672864:65536
nixbld25:1738400:65536
nixbld26:1803936:65536
nixbld27:1869472:65536
nixbld28:1935008:65536
nixbld29:2000544:65536
nixbld30:2066080:65536
(.venv) gitpod /workspace/megalinter (dev/remove-apk-go) $ cat /etc/subuid
gitpod:100000:65536
nixbld1:165536:65536
nixbld2:231072:65536
nixbld3:296608:65536
nixbld4:362144:65536
nixbld5:427680:65536
nixbld6:493216:65536
nixbld7:558752:65536
nixbld8:624288:65536
nixbld9:689824:65536
nixbld10:755360:65536
nixbld11:820896:65536
nixbld12:886432:65536
nixbld13:951968:65536
nixbld14:1017504:65536
nixbld15:1083040:65536
nixbld16:1148576:65536
nixbld17:1214112:65536
nixbld18:1279648:65536
nixbld19:1345184:65536
nixbld20:1410720:65536
nixbld21:1476256:65536
nixbld22:1541792:65536
nixbld23:1607328:65536
nixbld24:1672864:65536
nixbld25:1738400:65536
nixbld26:1803936:65536
nixbld27:1869472:65536
nixbld28:1935008:65536
nixbld29:2000544:65536
nixbld30:2066080:65536

In here, containers/podman#2542, at containers/podman#2542 (comment) and containers/podman#2542 (comment), it seems we can do something in the image

[Kurt-von-Laven]

I was able to reproduce this issue on the current beta javascript flavor when using rootless Docker on Linux. Noting for myself that this issue doesn't reproduce on Windows according to #2318 (comment).

[echoix]

I kinda have an idea for a temporary workaround in order to be able to release, without having finished finding the problem and fixing it. I was still searching this afternoon, but at somewhere I read something and tried it and it was working.

So if we build the image in an already restricted environment, the UID:GID problem can't arise, (since it won't be mapped to an unavailable range? I don't know). But I reconfirmed the bug in another gitpod, and I'm able to build CI-light and it works.
All the tools I know of to explore the images, to see where the bad UIDs are, are unusable on images that can't be pulled, even on other platforms, or docker extensions, or online. Dive and slim.ai. It's because they are themselves a container mounting another maybe?

[echoix]

I have to go for now, but there's a ghcr package on my profile (it doesn't appear on my megalinter fork).

[Kurt-von-Laven]

I am still hitting this when using rootless Docker in the javascript flavor of MegaLinter v6.20.0.

[echoix]

@nvuillam could we add a section to release notes with know issues, and maybe direct users to add info or resolution steps/ideas here?