[WIP] Refactor Dockerfile

[Raimo33]

This is a draft PR to refactor the dockerfile according to issues:

• Refactor Dockerfile after the removal of all built-in Python plugins #8409
• Docker: verify GPG keys #8405

[ShahanaFarooqui]

@Raimo33 As per @cdecker's suggestion here, we should modify the Dockerfile to run as a non-root user for improved security. Since we're already refactoring the Dockerfile, could we incorporate this change as well?

[Raimo33]

I'm removing the manual handling of tini in favor of a user controlled approach, more docker native and friendly.

ENTRYPOINT  [ "/usr/bin/tini", "-g", "--", "./entrypoint.sh" ]

becomes:

ENTRYPOINT  ["/entrypoint.sh"]

and the user is free to run the container with tini by using the official --init flag as follows:

docker run --init ...

[Raimo33]

The Dockerfile now should also support building the image from any host architecture as well. It previously only suppported amd64 to any. now it should be any to any, but I've no way of testing it

[Raimo33]

make install currently calls git describe to know the version of the program. This means that the .git/ folder needs to be copied into the Dockerfile for it to work. But copying the .git/ folder almost always invalidates the cache of the COPY . . command.

I'd like to explore ways to not include the .git/ folder (aka placing it in the .dockerignore). Maybe we can add an option to the Makefile that doesn't call git describe

[Raimo33]

also I wonder why our Makefile make install checks the availability of avx intrinsics:

#80 511.3 checking whether C compiler accepts -msse3... no
#80 512.0 checking for SSE3 instructions set... yes
#80 512.0 checking whether C compiler accepts -Wformat-nonliteral... no
#80 513.1 checking whether C compiler accepts -mssse3... yes
#80 513.6 checking whether C compiler accepts -Wformat-security... no
#80 513.9 checking for SSSE3 instructions set... no
#80 514.9 checking whether C compiler accepts -msse4.1... yes
#80 515.1 checking whether C compiler accepts -Wformat... no
#80 515.6 checking for SSE4.1 instructions set... yes
#80 516.3 checking whether C compiler accepts -Wstrict-prototypes... no
#80 517.0 checking whether C compiler accepts -mavx... no
#80 517.7 checking for AVX instructions set... yes
#80 517.8 checking whether C compiler accepts -Wshadow... no
#80 518.8 checking whether C compiler accepts -mavx2... yes
#80 519.1 checking whether C compiler accepts -Wnested-externs... no
#80 519.6 checking for AVX2 instructions set... yes
#80 520.3 checking whether C compiler accepts -Wcast-align... no
#80 520.5 checking whether C compiler accepts -mavx512f... no

For a generic -- even arch specific -- docker build, we shouldn't use any intrinsics. They are not guaranteed to be present on all CPUs. There are different kinds of arm64 CPUs and different kinds of amd64 CPUs.

We should NOT compile with -march=native or other flags such as -mavx2

[Raimo33]

@ShahanaFarooqui @cdecker feedback would be appreciated on the last 2 comments

[Raimo33]

An implementation of #8443 would speedup the build quite a bit.

45 minutes to be precise!

#80 1385.3 install -m 644 README.md LICENSE /tmp/lightning_install/share/doc/c-lightning
#80 4122.8 install cli/lightning-cli lightningd/lightningd tools/lightning-hsmtool tools/reckless /tmp/lightning_install/bin

2700 seconds elapsed for the installation of docs and manpages thanks to QEMU slowing down every open/close syscall and docker buildx chunked I/O

[cdecker]

Nice changes, and with the switch to uv things may get even a bit simpler still. Undraft once ready, and we'll review again and merge asap 👍

[Raimo33]

i dont want to undraft until these issues are addressed:

• arm 32 bit compilation fails
• manpages install takes 45 minutes
• COPY . . invalidates all following cache
• user creation
• clarity on [WIP] Refactor Dockerfile #8429 (comment)