KIP-1028: AK 3.8.0 Docker Official Image Assets #16768

KrishVora2912 · 2024-08-01T14:38:16Z

This PR adds the static assets for AK 3.8.0 Docker Official Images.

Post this PR:

Get these changes reviewed from Dockerhub folks.
Modify existing PR with PR template generation script for 3.8.0 under docker official images repo.

Committer Checklist (excluded from commit message)

Verify test coverage and CI build status

whalelines · 2024-08-12T19:01:20Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

+    for server in ha.pool.sks-keyservers.net $(shuf -e \
+                          hkp://p80.pool.sks-keyservers.net:80 \
+                          keyserver.ubuntu.com \
+                          hkp://keyserver.ubuntu.com:80 \
+                          pgp.mit.edu \
+                          hkp://keys.openpgp.org) ; do \
+      gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; \
+    done && \


See the first example under https://github.com/docker-library/official-images?tab=readme-ov-file#image-build

Thanks for the review @whalelines !

I went through the example, and made relevant changes to the Dockerfile:

used GNUPGHOME

Used only 2 keyservers - hkp://keys.openpgp.org and keyserver.ubuntu.com and removed the rest outdated keyservers

Hardcoded the GPG_KEY inside the command itself

Like flink-docker, used the practice of adding gpgconf --kill all as part of verification commands.

wget uses kafka_url env variable, which downloads from a https source

Please let us know if these changes are okay, and if any more are needed.
Thank you again!

github-actions · 2025-01-06T03:39:21Z

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

KrishVora01 · 2025-01-09T11:52:00Z

This PR is to be kept open as Docker Official Image PR is waiting on review from Dockerhub folks.

tianon · 2025-03-11T17:16:16Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

+    gpgconf --kill all; \
+    rm -rf "$GNUPGHOME" kafka.tgz.asc; \
+    mkdir opt/kafka; \
    tar xfz kafka.tgz -C /opt/kafka --strip-components 1; \


It's odd to mix "traditional" flags and Unix/GNU-style flags, and I'd recommend avoiding mixing them (https://manpages.debian.org/bookworm/tar/tar.1.en.html)

tianon · 2025-03-11T17:25:55Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

-# Generate jsa files using dynamic CDS for kafka server start command and kafka storage format command
-RUN /etc/kafka/docker/jsa_launch
+    # Generate jsa files using dynamic CDS for kafka server start command and kafka storage format command
+    /etc/kafka/docker/jsa_launch


I'm concerned about this script; it appears to start a server in the background, does not track the PID of that started server, waits for a file to exist, and then exits, and hopes that the file existing means the server has shut down successfully and cleanly, so there could absolutely be a race here where the file gets created, gets filled up partway and thus exists, the script exits, and the server is killed before it finishes writing the file.

Is this actually necessary? Does it dramatically improve something like performance, startup time, etc? Is it an artifact that could be shipped with the Kafka releases instead?

(This script seems to be the primary justification for the multi-stage build, and I'm sorry but I'm not seeing it. 🙈)

tianon · 2025-03-11T17:29:38Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

+      org.opencontainers.image.description="Apache Kafka" \
+      org.opencontainers.image.created="${build_date}" \
+      org.opencontainers.image.source="https://github.com/apache/kafka" \
      maintainer="Apache Kafka"


See docker-library/official-images#3540, especially docker-library/official-images#3540 (comment):

We don't actively recommend using labels. If an image maintainer wants to have labels, that is fine, but label names should adhere to the image spec: https://github.com/opencontainers/image-spec/blob/v1.0.1/annotations.md

(ie, maintainer is an unacceptable label)

tianon · 2025-03-11T17:32:00Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

 RUN set -eux ; \
-    apk update ; \
-    apk upgrade ; \
    apk add --no-cache wget gcompat gpg gpg-agent procps bash; \


gcompat is a strange inclusion here -- what in this process needs glibc compatibility?

tianon · 2025-03-11T17:35:42Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

-ENV build_date 2024-06-11
+# Get Kafka from https://archive.apache.org/dist/kafka, url passed as env var, for version 3.8.0
+ENV kafka_url https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz
+ENV build_date 2024-08-13


This is not going to be accurate -- this image will be rebuilt any time the base image updates, so this and the associated LABEL should be dropped.

tianon · 2025-03-11T17:38:37Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

 RUN set -eux ; \
-    apk update ; \
-    apk upgrade ; \
    apk add --no-cache wget gcompat gpg gpg-agent procps bash; \


Again, why gcompat? Also, I'd suggest splitting this up so that you can use --virtual and have an easier and more accurate time uninstalling "download-only" dependencies.

tianon · 2025-03-11T17:45:21Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

+    gpgconf --kill all; \
+    rm -rf "$GNUPGHOME" kafka.tgz.asc; \
+    mkdir opt/kafka; \
+    tar xfz kafka.tgz -C /opt/kafka --strip-components 1; \


Using a single stage would pretty dramatically decrease the amount of duplication here.

tianon · 2025-03-11T17:46:36Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

-    apk del wget gpg gpg-agent; \
-    apk cache clean;
+    rm kafka.tgz; \
+    apk del wget gpg gpg-agent; 


As a non-blocking suggestion, is there some simple command that could be run here to verify the installation? One we often use is some-command --version, but I'm not sure if anything in Kafka has something like that?

tianon · 2025-03-11T17:47:38Z

docker/docker_official_images/3.8.0/jvm/Dockerfile


 USER appuser

 VOLUME ["/etc/kafka/secrets", "/var/lib/kafka/data", "/mnt/shared/config"]


Are all of these paths intended to be user-specified / persistent storage? These are all spearately paths that Kafka will write to at runtime in the default configuration and that users will have a bad time if they don't save in some persistent place/way?

tianon · 2025-03-11T17:51:18Z

docker/docker_official_images/3.8.0/jvm/Dockerfile

 VOLUME ["/etc/kafka/secrets", "/var/lib/kafka/data", "/mnt/shared/config"]

-CMD ["/etc/kafka/docker/run"]
+CMD ["/etc/kafka/docker/run"]


I'm definitely still struggling to understand the necessity of so much Docker-image-custom behavior -- what problem is being solved here that's unique to running Kafka inside a container? I can't think of very many things that would be useful for containers that wouldn't also make the life of users in things like high-constrained systemd units better, such that any logic here should probably live in proper upstream scripts / code instead.

github-actions · 2025-06-10T03:46:52Z

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

github-actions · 2025-07-10T03:48:36Z

This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.

3.8.0 DOI assets

b473ae0

whalelines reviewed Aug 12, 2024

View reviewed changes

New 3.8.0 changed Dockerfile

64bebc7

github-actions bot added the stale Stale PRs label Jan 6, 2025

github-actions bot removed the stale Stale PRs label Jan 10, 2025

tianon reviewed Mar 11, 2025

View reviewed changes

github-actions bot added the stale Stale PRs label Jun 10, 2025

github-actions bot added the closed-stale PRs that were closed due to inactivity label Jul 10, 2025

github-actions bot closed this Jul 10, 2025


		USER appuser

		VOLUME ["/etc/kafka/secrets", "/var/lib/kafka/data", "/mnt/shared/config"]

KIP-1028: AK 3.8.0 Docker Official Image Assets #16768

KIP-1028: AK 3.8.0 Docker Official Image Assets #16768

Uh oh!

Conversation

KrishVora2912 commented Aug 1, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KrishVora2912 Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 6, 2025

Uh oh!

KrishVora01 commented Jan 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 10, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

Uh oh!

KrishVora2912 Aug 13, 2024 •

edited

Loading