Skip to content

Commit c73fd1b

Browse files
dervoetidervoeti
committed
Merge branch 'main' of https://github.com/stackabletech/docker-images into feat/operator-source-code
2 parents 5b59a4b + f8e41f1 commit c73fd1b

12 files changed

+446
-3
lines changed

‎CHANGELOG.md

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.
44

55
## [Unreleased]
66

7+
### Changed
8+
9+
- all: Use our build-repo to cache NPM dependencies ([#1219])
10+
11+
[#1219]: https://github.com/stackabletech/docker-images/pull/1219
12+
13+
## [25.7.0] - 2025-07-23
14+
15+
## [25.7.0-rc1] - 2025-07-18
16+
717
### Added
818

919
- airflow: check for correct permissions and ownerships in /stackable folder via
@@ -60,6 +70,8 @@ All notable changes to this project will be documented in this file.
6070
- zookeeper: bump netty version for CVE-2025-24970 in 3.9.3 ([#1180])
6171
- hadoop: backport HADOOP-19352, HADOOP-19335, HADOOP-19465, HADOOP-19456 and HADOOP-19225 to fix vulnerabilities in Hadoop `3.4.1` ([#1184])
6272
- Include `.tar.gz` snapshots of the operator source code in container images ([#1207])
73+
- hadoop: Backport HADOOP-18583 to make OpenSSL 3.x work with the native hadoop libraries ([#1209]).
74+
- spark: backport [SPARK-51311] Promote bcprov-jdk18on to compile scope ([#1212]).
6375

6476
### Changed
6577

@@ -223,14 +235,16 @@ All notable changes to this project will be documented in this file.
223235
[#1189]: https://github.com/stackabletech/docker-images/pull/1189
224236
[#1197]: https://github.com/stackabletech/docker-images/pull/1197
225237
[#1207]: https://github.com/stackabletech/docker-images/pull/1207
238+
[#1209]: https://github.com/stackabletech/docker-images/pull/1209
239+
[#1212]: https://github.com/stackabletech/docker-images/pull/1212
226240

227241
## [25.3.0] - 2025-03-21
228242

229243
### Added
230244

231245
- omid: Added 1.1.3-SNAPSHOT to allow for easier scanning pre-release
232246
- airflow: Add OPA support to Airflow ([#978]).
233-
- nifi: Activate `include-hadoop` profile for NiFi version 2.* ([#958]).
247+
- nifi: Activate `include-hadoop` profile for NiFi version 2 ([#958]).
234248
- nifi: Add NiFi hadoop Azure and GCP libraries ([#943]).
235249
- superset: Add role mapping from OPA ([#979]).
236250
- base: Add containerdebug tool ([#928], [#959]).
@@ -763,7 +777,7 @@ All notable changes to this project will be documented in this file.
763777
- BREAKING: Use RPM instead of tar.gz for Vector. Because of that, the
764778
location of the Vector executable changed, and the operator-rs version
765779
0.45.0 or newer is required ([#429]).
766-
- spark-k8s: Rework spark images to build on top of java-base image. This fixes the missing tzdata-java package in 0.0.0-dev versions ([#434]).
780+
- spark-k8s: Rework spark images to build on top of java-base image. This fixes the missing tzdata-java package in 0.0.0-dev versions ([#434]).
767781

768782
- airflow: Updated git-sync to 3.6.8 ([#431]).
769783
- airflow: Updated statsd-exporter to 0.24, this was accidentally moved to a very old version previously (0.3.0) ([#431]).

‎hadoop/Dockerfile

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,15 @@ ln -s "/stackable/jmx/jmx_prometheus_javaagent-${JMX_EXPORTER}.jar" /stackable/j
137137
chown --recursive ${STACKABLE_USER_UID}:0 /stackable/hadoop /stackable/jmx /stackable/async-profiler "/stackable/async-profiler-${ASYNC_PROFILER}-${TARGETOS}-${ARCH}"
138138
chmod --recursive g=u /stackable/jmx /stackable/async-profiler "/stackable/hadoop-${HADOOP_VERSION}-stackable${RELEASE}"
139139

140+
# Workaround for https://issues.apache.org/jira/browse/HADOOP-12845
141+
# The problem is that our stackable-devel image does contain the openssl-devel package
142+
# That package creates a symlink from /usr/lib/libcrypto.so to the real libcrypto
143+
# The non -devel package, which is used in this image, does NOT create this symlink.
144+
# That's why the Hadoop build works even with the 'require.openssl' flag but in the production
145+
# image the 'hadoop checknative' tool still fails because it can't find the 'libcrypto.so' symlink.
146+
# Therefore we create this symlink here.
147+
ln -s /usr/lib64/libcrypto.so.3 /usr/lib64/libcrypto.so
148+
140149
# ----------------------------------------
141150
# Checks
142151
# This section is to run final checks to ensure the created final images

‎hadoop/hadoop/Dockerfile

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,12 @@ sed -e '/<artifactId>hadoop-pipes<\/artifactId>/,/<\/dependency>/ { s/<version>.
6969
# Create snapshot of the source code including custom patches
7070
tar -czf /stackable/hadoop-${NEW_VERSION}-src.tar.gz .
7171

72+
# We do not pass require.snappy because that is only built in to the MapReduce client and we don't need that
73+
#
74+
# Passing require.openssl SHOULD make the build fail if OpenSSL is not present.
75+
# This does not work properly however because this builder image contains the openssl-devel package which creates a symlink from /usr/lib64/libcrypto.so to the real version.
76+
# Therefore, this build does work but the final image does NOT contain the openssl-devel package which is why it fails there which is why we have to create the symlink over there manually.
77+
# We still leave this flag in to automatically fail should anything with the packages or symlinks ever fail.
7278
mvn \
7379
--batch-mode \
7480
--no-transfer-progress \
@@ -77,6 +83,7 @@ mvn \
7783
-pl '!hadoop-tools/hadoop-pipes' \
7884
-Dhadoop.version=${NEW_VERSION} \
7985
-Drequire.fuse=true \
86+
-Drequire.openssl=true \
8087
-DskipTests \
8188
-Dmaven.javadoc.skip=true
8289

‎hadoop/hadoop/stackable/patches/3.3.6/0012-HADOOP-18583.-Fix-loading-of-OpenSSL-3.x-symbols-525.patch

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
From baa7ec826f3f6d044f5307efe4b5d3bdd111bf4e Mon Sep 17 00:00:00 2001
2+
From: Sebastian Klemke <[email protected]>
3+
Date: Thu, 7 Nov 2024 19:14:13 +0100
4+
Subject: HADOOP-18583. Fix loading of OpenSSL 3.x symbols (#5256) (#7149)
5+
6+
Contributed by Sebastian Klemke
7+
---
8+
.../org/apache/hadoop/crypto/OpensslCipher.c | 68 +++++++++++++++++--
9+
1 file changed, 64 insertions(+), 4 deletions(-)
10+
11+
diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
12+
index abff7ea5f1..f17169dec2 100644
13+
--- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
14+
+++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
15+
@@ -24,6 +24,57 @@
16+
17+
#include "org_apache_hadoop_crypto_OpensslCipher.h"
18+
19+
+/*
20+
+ # OpenSSL ABI Symbols
21+
+
22+
+ Available on all OpenSSL versions:
23+
+
24+
+ | Function | 1.0 | 1.1 | 3.0 |
25+
+ |--------------------------------|-----|-----|-----|
26+
+ | EVP_CIPHER_CTX_new | YES | YES | YES |
27+
+ | EVP_CIPHER_CTX_free | YES | YES | YES |
28+
+ | EVP_CIPHER_CTX_set_padding | YES | YES | YES |
29+
+ | EVP_CIPHER_CTX_test_flags | YES | YES | YES |
30+
+ | EVP_CipherInit_ex | YES | YES | YES |
31+
+ | EVP_CipherUpdate | YES | YES | YES |
32+
+ | EVP_CipherFinal_ex | YES | YES | YES |
33+
+ | ENGINE_by_id | YES | YES | YES |
34+
+ | ENGINE_free | YES | YES | YES |
35+
+ | EVP_aes_256_ctr | YES | YES | YES |
36+
+ | EVP_aes_128_ctr | YES | YES | YES |
37+
+
38+
+ Available on old versions:
39+
+
40+
+ | Function | 1.0 | 1.1 | 3.0 |
41+
+ |--------------------------------|-----|-----|-----|
42+
+ | EVP_CIPHER_CTX_cleanup | YES | --- | --- |
43+
+ | EVP_CIPHER_CTX_init | YES | --- | --- |
44+
+ | EVP_CIPHER_CTX_block_size | YES | YES | --- |
45+
+ | EVP_CIPHER_CTX_encrypting | --- | YES | --- |
46+
+
47+
+ Available on new versions:
48+
+
49+
+ | Function | 1.0 | 1.1 | 3.0 |
50+
+ |--------------------------------|-----|-----|-----|
51+
+ | OPENSSL_init_crypto | --- | YES | YES |
52+
+ | EVP_CIPHER_CTX_reset | --- | YES | YES |
53+
+ | EVP_CIPHER_CTX_get_block_size | --- | --- | YES |
54+
+ | EVP_CIPHER_CTX_is_encrypting | --- | --- | YES |
55+
+
56+
+ Optionally available on new versions:
57+
+
58+
+ | Function | 1.0 | 1.1 | 3.0 |
59+
+ |--------------------------------|-----|-----|-----|
60+
+ | EVP_sm4_ctr | --- | opt | opt |
61+
+
62+
+ Name changes:
63+
+
64+
+ | < 3.0 name | >= 3.0 name |
65+
+ |----------------------------|--------------------------------|
66+
+ | EVP_CIPHER_CTX_block_size | EVP_CIPHER_CTX_get_block_size |
67+
+ | EVP_CIPHER_CTX_encrypting | EVP_CIPHER_CTX_is_encrypting |
68+
+ */
69+
+
70+
#ifdef UNIX
71+
static EVP_CIPHER_CTX * (*dlsym_EVP_CIPHER_CTX_new)(void);
72+
static void (*dlsym_EVP_CIPHER_CTX_free)(EVP_CIPHER_CTX *);
73+
@@ -87,6 +138,15 @@ static __dlsym_EVP_aes_128_ctr dlsym_EVP_aes_128_ctr;
74+
static HMODULE openssl;
75+
#endif
76+
77+
+// names changed in OpenSSL 3 ABI - see History section in EVP_EncryptInit(3)
78+
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
79+
+#define CIPHER_CTX_BLOCK_SIZE "EVP_CIPHER_CTX_get_block_size"
80+
+#define CIPHER_CTX_ENCRYPTING "EVP_CIPHER_CTX_is_encrypting"
81+
+#else
82+
+#define CIPHER_CTX_BLOCK_SIZE "EVP_CIPHER_CTX_block_size"
83+
+#define CIPHER_CTX_ENCRYPTING "EVP_CIPHER_CTX_encrypting"
84+
+#endif /* OPENSSL_VERSION_NUMBER >= 0x30000000L */
85+
+
86+
static void loadAesCtr(JNIEnv *env)
87+
{
88+
#ifdef UNIX
89+
@@ -142,10 +202,10 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs
90+
LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_test_flags, env, openssl, \
91+
"EVP_CIPHER_CTX_test_flags");
92+
LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_block_size, env, openssl, \
93+
- "EVP_CIPHER_CTX_block_size");
94+
+ CIPHER_CTX_BLOCK_SIZE);
95+
#if OPENSSL_VERSION_NUMBER >= 0x10100000L
96+
LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_encrypting, env, openssl, \
97+
- "EVP_CIPHER_CTX_encrypting");
98+
+ CIPHER_CTX_ENCRYPTING);
99+
#endif
100+
LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CipherInit_ex, env, openssl, \
101+
"EVP_CipherInit_ex");
102+
@@ -173,11 +233,11 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs
103+
openssl, "EVP_CIPHER_CTX_test_flags");
104+
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_block_size, \
105+
dlsym_EVP_CIPHER_CTX_block_size, env, \
106+
- openssl, "EVP_CIPHER_CTX_block_size");
107+
+ openssl, CIPHER_CTX_BLOCK_SIZE);
108+
#if OPENSSL_VERSION_NUMBER >= 0x10100000L
109+
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_encrypting, \
110+
dlsym_EVP_CIPHER_CTX_encrypting, env, \
111+
- openssl, "EVP_CIPHER_CTX_encrypting");
112+
+ openssl, CIPHER_CTX_ENCRYPTING);
113+
#endif
114+
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CipherInit_ex, dlsym_EVP_CipherInit_ex, \
115+
env, openssl, "EVP_CipherInit_ex");

‎hadoop/hadoop/stackable/patches/3.4.1/0011-HADOOP-18583.-Fix-loading-of-OpenSSL-3.x-symbols-525.patch

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
From cd1c23ea5bddd2796caf2590fef467e488c3bcbf Mon Sep 17 00:00:00 2001
2+
From: Sebastian Klemke <[email protected]>
3+
Date: Thu, 7 Nov 2024 19:14:13 +0100
4+
Subject: HADOOP-18583. Fix loading of OpenSSL 3.x symbols (#5256) (#7149)
5+
6+
Contributed by Sebastian Klemke
7+
---
8+
.../org/apache/hadoop/crypto/OpensslCipher.c | 68 +++++++++++++++++--
9+
1 file changed, 64 insertions(+), 4 deletions(-)
10+
11+
diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
12+
index 976bf135ce..33be4a394f 100644
13+
--- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
14+
+++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
15+
@@ -24,6 +24,57 @@
16+
17+
#include "org_apache_hadoop_crypto_OpensslCipher.h"
18+
19+
+/*
20+
+ # OpenSSL ABI Symbols
21+
+
22+
+ Available on all OpenSSL versions:
23+
+
24+
+ | Function | 1.0 | 1.1 | 3.0 |
25+
+ |--------------------------------|-----|-----|-----|
26+
+ | EVP_CIPHER_CTX_new | YES | YES | YES |
27+
+ | EVP_CIPHER_CTX_free | YES | YES | YES |
28+
+ | EVP_CIPHER_CTX_set_padding | YES | YES | YES |
29+
+ | EVP_CIPHER_CTX_test_flags | YES | YES | YES |
30+
+ | EVP_CipherInit_ex | YES | YES | YES |
31+
+ | EVP_CipherUpdate | YES | YES | YES |
32+
+ | EVP_CipherFinal_ex | YES | YES | YES |
33+
+ | ENGINE_by_id | YES | YES | YES |
34+
+ | ENGINE_free | YES | YES | YES |
35+
+ | EVP_aes_256_ctr | YES | YES | YES |
36+
+ | EVP_aes_128_ctr | YES | YES | YES |
37+
+
38+
+ Available on old versions:
39+
+
40+
+ | Function | 1.0 | 1.1 | 3.0 |
41+
+ |--------------------------------|-----|-----|-----|
42+
+ | EVP_CIPHER_CTX_cleanup | YES | --- | --- |
43+
+ | EVP_CIPHER_CTX_init | YES | --- | --- |
44+
+ | EVP_CIPHER_CTX_block_size | YES | YES | --- |
45+
+ | EVP_CIPHER_CTX_encrypting | --- | YES | --- |
46+
+
47+
+ Available on new versions:
48+
+
49+
+ | Function | 1.0 | 1.1 | 3.0 |
50+
+ |--------------------------------|-----|-----|-----|
51+
+ | OPENSSL_init_crypto | --- | YES | YES |
52+
+ | EVP_CIPHER_CTX_reset | --- | YES | YES |
53+
+ | EVP_CIPHER_CTX_get_block_size | --- | --- | YES |
54+
+ | EVP_CIPHER_CTX_is_encrypting | --- | --- | YES |
55+
+
56+
+ Optionally available on new versions:
57+
+
58+
+ | Function | 1.0 | 1.1 | 3.0 |
59+
+ |--------------------------------|-----|-----|-----|
60+
+ | EVP_sm4_ctr | --- | opt | opt |
61+
+
62+
+ Name changes:
63+
+
64+
+ | < 3.0 name | >= 3.0 name |
65+
+ |----------------------------|--------------------------------|
66+
+ | EVP_CIPHER_CTX_block_size | EVP_CIPHER_CTX_get_block_size |
67+
+ | EVP_CIPHER_CTX_encrypting | EVP_CIPHER_CTX_is_encrypting |
68+
+ */
69+
+
70+
#ifdef UNIX
71+
static EVP_CIPHER_CTX * (*dlsym_EVP_CIPHER_CTX_new)(void);
72+
static void (*dlsym_EVP_CIPHER_CTX_free)(EVP_CIPHER_CTX *);
73+
@@ -106,6 +157,15 @@ static __dlsym_ENGINE_free dlsym_ENGINE_free;
74+
static HMODULE openssl;
75+
#endif
76+
77+
+// names changed in OpenSSL 3 ABI - see History section in EVP_EncryptInit(3)
78+
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
79+
+#define CIPHER_CTX_BLOCK_SIZE "EVP_CIPHER_CTX_get_block_size"
80+
+#define CIPHER_CTX_ENCRYPTING "EVP_CIPHER_CTX_is_encrypting"
81+
+#else
82+
+#define CIPHER_CTX_BLOCK_SIZE "EVP_CIPHER_CTX_block_size"
83+
+#define CIPHER_CTX_ENCRYPTING "EVP_CIPHER_CTX_encrypting"
84+
+#endif /* OPENSSL_VERSION_NUMBER >= 0x30000000L */
85+
+
86+
static void loadAesCtr(JNIEnv *env)
87+
{
88+
#ifdef UNIX
89+
@@ -170,10 +230,10 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs
90+
LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_test_flags, env, openssl, \
91+
"EVP_CIPHER_CTX_test_flags");
92+
LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_block_size, env, openssl, \
93+
- "EVP_CIPHER_CTX_block_size");
94+
+ CIPHER_CTX_BLOCK_SIZE);
95+
#if OPENSSL_VERSION_NUMBER >= 0x10100000L
96+
LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CIPHER_CTX_encrypting, env, openssl, \
97+
- "EVP_CIPHER_CTX_encrypting");
98+
+ CIPHER_CTX_ENCRYPTING);
99+
#endif
100+
LOAD_DYNAMIC_SYMBOL(dlsym_EVP_CipherInit_ex, env, openssl, \
101+
"EVP_CipherInit_ex");
102+
@@ -209,11 +269,11 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_crypto_OpensslCipher_initIDs
103+
openssl, "EVP_CIPHER_CTX_test_flags");
104+
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_block_size, \
105+
dlsym_EVP_CIPHER_CTX_block_size, env, \
106+
- openssl, "EVP_CIPHER_CTX_block_size");
107+
+ openssl, CIPHER_CTX_BLOCK_SIZE);
108+
#if OPENSSL_VERSION_NUMBER >= 0x10100000L
109+
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CIPHER_CTX_encrypting, \
110+
dlsym_EVP_CIPHER_CTX_encrypting, env, \
111+
- openssl, "EVP_CIPHER_CTX_encrypting");
112+
+ openssl, CIPHER_CTX_ENCRYPTING);
113+
#endif
114+
LOAD_DYNAMIC_SYMBOL(__dlsym_EVP_CipherInit_ex, dlsym_EVP_CipherInit_ex, \
115+
env, openssl, "EVP_CipherInit_ex");

‎kafka/Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ find . -type f -print0 | xargs -0 sed -i "s/\-stackable0\.0\.0\-dev/-stackable${
3131
tar -czf /stackable/kafka-${NEW_VERSION}-src.tar.gz .
3232

3333
# TODO: Try to install gradle via package manager (if possible) instead of fetching it from the internet
34+
# We patch Kafka to use our Nexus build repo instead
3435
# We don't specify "-x test" to skip the tests, as we might bump some Kafka internal dependencies in the future and
3536
# it's a good idea to run the tests in this case.
3637
./gradlew clean releaseTarGz

0 commit comments

Comments
 (0)