HADOOP-19343: Support -DskipShade in hadoop-gcp for faster test builds. #7874

cnauroth · 2025-08-15T16:29:02Z

Description of PR

Support -DskipShade in hadoop-gcp for faster test builds. This aligns with other sub-modules that support the same flag to skip shading.

How was this patch tested?

I ran local mvn clean package with and without -DskipShade to verify that it shades by default but skips shading with the flag.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2025-08-15T18:50:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ HADOOP-19343 Compile Tests _
+1 💚	mvninstall	51m 35s		HADOOP-19343 passed
+1 💚	compile	0m 30s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 28s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	0m 33s		HADOOP-19343 passed
+1 💚	javadoc	0m 35s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 29s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	89m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	10m 52s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 20s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	36m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 28s		hadoop-gcp in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		140m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/1/artifact/out/Dockerfile
GITHUB PR	#7874
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 897b20288b97 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HADOOP-19343 / `d66349f`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/1/testReport/
Max. process+thread count	721 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-gcp U: hadoop-tools/hadoop-gcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Closes apache#7874

hadoop-yetus · 2025-08-18T00:55:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	14m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ HADOOP-19343 Compile Tests _
+1 💚	mvninstall	48m 45s		HADOOP-19343 passed
+1 💚	compile	0m 29s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 26s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	0m 33s		HADOOP-19343 passed
+1 💚	javadoc	0m 35s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 29s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	86m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	10m 16s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 20s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	36m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 27s		hadoop-gcp in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		151m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/2/artifact/out/Dockerfile
GITHUB PR	#7874
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 66b79f24c87d 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HADOOP-19343 / `978ebba`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/2/testReport/
Max. process+thread count	553 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-gcp U: hadoop-tools/hadoop-gcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Closes apache#7874 Reviewed-by: Arunkumar Chacko <[email protected]> Reviewed-by: Cheng Pan <[email protected]>

hadoop-yetus · 2025-08-22T01:03:40Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ HADOOP-19343 Compile Tests _
+1 💚	mvninstall	49m 56s		HADOOP-19343 passed
+1 💚	compile	0m 28s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 28s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	0m 33s		HADOOP-19343 passed
+1 💚	javadoc	0m 34s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 28s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	87m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	10m 10s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	javadoc	0m 20s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	35m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 27s		hadoop-gcp in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		137m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/3/artifact/out/Dockerfile
GITHUB PR	#7874
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 5b5cf0ae8bf6 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HADOOP-19343 / `0767325`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/3/testReport/
Max. process+thread count	642 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-gcp U: hadoop-tools/hadoop-gcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Closes apache#7874 Reviewed-by: Arunkumar Chacko <[email protected]> Reviewed-by: Cheng Pan <[email protected]>

steveloughran

+1

Closes #7874 Signed-off-by: Steve Loughran <[email protected]> Reviewed-by: Arunkumar Chacko <[email protected]> Reviewed-by: Cheng Pan <[email protected]>

cnauroth · 2025-08-25T22:52:20Z

@steveloughran , @arunkumarchacko and @pan3793 , thank you for the reviews. I committed this to the feature branch.

hadoop-yetus · 2025-08-25T23:49:54Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ HADOOP-19343 Compile Tests _
-1 ❌	mvninstall	49m 51s	/branch-mvninstall-root.txt	root in HADOOP-19343 failed.
+1 💚	compile	0m 30s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 28s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	0m 32s		HADOOP-19343 passed
+1 💚	javadoc	0m 34s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 27s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	87m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	10m 48s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 21s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	36m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 28s		hadoop-gcp in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		138m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/4/artifact/out/Dockerfile
GITHUB PR	#7874
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 258b5c6a480f 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HADOOP-19343 / `b520212`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/4/testReport/
Max. process+thread count	628 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-gcp U: hadoop-tools/hadoop-gcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7874/4/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Closes apache#7874 Signed-off-by: Steve Loughran <[email protected]> Reviewed-by: Arunkumar Chacko <[email protected]> Reviewed-by: Cheng Pan <[email protected]>

github-actions bot added build trunk TOOLS labels Aug 15, 2025

cnauroth force-pushed the HADOOP-19343-skip-shade branch from d66349f to 978ebba Compare August 17, 2025 22:22

cnauroth added a commit to cnauroth/hadoop that referenced this pull request Aug 17, 2025

HADOOP-19343: Support -DskipShade in hadoop-gcp for faster test builds.

978ebba

Closes apache#7874

cnauroth mentioned this pull request Aug 17, 2025

HADOOP-19343: Include hadoop-gcp in distro. #7877

Closed

4 tasks

arunkumarchacko approved these changes Aug 19, 2025

View reviewed changes

pan3793 approved these changes Aug 19, 2025

View reviewed changes

cnauroth added a commit to cnauroth/hadoop that referenced this pull request Aug 21, 2025

HADOOP-19343: Support -DskipShade in hadoop-gcp for faster test builds.

0767325

Closes apache#7874 Reviewed-by: Arunkumar Chacko <[email protected]> Reviewed-by: Cheng Pan <[email protected]>

cnauroth force-pushed the HADOOP-19343-skip-shade branch from 978ebba to 0767325 Compare August 21, 2025 22:44

cnauroth requested a review from slfan1989 August 25, 2025 15:34

cnauroth force-pushed the HADOOP-19343 branch from 4b62b1d to 71c1be5 Compare August 25, 2025 21:20

HADOOP-19343: Support -DskipShade in hadoop-gcp for faster test builds.

b520212

Closes apache#7874 Reviewed-by: Arunkumar Chacko <[email protected]> Reviewed-by: Cheng Pan <[email protected]>

cnauroth force-pushed the HADOOP-19343-skip-shade branch from 0767325 to b520212 Compare August 25, 2025 21:29

steveloughran approved these changes Aug 25, 2025

View reviewed changes

cnauroth closed this Aug 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19343: Support -DskipShade in hadoop-gcp for faster test builds. #7874

HADOOP-19343: Support -DskipShade in hadoop-gcp for faster test builds. #7874

cnauroth commented Aug 15, 2025

Uh oh!

hadoop-yetus commented Aug 15, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 22, 2025

Uh oh!

steveloughran left a comment

Uh oh!

cnauroth commented Aug 25, 2025

Uh oh!

hadoop-yetus commented Aug 25, 2025

Uh oh!

Uh oh!

HADOOP-19343: Support -DskipShade in hadoop-gcp for faster test builds. #7874

HADOOP-19343: Support -DskipShade in hadoop-gcp for faster test builds. #7874

Conversation

cnauroth commented Aug 15, 2025

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Aug 15, 2025

Uh oh!

hadoop-yetus commented Aug 18, 2025

Uh oh!

hadoop-yetus commented Aug 22, 2025

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

cnauroth commented Aug 25, 2025

Uh oh!

hadoop-yetus commented Aug 25, 2025

Uh oh!

Uh oh!