HADOOP-19343: Add native support for GCS connector

[cnauroth]

Description of PR

Add native support for GCS connector

How was this patch tested?

The new module contains integration tests that we've run against a live GCS bucket.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 21 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	10m 23s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 13s		trunk passed
+1 💚	compile	8m 20s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	7m 31s		trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	checkstyle	1m 59s		trunk passed
+1 💚	mvnsite	1m 26s		trunk passed
+1 💚	javadoc	2m 10s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 50s		trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+0 🆗	spotbugs	0m 32s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	24m 11s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	24m 27s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	2m 53s		Maven dependency ordering for patch
+1 💚	mvninstall	11m 44s		the patch passed
+1 💚	compile	8m 5s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	8m 5s		the patch passed
+1 💚	compile	7m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	javac	7m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 59s		the patch passed
+1 💚	mvnsite	1m 50s		the patch passed
+1 💚	javadoc	2m 34s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+0 🆗	spotbugs	0m 27s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	24m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 25s		hadoop-project in the patch passed.
+1 💚	unit	0m 31s		hadoop-gcp in the patch passed.
+1 💚	unit	71m 26s		hadoop-tools in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		221m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7869/1/artifact/out/Dockerfile
GITHUB PR	#7869
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint
uname	Linux c884f9f4e391 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / faff8a4
Default Java	Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7869/1/testReport/
Max. process+thread count	1028 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-gcp hadoop-tools U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7869/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[slfan1989]

We need to remove the JUnit4 dependency.

[cnauroth]

Great point! I sent up #7872 for this.

[slfan1989]

If we update the version of the JAR package, the LICENSE-binary file should also be updated accordingly.

[pan3793]

Searching import com.google.common returns no results, just wondering how upgrading Guava is related to this PR?

No offense, but given that there were many painful experiences with Guava for Hadoop ecosystem projects, I think we'd better be careful to introduce new components that hardly depend on Guava, especially one that requires a specific version of Guava.

[cnauroth]

Guava is a dependency of the GCS SDK. Without this change, there is a dependency convergence problem:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.5.0:enforce (depcheck) on project hadoop-gcp: 
[ERROR] Rule 0: org.apache.maven.enforcer.rules.dependency.DependencyConvergence failed with message:
[ERROR] Failed while enforcing releasability.
[ERROR] 
[ERROR] Dependency convergence error for org.codehaus.mojo:animal-sniffer-annotations:jar:1.17 paths to dependency are:
[ERROR] +-org.apache.hadoop:hadoop-gcp:jar:3.5.0-SNAPSHOT
[ERROR]   +-com.google.cloud:google-cloud-storage:jar:2.52.0:compile
[ERROR]     +-com.google.guava:guava:jar:27.0-jre:compile
[ERROR]       +-org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[ERROR] and
[ERROR] +-org.apache.hadoop:hadoop-gcp:jar:3.5.0-SNAPSHOT
[ERROR]   +-com.google.cloud:google-cloud-storage:jar:2.52.0:compile
[ERROR]     +-org.codehaus.mojo:animal-sniffer-annotations:jar:1.24:compile

However, we don't necessarily need to upgrade it project-wide. I sent up #7883 to revert this change in hadoop-project/pom.xml and mention the versioning needs entirely within hadoop-gcp/pom.xml.