Skip to content

HADOOP-19343: Add native support for GCS connector #7869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: trunk
Choose a base branch
from
+9,741 −1
Open

HADOOP-19343: Add native support for GCS connector #7869

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f94c3f6
HADOOP-19343: GCS / Add implementation for create() API
arunkumarchacko May 27, 2025
0b104fd
HADOOP-19343: Add support for mkdir() and getFileStatus()
arunkumarchacko Jun 10, 2025
115a0e3
HADOOP-19343: Add instruction for running tests
arunkumarchacko Jun 16, 2025
4e18572
HADOOP-19343: Add support for open() and rename()
arunkumarchacko Jun 18, 2025
48b3198
HADOOP-19343. Add support for hflush()
arunkumarchacko Jun 25, 2025
cdf1c38
HADOOP-19343. Add support for append(), compose(), concat()
arunkumarchacko Jul 2, 2025
d3892f2
HADOOP-19343. Add additional authentication support
arunkumarchacko Jul 8, 2025
faff8a4
HADOOP-19343. Add core-site.xml for testing
arunkumarchacko Jul 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion hadoop-project/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@
<findbugs.version>3.0.5</findbugs.version>
<dnsjava.version>3.6.1</dnsjava.version>

<guava.version>27.0-jre</guava.version>
<guava.version>33.1.0-jre</guava.version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we update the version of the JAR package, the LICENSE-binary file should also be updated accordingly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Searching import com.google.common returns no results, just wondering how upgrading Guava is related to this PR?

No offense, but given that there were many painful experiences with Guava for Hadoop ecosystem projects, I think we'd better be careful to introduce new components that hardly depend on Guava, especially one that requires a specific version of Guava.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guava is a dependency of the GCS SDK. Without this change, there is a dependency convergence problem:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.5.0:enforce (depcheck) on project hadoop-gcp: 
[ERROR] Rule 0: org.apache.maven.enforcer.rules.dependency.DependencyConvergence failed with message:
[ERROR] Failed while enforcing releasability.
[ERROR] 
[ERROR] Dependency convergence error for org.codehaus.mojo:animal-sniffer-annotations:jar:1.17 paths to dependency are:
[ERROR] +-org.apache.hadoop:hadoop-gcp:jar:3.5.0-SNAPSHOT
[ERROR]   +-com.google.cloud:google-cloud-storage:jar:2.52.0:compile
[ERROR]     +-com.google.guava:guava:jar:27.0-jre:compile
[ERROR]       +-org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[ERROR] and
[ERROR] +-org.apache.hadoop:hadoop-gcp:jar:3.5.0-SNAPSHOT
[ERROR]   +-com.google.cloud:google-cloud-storage:jar:2.52.0:compile
[ERROR]     +-org.codehaus.mojo:animal-sniffer-annotations:jar:1.24:compile

However, we don't necessarily need to upgrade it project-wide. I sent up #7883 to revert this change in hadoop-project/pom.xml and mention the versioning needs entirely within hadoop-gcp/pom.xml.

<guice.version>5.1.0</guice.version>

<bouncycastle.version>1.78.1</bouncycastle.version>
Expand Down Expand Up @@ -2157,6 +2157,11 @@
<artifactId>failsafe</artifactId>
<version>2.4.4</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<version>2.52.0</version>
</dependency>
</dependencies>
</dependencyManagement>

Expand Down
34 changes: 34 additions & 0 deletions hadoop-tools/hadoop-gcp/dev-support/findbugs-exclude.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<FindBugsFilter>
<!-- Redundant null check makes code clearer, future-proof here. -->
<Match>
<Class name="org.apache.hadoop.fs.gs.GoogleCloudStorage" />
<Method name="createItemInfoForBlob" />
<Bug pattern="RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE" />
</Match>
<Match>
<Class name="org.apache.hadoop.fs.gs.GoogleCloudStorage" />
<Method name="createItemInfoForBlob" />
<Bug pattern="NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE" />
</Match>
<Match>
<Class name="org.apache.hadoop.fs.gs.GoogleCloudStorageExceptions" />
<Method name="createCompositeException" />
<Bug pattern="NP_NULL_ON_SOME_PATH" />
</Match>
</FindBugsFilter>
Loading