YARN-11881. Use hbase-shaded-client-byo-hadoop in hadoop-yarn-server-timelineservice-hbase #8046
+36
−109
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Testing on pseudo-distributed cluster not yet done. |
|
💔 -1 overall
This message was automatically generated. |
steveloughran
approved these changes
Oct 22, 2025
| <dependency> | ||
| <groupId>org.apache.hbase</groupId> | ||
| <artifactId>hbase-common</artifactId> | ||
| <scope>provided</scope> |
There was a problem hiding this comment.
do we need this at all, given the shaded one is there? Is it testing related?
There was a problem hiding this comment.
No, this is a server side component, and needs the hbase-server to compile.
The shaded one is not there, as we're excluding it form the dependencies to avoid conflicts.
|
|
||
| <dependency> | ||
| <groupId>org.apache.hbase</groupId> | ||
| <artifactId>hbase-client</artifactId> |
There was a problem hiding this comment.
same comment -seems superflous.
There was a problem hiding this comment.
Same reply, we don't want to mix the shaded and unshaded jars, so we use the unshaded version of everything.
|
This uses the shaded Hbase client for the hbase client side modules: However, we need the unshaded hbase-server JAR for compiling the Hbase coprocessor in hadoop-yarn-server-timelineservice-hbase-server-2, so we use the unshaded Hbase jars there. For simlipicity's sake (and beacuse I'm not sure we do otherwise) we're still using the unshaded hbase jars for testing, as the protobuf 2.5 dependency is not a problem there. Hbase does provide an hbase-shaded-testing-util package, but I am not sure that works with custom coprocessors, and I didn't attempt using that. Mixing the unshaded hbase jars with the shaded client is a bad idea, so we exclude the shaded client when building the coprocessor and in the tests. |
…timelineservice-hbase Replaces the unshaded maven dpendencies with hbase-shaded-client-byo-hadoop in hadoop-yarn-server-timelineservice-hbase-common and hadoop-yarn-server-timelineservice-hbase-client. Building the HBase coprocessor requires the hbase-server dependency, so we keep using the unshaded dependencies for hadoop-yarn-server-timelineservice-hbase-server-2. We're also using the unshaded HBase dependencies in hadoop-yarn-server-timelineservice-hbase-tests because that does not affect the production classpath, and I'm not sure we can even test coprocessors with hbase-shaded-testing-util.
|
I have added the explanation to the commit message. |
|
Only hadoop-yarn-server-timelineservice-hbase-common, hadoop-yarn-server-timelineservice-hbase-client are included in YARN classpath. They are also only using the HBase client API, so we can use hbase-shaded-client... with them without problems. hadoop-yarn-server-timelineservice-hbase-server-2 only needs to be present on the HBase classpath and it is packaged without dependencies, so using the unshaded dependencies there is not a problem. As discussed in YARN-11882, hadoop-yarn-server-timelineservice-hbase-server-2 should not even be added to the Yarn classpath. It can not work outside the HBase server environment, it is dead code in the Yarn processes. |
|
💔 -1 overall
This message was automatically generated. |
steveloughran
approved these changes
Oct 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Use hbase-shaded-client-byo-hadoop in hadoop-yarn-server-timelineservice-hbase
How was this patch tested?
Test suite and testing on pesudodistributed cluster.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?