Skip to content

[BUILD] Upgrade RoaringBitmap to 1.5.2 #49710

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: master
Choose a base branch
from
Draft

[BUILD] Upgrade RoaringBitmap to 1.5.2 #49710

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d4b1054
init
LuciferYang Jan 28, 2025
cfa126b
add bench result
LuciferYang Jan 28, 2025
34d60a9
init
LuciferYang Jan 28, 2025
71b7d39
test java 21
LuciferYang Jan 28, 2025
6e3b4be
Merge branch 'apache:master' into r-map-1.3.19
LuciferYang Jan 31, 2025
08b1495
Merge branch 'upmaster' into r-map-1.3.19
LuciferYang Apr 24, 2025
82836f9
up
LuciferYang Apr 24, 2025
403bda0
Merge branch 'apache:master' into r-map-1.3.19
LuciferYang Jun 24, 2025
2a24a34
upgrade to 1.4.1
LuciferYang Jun 24, 2025
3728563
1.5.2
LuciferYang Aug 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ on:
java:
required: false
type: string
default: 17
default: 21
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will revert after test

branch:
description: Branch to run the build against
required: false
Expand Down
2 changes: 1 addition & 1 deletion LICENSE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,7 @@ org.json4s:json4s-jackson_2.13
org.json4s:json4s-scalap_2.13
org.lz4:lz4-java
org.objenesis:objenesis
org.roaringbitmap:RoaringBitmap
com.github.RoaringBitmap.RoaringBitmap:roaringbitmap
org.rocksdb:rocksdbjni
org.scala-lang:scala-compiler
org.scala-lang:scala-library
Expand Down
4 changes: 2 additions & 2 deletions common/network-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,8 @@
<artifactId>tink</artifactId>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, interesting.

<artifactId>roaringbitmap</artifactId>
</dependency>

<!-- Test dependencies -->
Expand Down
4 changes: 2 additions & 2 deletions common/network-shuffle/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
<artifactId>roaringbitmap</artifactId>
</dependency>

<!-- Test dependencies -->
Expand Down
4 changes: 2 additions & 2 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,8 +244,8 @@
<artifactId>zstd-jni</artifactId>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
<artifactId>roaringbitmap</artifactId>
</dependency>
<dependency>
<groupId>org.scala-lang.modules</groupId>
Expand Down
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
HikariCP/2.5.1//HikariCP-2.5.1.jar
JLargeArrays/1.5//JLargeArrays-1.5.jar
JTransforms/3.1//JTransforms-3.1.jar
RoaringBitmap/1.3.0//RoaringBitmap-1.3.0.jar
ST4/4.0.4//ST4-4.0.4.jar
aircompressor/2.0.2//aircompressor-2.0.2.jar
algebra_2.13/2.8.0//algebra_2.13-2.8.0.jar
Expand Down Expand Up @@ -249,6 +248,7 @@ parquet-jackson/1.15.2//parquet-jackson-1.15.2.jar
pickle/1.5//pickle-1.5.jar
py4j/0.10.9.9//py4j-0.10.9.9.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
roaringbitmap/1.5.2//roaringbitmap-1.5.2.jar
rocksdbjni/9.8.4//rocksdbjni-9.8.4.jar
scala-collection-compat_2.13/2.7.0//scala-collection-compat_2.13-2.7.0.jar
scala-compiler/2.13.16//scala-compiler-2.13.16.jar
Expand Down
17 changes: 14 additions & 3 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,17 @@
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
Copy link
Contributor Author

@LuciferYang LuciferYang Jan 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new version of RoaringBitmap requires configuring additional repositories:

image

The second method requires additional configuration of a GitHub token, so Method One was chosen here.

<id>jitpack.io</id>
<name>JitPack</name>
<url>https://jitpack.io</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<!--
This is used as a fallback when the first try fails.
Expand Down Expand Up @@ -899,9 +910,9 @@
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<version>1.3.0</version>
<groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
<artifactId>roaringbitmap</artifactId>
<version>1.5.2</version>
</dependency>

<!-- Netty Begin -->
Expand Down
1 change: 1 addition & 0 deletions project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,7 @@ object SparkBuild extends PomBuild {
// Google Mirror of Maven Central, placed first so that it's used instead of flaky Maven Central.
// See https://storage-download.googleapis.com/maven-central/index.html for more info.
"gcs-maven-central-mirror" at "https://maven-central.storage-download.googleapis.com/maven2/",
"jitpack" at "https://jitpack.io",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering if this can cause a flakiness like the default Maven.

Due to the Apache Spark's high traffic, the Maven was flaky. So, we are using GCS Maven as the first cache here inevitably.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understand your concerns. We can wait for a while longer to see if RoaringBitmap can resume its publish on Maven Central

DefaultMavenRepository,
Resolver.mavenLocal,
Resolver.file("ivyLocal", file(Path.userHome.absolutePath + "/.ivy2/local"))(Resolver.ivyStylePatterns)
Expand Down