Use sketches-ddsketch fork with Java-compatible binary encoding by congx4 · Pull Request #2842 · quickwit-oss/tantivy

congx4 · 2026-02-18T17:41:21Z

Description

Reference the quickwit-oss/rust-sketches-ddsketch fork (via git + rev in Cargo.toml) which adds native Java-compatible binary serialization for DDSketch. This enables Rust applications to produce DDSketch bytes that Java consumers can directly deserialize and merge via DDSketchWithExactSummaryStatistics.decode() from the sketches-java library.

Why?

The upstream sketches-ddsketch Rust crate only supports serde-based serialization (JSON), while Java's sketches-java library uses a custom binary wire format. For distributed aggregation pipelines where Rust search nodes produce intermediate DDSketch results consumed by Java query orchestrators, binary compatibility is required.

What changed

Removed the vendored sketches-ddsketch/ directory from the tantivy workspace
Changed the sketches-ddsketch dependency from path = "./sketches-ddsketch" to git + rev pointing to quickwit-oss/rust-sketches-ddsketch@555caf1
The fork contains:
- encoding.rs: Java-compatible binary encode/decode using signed/unsigned varint and VarDouble encoding
- Cross-language golden-byte tests verifying binary compatibility with Java's DDSketch
- Config/Store/DDSketch modifications to support encode()/decode() methods

Testing

All existing tantivy aggregation tests pass (percentiles, cardinality, etc.)
All sketches-ddsketch tests pass including cross-language golden-byte tests

Fork sketches-ddsketch as a workspace member to add native Java binary serialization (to_java_bytes/from_java_bytes) for DDSketch. This enables pomsky to return raw DDSketch bytes that event-query can deserialize via DDSketchWithExactSummaryStatistics.decode(). Key changes: - Vendor sketches-ddsketch crate with encoding.rs implementing VarEncoding, flag bytes, and INDEX_DELTAS_AND_COUNTS store format - Align Config::key() to floor-based indexing matching Java's LogarithmicMapping - Add PercentilesCollector::to_sketch_bytes() for pomsky integration - Cross-language golden byte tests verified byte-identical with Java output Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Reference the exact Java source files in DataDog/sketches-java for Config::new(), Config::key(), Config::value(), Config::from_gamma(), and Store::add_count() so readers can verify the alignment. Co-authored-by: Cursor <cursoragent@cursor.com>

- manual_range_contains: use !(0.0..=1.0).contains(&q) - identity_op: simplify (0 << 2) | FLAG_TYPE to just FLAG_TYPE - manual_clamp: use .clamp(0, 8) instead of .max(0).min(8) - manual_repeat_n: use repeat_n() instead of repeat().take() - cast_abs_to_unsigned: use .unsigned_abs() instead of .abs() as usize Co-authored-by: Cursor <cursoragent@cursor.com>

- Replace bare constants with FlagType and BinEncodingMode enums - Use const fn for flag byte construction instead of raw bit ops - Replace if-else chain with nested match in decode_from_java_bytes - Use split_first() in read_byte for idiomatic slice consumption - Use split_at in read_f64_le to avoid TryInto on edition 2018 - Use u64::from(next) instead of `next as u64` casts - Extract assert_golden, assert_quantiles_match, bytes_to_hex helpers to reduce duplication across golden byte tests - Fix edition-2018 assert! format string compatibility - Clean up is_valid_flag_byte with let-else and match Co-authored-by: Cursor <cursoragent@cursor.com>

- Replace approximate PI/E constants with non-famous value in test - Fix reversed empty range (2048..0) → (0..2048).rev() in store test Co-authored-by: Cursor <cursoragent@cursor.com>

congx4 · 2026-02-19T16:05:59Z

sketches-ddsketch/src/encoding.rs

+use crate::ddsketch::DDSketch;
+use crate::store::Store;
+
+// ---------------------------------------------------------------------------


This file is generated by Opus and it generates tests to ensure it is fully correct. See more details from the pr description.

PSeitz · 2026-02-19T16:22:47Z

I don't think this should be in the tantivy repo.

The changes should be in your sketches-ddsketch fork, where we try to upstream it.

fulmicoton-dd · 2026-02-19T17:03:52Z

@congx4 I am not sure @PSeitz comment was clear. Can you just point to your repo with {git =, rev=}

congx4 · 2026-02-19T17:16:54Z

@congx4 I am not sure @PSeitz comment was clear. Can you just point to your repo with {git =, rev=}

Thanks, but looks like we don't have permissions to folk it under Datadog, I will just folk it under quickwit-oss.

Move the vendored sketches-ddsketch crate (with Java-compatible binary encoding) to its own repo at quickwit-oss/rust-sketches-ddsketch and reference it via git+rev in Cargo.toml. Co-authored-by: Cursor <cursoragent@cursor.com>

src/aggregation/metric/percentiles.rs

Cargo.toml

fulmicoton-dd

Approved, but please have a look at the comments and see if they make sense

Address review feedback: replace assert_eq! with assert_nearly_equals! for float values that go through JSON serialization roundtrips, which can introduce minor precision differences. Co-authored-by: Cursor <cursoragent@cursor.com>

…ctor Replace the derived Serialize/Deserialize on PercentilesCollector with custom impls that use DDSketch's Java-compatible binary encoding (encode_to_java_bytes / decode_from_java_bytes). This removes the need for the use_serde feature on sketches-ddsketch entirely. Also restore original float test values and use assert_nearly_equals! for all float comparisons in percentile tests, since DDSketch quantile estimates can have minor precision differences across platforms. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep use_serde on sketches-ddsketch so DDSketch derives Serialize/Deserialize, removing the need for custom impls on PercentilesCollector. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

congx4 and others added 7 commits February 18, 2026 11:36

fix: rustfmt nightly formatting for vendored sketches-ddsketch

df04c7d

Co-authored-by: Cursor <cursoragent@cursor.com>

fix: remove internal reference from code comment

cf760fd

Co-authored-by: Cursor <cursoragent@cursor.com>

fix: resolve remaining clippy errors in ddsketch

2dc4e9e

- Replace approximate PI/E constants with non-famous value in test - Fix reversed empty range (2048..0) → (0..2048).rev() in store test Co-authored-by: Cursor <cursoragent@cursor.com>

congx4 commented Feb 19, 2026

View reviewed changes

congx4 requested review from PSeitz, fulmicoton-dd and trinity-1686a February 19, 2026 16:06

Replace vendored sketches-ddsketch with git dependency

b9ace00

Move the vendored sketches-ddsketch crate (with Java-compatible binary encoding) to its own repo at quickwit-oss/rust-sketches-ddsketch and reference it via git+rev in Cargo.toml. Co-authored-by: Cursor <cursoragent@cursor.com>

congx4 changed the title ~~Vendor sketches-ddsketch with Java-compatible binary encoding~~ Use sketches-ddsketch fork with Java-compatible binary encoding Feb 19, 2026

fulmicoton-dd reviewed Feb 19, 2026

View reviewed changes

src/aggregation/metric/percentiles.rs Outdated Show resolved Hide resolved

fulmicoton-dd reviewed Feb 19, 2026

View reviewed changes

Cargo.toml Show resolved Hide resolved

fulmicoton-dd approved these changes Feb 19, 2026

View reviewed changes

congx4 and others added 3 commits February 19, 2026 13:21

Restore use_serde feature and simplify PercentilesCollector

2098fca

Keep use_serde on sketches-ddsketch so DDSketch derives Serialize/Deserialize, removing the need for custom impls on PercentilesCollector. Co-authored-by: Cursor <cursoragent@cursor.com>

fulmicoton-dd approved these changes Feb 19, 2026

View reviewed changes

Fix nightly fmt: merge crate imports in percentiles tests

18fedd9

Co-authored-by: Cursor <cursoragent@cursor.com>

congx4 force-pushed the congxie/replaceHll branch 2 times, most recently from 9f764cb to 18fedd9 Compare February 20, 2026 02:07

trinity-1686a merged commit d0c5ffb into main Feb 20, 2026
12 checks passed

trinity-1686a deleted the congxie/replaceHll branch February 20, 2026 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use sketches-ddsketch fork with Java-compatible binary encoding#2842

Use sketches-ddsketch fork with Java-compatible binary encoding#2842
trinity-1686a merged 12 commits intomainfrom
congxie/replaceHll

congx4 commented Feb 18, 2026 •

edited

Loading

Uh oh!

congx4 Feb 19, 2026

Uh oh!

PSeitz commented Feb 19, 2026

Uh oh!

fulmicoton-dd commented Feb 19, 2026

Uh oh!

congx4 commented Feb 19, 2026

Uh oh!

Uh oh!

Uh oh!

fulmicoton-dd left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

congx4 commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Why?

What changed

Testing

Uh oh!

congx4 Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

PSeitz commented Feb 19, 2026

Uh oh!

fulmicoton-dd commented Feb 19, 2026

Uh oh!

congx4 commented Feb 19, 2026

Uh oh!

Uh oh!

Uh oh!

fulmicoton-dd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

congx4 commented Feb 18, 2026 •

edited

Loading