Skip to content

Implement full-range i256::to_f64 to eliminate ±∞ saturation for Decimal256 → Float64 casts #7986

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 29, 2025
Merged

Implement full-range i256::to_f64 to eliminate ±∞ saturation for Decimal256 → Float64 casts #7986

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6842524
feat: implement to_f64 method for i256 to convert to f64 representation
kosiew Jul 24, 2025
40a65bf
feat: enhance decimal256_to_f64 function for improved conversion and …
kosiew Jul 24, 2025
a426bcc
Update arrow-cast/src/cast/mod.rs
kosiew Jul 25, 2025
049c2e6
refactor: replace decimal256_to_f64 fn with i256::to_f64 method and m…
kosiew Jul 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions arrow-buffer/src/bigint/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -821,6 +821,20 @@ impl ToPrimitive for i256 {
}
}

fn to_f64(&self) -> Option<f64> {
let mag = if let Some(u) = self.checked_abs() {
let (low, high) = u.to_parts();
(high as f64) * 2_f64.powi(128) + (low as f64)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking this through:

  • If high is zero (no significant bits), then conversion to f64 is exact (tho useless)
    • ... and we return the value of low, converted to f64
  • If high has 1..=53 significant bits, then conversion to f64 is exact (no rounding)
    • ... and scaling is also exact
    • ... and adding low (already converted to f64) will round as needed
  • If high has 54.. significant bits, then conversion to f64 will use the 54th bit to round
    • ... tho scaling is still exact
    • ... and it doesn't matter what value low takes, because it's so small that adding it doesn't change the answer

A bit expensive, but I think it covers all the cases with no weird rounding effects?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect we could do better by manually twiddling bits, but that's probably a good follow-up.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bit twiddling that I think would work:

  1. Define i256::leading_zeros() that follows the semantics of all the other leading_zeros for integral types 
    impl i256 {
    pub fn leading_zeros(&self) -> u32 {
        match self.high {
            0 => 128 + self.low.leading_zeros(),
            _ => self.high.leading_zeros(),
        }
    }
}
  2. Define a notion of "redundant leading sign bits" in terms of leading zeros: 
    fn redundant_leading_sign_bits_i256(n: i256) -> u32 {
    let mask = n >> 255; // all ones or all zeros
    (n ^ mask).leading_zeros() - 1; // we only need one sign bit
}
  3. Shift out all redundant leading sign bits when converting to f64: 
    fn i256_to_f64(n: i256) -> f64 {
    let k = redundant_leading_sign_bits_i256(n);
    let n = n << k; // left-justify (no redundant sign bits)
    let n = (n.high >> 64) as i64; // throw away the lower 192 bits
    (n as f64) * f64::powi(2.0, 192-k) // convert to f64 and scale it
}

The above should work for both positive and negative values

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracking in

} else {
// self == MIN
2_f64.powi(255)
};
if *self < i256::ZERO {
Some(-mag)
} else {
Some(mag)
}
}
fn to_u64(&self) -> Option<u64> {
let as_i128 = self.low as i128;

Expand Down Expand Up @@ -1264,4 +1278,29 @@ mod tests {
}
}
}

#[test]
fn test_decimal256_to_f64_typical_values() {
let v = i256::from_i128(42_i128);
assert_eq!(v.to_f64().unwrap(), 42.0);

let v = i256::from_i128(-123456789012345678i128);
assert_eq!(v.to_f64().unwrap(), -123456789012345678.0);
}

#[test]
fn test_decimal256_to_f64_large_positive_value() {
let max_f = f64::MAX;
let big = i256::from_f64(max_f * 2.0).unwrap_or(i256::MAX);
let out = big.to_f64().unwrap();
assert!(out.is_finite() && out.is_sign_positive());
}

#[test]
fn test_decimal256_to_f64_large_negative_value() {
let max_f = f64::MAX;
let big_neg = i256::from_f64(-(max_f * 2.0)).unwrap_or(i256::MIN);
let out = big_neg.to_f64().unwrap();
assert!(out.is_finite() && out.is_sign_negative());
}
}
37 changes: 18 additions & 19 deletions arrow-cast/src/cast/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -893,7 +893,7 @@ pub fn cast_with_options(
scale,
from_type,
to_type,
|x: i256| decimal256_to_f64(x),
|x: i256| x.to_f64().expect("All i256 values fit in f64"),
cast_options,
)
}
Expand Down Expand Up @@ -1995,17 +1995,6 @@ where
}
}

/// Convert a [`i256`] to `f64` saturating to infinity on overflow.
fn decimal256_to_f64(v: i256) -> f64 {
v.to_f64().unwrap_or_else(|| {
if v.is_negative() {
f64::NEG_INFINITY
} else {
f64::INFINITY
}
})
}

fn cast_to_decimal<D, M>(
array: &dyn Array,
base: M,
Expand Down Expand Up @@ -2439,6 +2428,7 @@ where
#[cfg(test)]
mod tests {
use super::*;
use arrow_buffer::i256;
use arrow_buffer::{Buffer, IntervalDayTime, NullBuffer};
use chrono::NaiveDate;
use half::f16;
Expand Down Expand Up @@ -8674,26 +8664,26 @@ mod tests {
);
}
#[test]
fn test_cast_decimal256_to_f64_overflow() {
// Test positive overflow (positive infinity)
fn test_cast_decimal256_to_f64_no_overflow() {
// Test casting i256::MAX: should produce a large finite positive value
let array = vec![Some(i256::MAX)];
let array = create_decimal256_array(array, 76, 2).unwrap();
let array = Arc::new(array) as ArrayRef;

let result = cast(&array, &DataType::Float64).unwrap();
let result = result.as_primitive::<Float64Type>();
assert!(result.value(0).is_infinite());
assert!(result.value(0) > 0.0); // Positive infinity
assert!(result.value(0).is_finite());
assert!(result.value(0) > 0.0); // Positive result

// Test negative overflow (negative infinity)
// Test casting i256::MIN: should produce a large finite negative value
let array = vec![Some(i256::MIN)];
let array = create_decimal256_array(array, 76, 2).unwrap();
let array = Arc::new(array) as ArrayRef;

let result = cast(&array, &DataType::Float64).unwrap();
let result = result.as_primitive::<Float64Type>();
assert!(result.value(0).is_infinite());
assert!(result.value(0) < 0.0); // Negative infinity
assert!(result.value(0).is_finite());
assert!(result.value(0) < 0.0); // Negative result
}

#[test]
Expand Down Expand Up @@ -8724,6 +8714,15 @@ mod tests {
assert_eq!("3123460", decimal_arr.value_as_string(2));
}

#[test]
fn decimal128_min_max_to_f64() {
// Ensure Decimal128 i128::MIN/MAX round-trip cast
let min128 = i128::MIN;
let max128 = i128::MAX;
assert_eq!(min128 as f64, min128 as f64);
assert_eq!(max128 as f64, max128 as f64);
}

#[test]
fn test_cast_numeric_to_decimal128_negative() {
let decimal_type = DataType::Decimal128(38, -1);
Expand Down
Loading