Skip to content

fix(spark): format_string %s of a DECIMAL and %g like Java's Formatter - #26346

Draft
ozenyunus35 wants to merge 1 commit into
apache:mainfrom
ozenyunus35:spark-format-string-decimal-g
Draft

ozenyunus35 wants to merge 1 commit into
apache:mainfrom
ozenyunus35:spark-format-string-decimal-g

Conversation

@ozenyunus35

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Spark's format_string uses Java's Formatter, and datafusion-spark differed from it in two ways: %s of a DECIMAL printed the unscaled integer (123456 for 123.456), and %g / %G dropped trailing zeros like C (1234.5), while Java always prints the requested number of significant digits (1234.50).

What changes are included in this PR?

  • %s / %S of a Decimal128 or Decimal256 prints the value with its scale in plain notation (123.456, 0.50), through a small decimal_to_plain_string helper.
  • %g / %G no longer strip trailing zeros, for float and decimal arguments. The strip_trailing_0s flag and trim_trailing_0s were only used for this, so they are removed.
  • While adding cases, %g of 0.0 panicked with "attempt to subtract with overflow", because the exponent came from log10(0). A decimal_exponent helper returns 0 for zero, so %g gives 0.00000 and %e gives 0.000000e+00, as in Java.

I checked every new expected value by running String.format with the same pattern on JDK 21 (BigDecimal for the decimal %s cases).

One thing I left as is: Java's BigDecimal.toString switches to E notation for very small values (1E-7 for 0.0000001); this PR prints those in plain notation. I couldn't confirm what Spark's own Decimal.toString gives there without a Spark install, so I didn't guess.

What is the testing strategy for this PR?

  • format_string.slt: the expectations the issue points at are updated (%s / %S of Decimal128 and Decimal256, %g / %G of Float64, Float32, Decimal128 and Decimal256), and new cases cover %g on small, large, zero, negative and rounded values, %.3g, %.0g, %10.2g, %e of zero, and %s / %S / %10s of decimals. On main the file fails (and the %g of 0.0 case panics).
  • cargo test -p datafusion-spark --lib format_string and cargo clippy -p datafusion-spark --all-targets -- -D warnings pass.

Are there any user-facing changes?

Yes: format_string('%s', <decimal>) prints the decimal with its scale, %g / %G keep trailing zeros, and %g of zero no longer panics.

I used an AI assistant to help draft parts of the code and tests; I reviewed the changes and ran the tests myself.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) spark labels Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] format_string prints the unscaled value for %s of a DECIMAL and strips trailing zeros for %g

1 participant