feat: support Spark-compatible abs math function
#18205
Conversation
|
cc @comphead for code review, thank you. |
|
|
||
| # abs: signed int minimal values | ||
| query IIII | ||
| select abs(c1), abs(c2), abs(c3), abs(c4) from test_nullable_integer where dataset = 'mins' |
There was a problem hiding this comment.
wondering would be that easier to test like
query II
select abs(1), abs(-1)
----
1 1
?
instead of creating/dropping tables
| 0 0 | ||
| 1 1 | ||
| 1 1 | ||
| NULL NULL |
There was a problem hiding this comment.
its better to use inline query, in this example the answers and input data are out of order and it might be more difficult to read
| ## PySpark 3.5.5 Result: {"abs(INTERVAL '-1-1' YEAR TO MONTH)": 13, "typeof(abs(INTERVAL '-1-1' YEAR TO MONTH))": 'interval year to month', "typeof(INTERVAL '-1-1' YEAR TO MONTH)": 'interval year to month'} | ||
| #query | ||
| #SELECT abs(INTERVAL '-1-1' YEAR TO MONTH::interval year to month); | ||
| query error DataFusion error: This feature is not implemented: Unsupported SQL type INTERVAL YEAR TO MONTH |
There was a problem hiding this comment.
Lets create a github ticket to fix this and refer to it in the comments in addition to the error.
Looks like abs works with intervals for Spark only
| impl SparkAbs { | ||
| pub fn new() -> Self { | ||
| Self { | ||
| signature: Signature::user_defined(Volatility::Immutable), |
There was a problem hiding this comment.
| signature: Signature::user_defined(Volatility::Immutable), | |
| signature: Signature::numeric(1, Volatility::Immutable), |
Lets keep it this way for now since the PR doesn't support Intervals
| let fail_on_error = if args.len() == 2 { | ||
| match &args[1] { | ||
| ColumnarValue::Scalar(ScalarValue::Boolean(Some(fail_on_error))) => { | ||
| *fail_on_error | ||
| } |
There was a problem hiding this comment.
Is this branch actually being tested?
| #[test] | ||
| fn test_abs_u8_scalar() { | ||
| with_fail_on_error(|fail_on_error| { | ||
| let args = ColumnarValue::Scalar(ScalarValue::UInt8(Some(u8::MAX))); | ||
| let fail_on_error_arg = | ||
| ColumnarValue::Scalar(ScalarValue::Boolean(Some(fail_on_error))); | ||
| match spark_abs(&[args, fail_on_error_arg]) { | ||
| Ok(ColumnarValue::Scalar(ScalarValue::UInt8(Some(result)))) => { | ||
| assert_eq!(result, u8::MAX); | ||
| Ok(()) | ||
| } | ||
| Err(e) => { | ||
| if fail_on_error { | ||
| assert!( | ||
| e.to_string().contains("ARITHMETIC_OVERFLOW"), | ||
| "Error message did not match. Actual message: {e}" | ||
| ); | ||
| Ok(()) | ||
| } else { | ||
| panic!("Didn't expect error, but got: {e:?}") | ||
| } | ||
| } | ||
| _ => unreachable!(), | ||
| } | ||
| }); | ||
| } |
There was a problem hiding this comment.
This test design is very confusing; we can't tell if a test case is meant to return Ok or Err as it automatically does the "correct" verification for each case. This automatic way of passing the test on Err should be switched so if we have a test case that is meant to return Err, that is the only thing we check for.
| fn arithmetic_overflow_error(from_type: &str) -> DataFusionError { | ||
| ArrowError( | ||
| Box::from(arrow::error::ArrowError::ComputeError(format!( | ||
| "arithmetic overflow from {from_type}", | ||
| ))), | ||
| None, | ||
| ) | ||
| } |
There was a problem hiding this comment.
I feel we should return a DataFusionError::Execution here instead of creating an arrow error and wrapping it in datafusion error, given the error occurs in our datafusion code
| let n = $ARRAY.as_any().downcast_ref::<$TYPE>(); | ||
| match n { | ||
| Some(array) => { |
There was a problem hiding this comment.
I would prefer if we unwrap n directly instead of matching on it, as we are guaranteed it would be of the correct array type; same goes for ansi_compute_op below
3 participants
Which issue does this PR close?
datafusion-sparkSpark Compatible Functions #15914Rationale for this change
abs()behaves differently than DataFusion.spark.sql.ansi.enabled. When it is off, arithmetic overflow doesn't throw exception like DataFusion does.YearMonthIntervalTypeandDayTimeIntervalTypeabsdatafusion-comet#2595What changes are included in this PR?
v4.0.1abs expressionabs()API takes an additional flagfail_on_errorifspark.sql.ansi.enabled=trueat caller's side.Are these changes tested?
test_files/spark/math/abs.sltAre there any user-facing changes?
Yes, the abs function can be specified in the SQL.
spark.sql.ansi.enabled=trueYearMonthIntervalTypeandDayTimeIntervalType