Skip to content

Commit ca629aa

Browse files
committed
[SPARK-53001][TESTS] Fix test_df_unpivot to pass with Spark 4.0
# Description This PR proposes to fix the issue that `test_df_unpivot` doesn't pass with Spark 4.0. ``` ---- dataframe::tests::test_df_unpivot stdout ---- SparkSession Setup thread 'dataframe::tests::test_df_unpivot' panicked at crates/connect/src/dataframe.rs:2463:9: assertion `left == right` failed left: RecordBatch { schema: Schema { fields: [Field { name: "id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "var", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "val", data_type: Float32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, columns: [PrimitiveArray<Int64> [ 1, 1, 2, 2, ], StringArray [ "int", "float", "int", "float", ], PrimitiveArray<Float32> [ 11.0, 1.1, 12.0, 1.2, ]], row_count: 4 } right: RecordBatch { schema: Schema { fields: [Field { name: "id", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "var", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "val", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }, columns: [PrimitiveArray<Int64> [ 1, 1, 2, 2, ], StringArray [ "int", "float", "int", "float", ], PrimitiveArray<Float64> [ 11.0, 1.100000023841858, 12.0, 1.2000000476837158, ]], row_count: 4 } ``` As of Spark 4.0, ANSI mode is enabled by default but this test doesn't consider it. To fix this issue, I tweaked the test so that the test pass in both ANSI mode is enabled or not. ## Related Issue(s) SPARK-53001 Closes #4 from sarutak/fix-unpivot-test. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Kousuke Saruta <[email protected]>
1 parent 589ea87 commit ca629aa

File tree

1 file changed

+8
-6
lines changed

1 file changed

+8
-6
lines changed

crates/connect/src/dataframe.rs

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2445,12 +2445,14 @@ mod tests {
24452445

24462446
let df = spark.create_dataframe(&data)?;
24472447

2448-
let df = df.unpivot(
2449-
[col("id")],
2450-
Some(vec![col("int"), col("float")]),
2451-
"var",
2452-
"val",
2453-
);
2448+
let df = df
2449+
.unpivot(
2450+
[col("id")],
2451+
Some(vec![col("int"), col("float")]),
2452+
"var",
2453+
"val",
2454+
)
2455+
.select(vec![col("id"), col("var"), col("val").cast("float")]);
24542456

24552457
let res = df.collect().await?;
24562458

0 commit comments

Comments
 (0)