Skip to content

Spark: Add support for Spark 4.1 #14155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Spark: Add support for Spark 4.1 #14155

wants to merge 3 commits into from

Conversation

@manuzhang
Copy link
Member

No description provided.

@manuzhang manuzhang force-pushed the spark4.1-preview branch 2 times, most recently from f506107 to 7c1ebb6 Compare September 23, 2025 16:00
@manuzhang manuzhang mentioned this pull request Sep 23, 2025
Closed
@manuzhang
Copy link
Member Author

manuzhang commented Sep 23, 2025 •
edited
Loading

Created apache/spark#52423 at Spark side to fix failed tests for Spark 4.1.0-preview1.

025-09-22T16:49:13.6462602Z TestCreateActions > testAddColumnOnMigratedTableAtEnd() > catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, parquet-enabled=true, cache-enabled=false}, type = hive FAILED
2025-09-22T16:49:13.6466422Z     org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 431.0 failed 1 times, most recent failure: Lost task 0.0 in stage 431.0 (TID 3270) (localhost executor driver): java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.types.DataType.transformRecursively(scala.PartialFunction)" because "type" is null
2025-09-22T16:49:13.6469913Z     	at org.apache.spark.sql.vectorized.ColumnVector.<init>(ColumnVector.java:341)
2025-09-22T16:49:13.6471238Z     	at org.apache.iceberg.spark.data.vectorized.ConstantColumnVector.<init>(ConstantColumnVector.java:41)

dongjoon-hyun pushed a commit to apache/spark that referenced this pull request Sep 24, 2025
@manuzhang @dongjoon-hyun
[SPARK-53678][SQL] Fix NPE when subclass of ColumnVector is created w…
b4592c4 
…ith null DataType

### What changes were proposed in this pull request?
Check whether the parameter DataType is null in ColumnVector constructor before transforming it

### Why are the changes needed?
A subclass of ColumnVector, e.g. Iceberg's [ConstantColumnVector](https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ConstantColumnVector.java#L41), could be created with null `DataType`. It throws NPE after #51349, which can be verified by failed tests in [integrating Spark 4.1.0-preview1 in Iceberg](apache/iceberg#14155)

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
UT.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #52423 from manuzhang/SPARK-53678.

Authored-by: manuzhang <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
@manuzhang manuzhang changed the title Spark: Add support for Spark 4.1 (4.1.0-preview1) Spark: Add support for Spark 4.1 Sep 24, 2025
@manuzhang manuzhang force-pushed the spark4.1-preview branch 5 times, most recently from 7df730b to 030d9e8 Compare September 26, 2025 02:05
@manuzhang manuzhang mentioned this pull request Sep 26, 2025
Merged
@manuzhang manuzhang force-pushed the spark4.1-preview branch 2 times, most recently from 6efbecc to cac12c7 Compare September 30, 2025 06:49
@manuzhang manuzhang force-pushed the spark4.1-preview branch 2 times, most recently from 4388c21 to c3f856b Compare October 27, 2025 08:47
@manuzhang manuzhang mentioned this pull request Oct 27, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

build INFRA not-stale spark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@manuzhang