2 major changes:
Spark 4 Support ⭐
DataFlint now support spark 4! due to breaking changes in spark, it does require using different dataflint artifact name:
io.dataflint:dataflint_spark4_2.13:0.6.0
(Instead of io.dataflint:dataflint_spark4_2.13:0.6.0 ) for spark 3
Stage identification by stages with statistics 📊
This means better DataFlint support for spark accelerators such as Nvidia RAPIDs for Spark.
Now DataFlint can also identify sql node stages by metrics with statistics that mentioned the stage and task data it got the min/median/max from.