Wrong cast between structs with different field order

[aroig]

Describe the bug

We have observed that casting a struct with fields of the same type, to a schema that re-orders the fields, miss-assigns the values. It appears like the values are assigned positionally, instead of by field name.

To Reproduce

I've prepared a reproducing example here: https://github.com/aroig/datafusion-cast-bug

Just do cargo run to reproduce.

Expected behavior

I would expect that casting {"b": 3, "a": 4}, to schema {"a": Int, "b": Int} would produce {"a": 4, "b": 3}, however the obtained result is {"a": 3, "b": 4}, as if the values are assigned positionally instead of by field name.

Additional context

No response

[alamb]

I think this is also related:

• Inconsistent and incorrect struct field coercion #14396

[alamb]

This might be related to this PR from @kosiew as well:

• DRAFT - Expose and generalize cast_column to enable struct → struct casting in more contexts #17281

[aroig]

I missed that PR! I just tested it and unfortunately, it does not fix this ordering issue.

[kosiew]

hi @aroig ,

Are you using ColumnarValue::cast_to, by any chance?
The current casting path in ColumnarValue::cast_to delegates to Arrow’s cast_with_options, which casts structs positionally.

[aroig]

Are you using ColumnarValue::cast_to, by any chance?

I am using logical_expr::cast, as in here. Not sure if that uses ColumnarValue::cast_to underneath.

For context, I came to this as a user of delta-rs. This is the original problem we had. After some digging, it seemed the unexpected behaviour was propagated from datafusion.

So, it appears logical_expr::cast does cast structs positionally, is this the intended behaviour? Is there functionality available to cast structs in a safer way (by field name, not positionally?). Casting positionally was surprising and unexpected behaviour for us, which lead to silent data corruption.

Thanks!