You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While working on #1887 I looked at logic related to wrapping nodes with CopyExec (mostly for unpacking dictionaries. We have logic in planner.rs in several places that determines when to insert CopyExec nodes in the native physical plan. We should move that logic to the Spark side and define a new CopyExec node that we can translate 1:1 on the native side. This has a few benefits:
CopyExec nodes will show up in the Spark UI representation of the plan
More logic about plan translation is consolidated on the Spark side, which offers benefits of its own:
Easier to understand what the final native plan will be during Comet rule execution on the Spark side
Easier to cache serialized native plans
Reduce redundant work across Spark workers in native code