[WIP][PYTHON][ML] Don't use pickle to save/load models in pyspark.ml.connect

[zhengruifeng]

What changes were proposed in this pull request?

save/load models in pyspark.ml.connect with arrow-based parquet format

Why are the changes needed?

pickle.load can run arbitrary codes and cause security issues

Does this PR introduce any user-facing change?

no, the whole module pyspark.ml.connect is never documented, so should not affect any end users

How was this patch tested?

ci

Was this patch authored or co-authored using generative AI tooling?

no

[dongjoon-hyun]

Thank you, @zhengruifeng .

Do you mean this is about storing and loading a transient and temporary data?

no, the whole module pyspark.ml.connect is never documented, so should not affect any end users

I'm wondering if we need to keep the existing code path. Is there any chance for the existing users to try to load old data format?

[allisonwang-db]

Will this introduce any behavior changes?

[HyukjinKwon]

From my understanding, it should be fine. Arrow is a required dependency for connect in ML, and the data being used here has the explicit type.

pickle itself isn't considered as a stable type for storage (it does change even between minor Python versions)

[HyukjinKwon]

I am not against adding the legacy code path but TBH ML connect is still sort of immature (hence not documented) cc @WeichenXu123 for second opinion if you have

[zhengruifeng]

@dongjoon-hyun @allisonwang-db the affected algorithms were never documented in the API Reference so far, they were added in 3.5 as the first attempt to support ml on connect, then they were deprecated in 4.0. (To be conservative, we didn't remove them in 4.0)

This issue was raised by apache security team, I guess no end users will be affected.

[dongjoon-hyun]

It sounds good to me.

[dongjoon-hyun]

BTW, please remove [WIP] before you are going to land this. I just switched to Draft status to prevent accidental merging with [WIP] in the title because this PR got approval already.