[SPARK-52922][PS] Avoid CAST_INVALID_INPUT of "astype" in ANSI mode

[xinrong-meng]

What changes were proposed in this pull request?

Avoid CAST_INVALID_INPUT of "astype" in ANSI mode

Why are the changes needed?

Ensure pandas on Spark works well with ANSI mode on.
Part of https://issues.apache.org/jira/browse/SPARK-52556.

Does this PR introduce any user-facing change?

>>> import pandas as pd
>>> import numpy as np
>>> 
>>> ps.set_option("compute.fail_on_ansi_mode", False)
>>> ps.set_option("compute.ansi_mode_support", True)

BEFORE
```py
>>> ps.Series(["abc"]).astype(int)
...
org.apache.spark.SparkNumberFormatException: [CAST_INVALID_INPUT] The value 'abc' of the type "STRING" cannot be cast to "BIGINT" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. SQLSTATE: 22018
...

AFTER

>>> ps.Series(["abc"]).astype(int)
0   NaN                                                                         
dtype: float64

Following non ANSI:

>>> spark.conf.set("spark.sql.ansi.enabled", False)
>>> ps.Series(["abc"]).astype(int)
0   NaN
dtype: float64

How was this patch tested?

Existing tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[xinrong-meng]

@ueshin may I get a review please?

[ueshin]

I'm wondering which we should follow here, pandas or non-ansi:

>>> pd.Series(["abc"]).astype(int)
Traceback (most recent call last):
...
ValueError: invalid literal for int() with base 10: 'abc'

[xinrong-meng]

Hi @ueshin as for astype, it reaches parity with non-ansi, as shown below:

>>> spark.conf.set("spark.sql.ansi.enabled", True)
>>> ps.Series(["abc"]).astype(int)
0   NaN                                                                         
dtype: float64

>>> spark.conf.set("spark.sql.ansi.enabled", False)
>>> ps.Series(["abc"]).astype(int)
0   NaN
dtype: float64

To mimic pandas, we can check whether values of the string column all represent valid integers or decimals, for example

numeric_regex = r"^\s*-?\d+(\.\d+)?\s*$"
col.rlike(numeric_regex)).count() == 0

but that is costly, and may not cover all corner cases.

WDYT?

[ueshin]

I guess we can leave it as is, raising the error, as pandas also raises an error, although the error type is different.

[xinrong-meng]

Thank you @ueshin !

The error is thrown inside the JVM executor as SparkNumberFormatException for now

org.apache.spark.SparkNumberFormatException: [CAST_INVALID_INPUT] The value 'abc' of the type "STRING" cannot be cast to "BIGINT" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. SQLSTATE: 22018

We might want to propagate the exception to Python driver as a follow up for error improvement.
I will close the PR for now.