[SPARK-52837][CONNECT][PYTHON] Support TimeType literal in Connect #51515
Conversation
dengziming
changed the title
|
Hello @zhengruifeng @MaxGekk @peter-toth , please take a look at this, we should also support time literals in Connect. |
There was a problem hiding this comment.
@dengziming Please, open new ticket specifically for the python client.
dengziming
changed the title
dengziming
changed the title
|
@MaxGekk I have created a separate ticket. |
| @@ -384,6 +385,33 @@ def fromInternal(self, v: int) -> datetime.date: | |||
| return datetime.date.fromordinal(v + self.EPOCH_ORDINAL) | |||
|
|
|||
|
|
|||
| class TimeType(AtomicType): | |||
There was a problem hiding this comment.
this PR actually add a new datatype in pyspark, not just python client.
It should work with both python client and pyspark classic.
We'd better also add test for pyspark classic.
There was a problem hiding this comment.
And I just notice that the class hierarchy is not consistent with the JVM side:
case class TimeType(precision: Int) extends AnyTimeType
We'd better make them the same, by also introducing the AnyTimeType in pyspark. If we need to touch other existing date time types, we can do it in a separate PR.
There was a problem hiding this comment.
I'm not very professional with PySpark, this will take some time, I changed it to WIP by now.
What changes were proposed in this pull request?
This is the follow-up of #51462 to support TimeType literal in pyspark connect.
Why are the changes needed?
To align the Python Connect client with the Java/Scala Connect client.
Does this PR introduce any user-facing change?
Yes, we can use TimeType literal in several ways, for example,
PySparkSession.sql("SELECT TIME '12:13:14'")and
pyspark.sql.connect.functions.lit(datetime.time(12, 13, 14)).How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?
No