Skip to content

[SPARK-52869][SQL] Add FrameLessOffsetWindowFunction validation to validateResolvedWindowExpression for reuse in single-pass analyzer #51559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[SPARK-52869][SQL] Add FrameLessOffsetWindowFunction validation to validateResolvedWindowExpression for reuse in single-pass analyzer #51559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6b10cc1
Initial commit.
nikola-jovicevic-db Jul 18, 2025
9291c09
Remove check from `checkAnalysis`.
nikola-jovicevic-db Jul 18, 2025
e7152b7
Separate into two checks to reduce diff.
nikola-jovicevic-db Jul 21, 2025
5489c52
Separate checks into functions.
nikola-jovicevic-db Jul 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,15 +437,6 @@ trait CheckAnalysis extends LookupCatalog with QueryErrorsBase with PlanToString
errorClass = "WINDOW_FUNCTION_WITHOUT_OVER_CLAUSE",
messageParameters = Map("funcName" -> toSQLExpr(w)))

case w @ WindowExpression(wf: FrameLessOffsetWindowFunction,
WindowSpecDefinition(_, order, frame: SpecifiedWindowFrame))
if order.isEmpty || !frame.isOffset =>
w.failAnalysis(
errorClass = "WINDOW_FUNCTION_AND_FRAME_MISMATCH",
messageParameters = Map(
"funcName" -> toSQLExpr(wf),
"windowExpr" -> toSQLExpr(w)))

case agg @ AggregateExpression(listAgg: ListAgg, _, _, _, _)
if agg.isDistinct && listAgg.needSaveOrderValue =>
throw QueryCompilationErrors.functionAndOrderExpressionMismatchError(
Expand Down
27 changes: 26 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/WindowResolution.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,15 +107,40 @@ object WindowResolution {
*
* By checking the type and configuration of [[WindowExpression.windowFunction]] it enforces the
* following rules:
* - Disallows [[FrameLessOffsetWindowFunction]] (e.g. [[Lag]]) without defined ordering or
* one with a frame which is defined as something other than an offset frame (e.g.
* `ROWS BETWEEN` is logically incompatible with offset functions).
* - Disallows distinct aggregate expressions in window functions.
* - Disallows use of certain aggregate functions - [[ListaAgg]], [[PercentileCont]],
* - Disallows use of certain aggregate functions - [[ListAgg]], [[PercentileCont]],
* [[PercentileDisc]], [[Median]]
* - Allows only window functions of following types:
* - [[AggregateExpression]] (non-distinct)
* - [[FrameLessOffsetWindowFunction]]
* - [[AggregateWindowFunction]]
*/
def validateResolvedWindowExpression(windowExpression: WindowExpression): Unit = {
checkWindowFunctionAndFrameMismatch(windowExpression)
checkWindowFunction(windowExpression)
}

def checkWindowFunctionAndFrameMismatch(windowExpression: WindowExpression): Unit = {
windowExpression match {
case _ @ WindowExpression(
windowFunction: FrameLessOffsetWindowFunction,
WindowSpecDefinition(_, order, frame: SpecifiedWindowFrame)
) if order.isEmpty || !frame.isOffset =>
windowExpression.failAnalysis(
errorClass = "WINDOW_FUNCTION_AND_FRAME_MISMATCH",
messageParameters = Map(
"funcName" -> toSQLExpr(windowFunction),
"windowExpr" -> toSQLExpr(windowExpression)
)
)
case _ =>
}
}

def checkWindowFunction(windowExpression: WindowExpression): Unit = {
windowExpression.windowFunction match {
case AggregateExpression(_, _, true, _, _) =>
windowExpression.failAnalysis(
Expand Down