Skip to content

Add sql_parser.default_null_ordering config option to customize the default null ordering #16963

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 31, 2025

Conversation

goldmedal
Copy link
Contributor

@goldmedal goldmedal commented Jul 29, 2025

Which issue does this PR close?

  • Closes #.

Rationale for this change

The default null ordering is much different in different database.
In Postgres, the null value is considered as the max value.

psql=# select * from (values (1, 'one'), (2, 'two'), (null, 'three')) t(c1, c2) order by c1 asc;
 c1 |  c2   
----+-------
  1 | one
  2 | two
    | three
(3 rows)

psql=# select * from (values (1, 'one'), (2, 'two'), (null, 'three')) t(c1, c2) order by c1 desc;
 c1 |  c2   
----+-------
    | three
  2 | two
  1 | one
(3 rows)

In DuckDB, the null value is always at the last

D select * from (values (1, 'one'), (2, 'two'), (null, 'three')) t(c1, c2) order by c1 asc;
┌───────┬─────────┐
│  c1   │   c2    │
│ int32 │ varchar │
├───────┼─────────┤
│     1 │ one     │
│     2 │ two     │
│  NULL │ three   │
└───────┴─────────┘
D select * from (values (1, 'one'), (2, 'two'), (null, 'three')) t(c1, c2) order by c1 desc;
┌───────┬─────────┐
│  c1   │   c2    │
│ int32 │ varchar │
├───────┼─────────┤
│     2 │ two     │
│     1 │ one     │
│  NULL │ three   │
└───────┴─────────┘

As a composable database engine, I think it make sense to provide a config to customize the default behavior of null ordering.

What changes are included in this PR?

This PR introduces a new parser option default_null_ordering:

        /// Specifies the default null ordering for query results. There are 4 options:
        /// - `nulls_max`: Nulls appear last in ascending order.
        /// - `nulls_min`: Nulls appear first in ascending order.
        /// - `nulls_first`: Nulls always be first in any order.
        /// - `nulls_last`: Nulls always be last in any order.
        ///
        /// By default, `nulls_max` is used to follow Postgres's behavior.
        /// postgres rule: <https://www.postgresql.org/docs/current/queries-order.html>
        pub default_null_ordering: String, default = "nulls_max".to_string()

There are 4 cases for the null ordering. nulls_max is used by default to follow the Postgres's behavior.

Are these changes tested?

Are there any user-facing changes?

add a new config.

goldmedal and others added 3 commits July 29, 2025 19:07
* add default_null_ordering config

* add test for different config

* Update datafusion/sql/src/planner.rs

Co-authored-by: Copilot <[email protected]>

* update doc

* fix sqllogictest

---------

Co-authored-by: Copilot <[email protected]>
@github-actions github-actions bot added documentation Improvements or additions to documentation sql SQL Planner core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Jul 29, 2025
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @goldmedal -- this looks great to me. I had some small suggestions but nothing I think is required

@alamb alamb changed the title Allow to customize the default null ordering Add sql_parser.default_null_ordering config option to customize the default null ordering Jul 29, 2025
@goldmedal goldmedal merged commit d376a32 into apache:main Jul 31, 2025
28 checks passed
@goldmedal goldmedal deleted the feat/custom-null-ordering branch July 31, 2025 12:55
@goldmedal
Copy link
Contributor Author

Thanks @alamb for reviewing 👍

Standing-Man pushed a commit to Standing-Man/datafusion that referenced this pull request Aug 4, 2025
… default null ordering (apache#16963)

* Allow to set up the default null ordering (apache#3)

* add default_null_ordering config

* add test for different config

* Update datafusion/sql/src/planner.rs

Co-authored-by: Copilot <[email protected]>

* update doc

* fix sqllogictest

---------

Co-authored-by: Copilot <[email protected]>

* rename config and fix test

* update doc

* fix default

* fix doc

* fix sqllogictests

* address comments

---------

Co-authored-by: Copilot <[email protected]>
goldmedal added a commit to Canner/datafusion that referenced this pull request Aug 19, 2025
… default null ordering (apache#16963)

* Allow to set up the default null ordering (#3)

* add default_null_ordering config

* add test for different config

* Update datafusion/sql/src/planner.rs

Co-authored-by: Copilot <[email protected]>

* update doc

* fix sqllogictest

---------

Co-authored-by: Copilot <[email protected]>

* rename config and fix test

* update doc

* fix default

* fix doc

* fix sqllogictests

* address comments

---------

Co-authored-by: Copilot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
common Related to common crate core Core DataFusion crate documentation Improvements or additions to documentation sql SQL Planner sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants