chore(parser)!: refactor JSON extract by geooo109 · Pull Request #7302 · tobymao/sqlglot

geooo109 · 2026-03-16T18:58:58Z

Refactored the _parse_colon_as_variant_extract method.

I added extensive tests for sf and sf -> duckdb:

- v:a[0] — static index                                                                                                                                                                                   - v:a[0]:b — static index + path                                                                                                                                                                          
- v:a[s.x] — dynamic index
- v:c[s.x]:r — dynamic index + path                                                                                                                                                                       
- v:c[s.x]:r:d::varchar — dynamic index + path + cast      
- v:a:b[s.x] — nested path + dynamic index                                                                                                                                                                
- v:a.b[s.x].r.d — dot path + dynamic index + dot path                                                                                                                                                    
- v:a.b[s.x].r.d[s.y] — two dynamic indices with path between
- v:a[s.x][s.y] — chained dynamic indices
- v:a[s.x][0] — dynamic + static index
- v:c::variant[1] — cast + static index
- v:a::variant[0]::varchar — cast + static index + cast
- v:a.b[s.x].c::variant[0] — dynamic index + path + cast + static index
- v:a::variant[s.x] — cast + dynamic index
- col:"customer's department" — quoted key with special chars

/integration-tests

github-actions · 2026-03-16T19:42:16Z

SQLGlot Integration Test Results

Comparing:

this branch (sqlglot:geooo109/json_path_refactor, sqlglot version: geooo109/json_path_refactor)
baseline (main, sqlglot version: 0.0.1.dev1)

By Dialect

dialect	main	sqlglot:geooo109/json_path_refactor	transitions	links
bigquery -> bigquery	23872/23877 passed (100.0%)	23872/23877 passed (100.0%)	No change	full result / delta
bigquery -> duckdb	2106/2624 passed (80.3%)	2106/2624 passed (80.3%)	No change	full result / delta
duckdb -> duckdb	4003/4004 passed (100.0%)	4003/4004 passed (100.0%)	No change	full result / delta
snowflake -> duckdb	1508/2674 passed (56.4%)	1508/2674 passed (56.4%)	No change	full result / delta
snowflake -> snowflake	65910/65910 passed (100.0%)	65878/65910 passed (100.0%)	32 pass -> fail	full result / delta
databricks -> databricks	1364/1364 passed (100.0%)	1364/1364 passed (100.0%)	No change	full result / delta
postgres -> postgres	6040/6040 passed (100.0%)	6040/6040 passed (100.0%)	No change	full result / delta
redshift -> redshift	7059/7059 passed (100.0%)	7059/7059 passed (100.0%)	No change	full result / delta

Overall

main: 113552 total, 111862 passed (pass rate: 98.5%), sqlglot version: 0.0.1.dev1

sqlglot:geooo109/json_path_refactor: 113552 total, 111830 passed (pass rate: 98.5%), sqlglot version: geooo109/json_path_refactor

Transitions:
32 pass -> fail

geooo109 · 2026-03-17T13:11:23Z

Will solve the tests and re-push

geooo109 · 2026-03-17T16:07:17Z

sqlglot/parser.py

-                        break
+                        self.raise_error("Expected key name after '.'")
+                elif self._match(TokenType.L_BRACKET):
+                    bracket_expr = self._parse_bracket_key_value()


bracket_expr = self._parse_disjunction()

^ this should be true for every case, less function calls

geooo109 · 2026-03-17T16:07:29Z

sqlglot/parser.py

+                    if not bracket_expr:
+                        continue


Should we raise here or even ignore the None ?

What case does this correspond to? Is it a valid input? If not, ignore.

georgesittas

@geooo109 you checked all test cases against engines, right?

sqlglot/parser.py

georgesittas · 2026-03-18T08:38:42Z

sqlglot/parser.py

+                        path_parts.append(exp.JSONPathKey(this=next_key.name))
                    else:
-                        break
+                        self.raise_error("Expected key name after '.'")


I'm not sure about these exceptions.. If we fail to convert the JSON path into a canonical / transpilable AST, is there any way to fallback to a safe representation so that Snowflake roundtrip is not messed up?

georgesittas · 2026-03-18T08:39:22Z

sqlglot/parser.py

+                    if not bracket_expr:
+                        continue


What case does this correspond to? Is it a valid input? If not, ignore.

georgesittas · 2026-03-18T08:40:31Z

sqlglot/parser.py

+                elif self._match(TokenType.DCOLON):
+                    this, path_parts = self._build_json_extract(this, path_parts, escape)


Are brackets, dots and dcolons the only tokens we can find in these JSON path expressions, syntactically? In other words, is this branching logic complete?

geooo109 force-pushed the geooo109/json_path_refactor branch from 6bc278e to 35e1763 Compare March 16, 2026 19:01

geooo109 marked this pull request as draft March 16, 2026 19:11

geooo109 marked this pull request as ready for review March 17, 2026 10:45

geooo109 force-pushed the geooo109/json_path_refactor branch from 7c54312 to 383d216 Compare March 17, 2026 11:35

geooo109 commented Mar 17, 2026

View reviewed changes

georgesittas reviewed Mar 18, 2026

View reviewed changes

geooo109 added 9 commits March 18, 2026 14:08

chore(parser)!: refactor JSON extract

157afcc

refactor impl

880f233

ref 2

4711c4f

added tests

03bff4d

ref tests

ea6f5dc

refactor build extract method

59c0fb4

style

d280a26

ref tests

9643362

partial comments

afe1ba0

geooo109 force-pushed the geooo109/json_path_refactor branch from 49d750d to afe1ba0 Compare March 18, 2026 12:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(parser)!: refactor JSON extract#7302

chore(parser)!: refactor JSON extract#7302
geooo109 wants to merge 9 commits intomainfrom
geooo109/json_path_refactor

geooo109 commented Mar 16, 2026 •

edited by georgesittas

Loading

Uh oh!

github-actions bot commented Mar 16, 2026 •

edited

Loading

Uh oh!

geooo109 commented Mar 17, 2026

Uh oh!

geooo109 Mar 17, 2026

Uh oh!

geooo109 Mar 17, 2026 •

edited

Loading

Uh oh!

georgesittas Mar 18, 2026

Uh oh!

georgesittas left a comment

Uh oh!

Uh oh!

georgesittas Mar 18, 2026

Uh oh!

georgesittas Mar 18, 2026

Uh oh!

georgesittas Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		elif self._match(TokenType.DCOLON):
		this, path_parts = self._build_json_extract(this, path_parts, escape)

Conversation

geooo109 commented Mar 16, 2026 • edited by georgesittas Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SQLGlot Integration Test Results

By Dialect

Overall

Uh oh!

geooo109 commented Mar 17, 2026

Uh oh!

geooo109 Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

geooo109 Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

georgesittas Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

georgesittas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

georgesittas Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

georgesittas Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

georgesittas Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

geooo109 commented Mar 16, 2026 •

edited by georgesittas

Loading

github-actions bot commented Mar 16, 2026 •

edited

Loading

geooo109 Mar 17, 2026 •

edited

Loading