-
Notifications
You must be signed in to change notification settings - Fork 171
Add NDJSON support in polars #1431
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@AvinashYerra would you mind rebasing against the latest hamilton main? I should have merged a fix for a bunch of unit tests. |
5dc8d8a to
a6ea097
Compare
|
Done @skrawcz, rebased to latest main. Thanks! |
|
ugh sorry -- fix here #1437 -- if you wanted to apply it to this PR that would also work... |
|
One more rebase please :) |
|
Hi @skrawcz , just for confirmation. you want me to rebase to the latest main right? |
|
|
Strange. You can see that I merged #1437 ? So it's on main. Do you have that commit? |
a6ea097 to
4342d84
Compare
|
@skrawcz, It is done now. |
|
Hi @skrawcz , hope you had a great weekend. Can you check now whether the PR is mergeable |
skrawcz
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks!

This PR implements NDJSON (newline-delimited JSON) reader and writer support for Polars, following the same pattern as the existing JSON reader/writer implementation.
coming from #1197
Changes
Added PolarsNDJSONReader class in hamilton/plugins/polars_post_1_0_0_extensions.py
Added PolarsNDJSONWriter class in hamilton/plugins/polars_post_1_0_0_extensions.py
Supports writing NDJSON files using data.write_ndjson()
Registered both classes in
register_data_loaders()function to make them available through the materialization systemAdded the following tests:
Added example in examples/polars/materialization/my_script.py
How I tested this
Ran the specific NDJSON tests:
Both tests passed
Notes
Checklist