LazyFrame.join() documentation for `validate=` parameter seems inaccurate

[itamarst]

Description

LazyFrame.join() takes a validate parameter. Current docs say:

This is currently not supported by the streaming engine.

This appears to be inaccurate. As far as I can tell from reading the code, if validation is enabled (m:1, 1:m, 1:1) the streaming engine falls back to the in-memory engine, which then does validation. Empirical tests also suggest this is the case. So validation does happen, and memory usage can be significantly higher as streaming will be disabled.

If this is correct, and I'm not missing some edge case, a more accurate documentation for validate would be:

Using this parameter will result in higher memory usage, as it will disable streaming for the join.

Happy to submit a PR for this if I'm not misunderstanding.

Link

https://docs.pola.rs/api/python/stable/reference/lazyframe/api/polars.LazyFrame.join.html

[orlp]

I mean, in general "not supported by the streaming engine" means "falls back to in-memory".

[itamarst]

I mean, in general "not supported by the streaming engine" means "falls back to in-memory".

1. As context, this issue is conveying a real misunderstanding from someone who spent a bunch of time trying to figure this out.
2. If this is a general issue, it's arguably a general documentation problem? A subset of readers who haven't fully internalized the implications of the streaming model may have this problem in other places as well.
3. In this particular case, though, misunderstanding it's even more likely, since validation is a correctness check, not something that changes the output for correct inputs. So it's even more plausible to interpret it as "validation won't happen", rather than "validation will happen but you'll use more memory".