[ISSUE-149] Add column projection support to Python LogScanner #151
Conversation
leekeiabstraction
left a comment
leekeiabstraction
left a comment
There was a problem hiding this comment.
Thank you for the PR! Left a couple of comments. PTAL
| let rust_scanner = table_scan.create_log_scanner().map_err(|e| { | ||
| FlussError::new_err(format!("Failed to create log scanner: {e}")) | ||
| })?; |
There was a problem hiding this comment.
We seem to use FlussError and PyErr within this file, for example line 72 to 75 uses PyErr. Can you clarify when each should be used?
let rust_scanner = table_scan.create_log_scanner().map_err(|e| {
PyErr::new::<pyo3::exceptions::PyRuntimeError, _>(format!(
"Failed to create log scanner: {e:?}"
))
})?;
There was a problem hiding this comment.
we should use FlussError, this is a leftover.
Thank you for catching this 👍
| # Project specific columns by index (C++ parity) | ||
| print("\n1. Projection by index [0, 1] (id, name):") | ||
| scanner_index = await table.new_log_scanner_with_projection([0, 1]) | ||
| scanner_index.subscribe(None, None) | ||
| df_projected = scanner_index.to_pandas() | ||
| print(df_projected.head()) | ||
| print(f" Projected {df_projected.shape[1]} columns: {list(df_projected.columns)}") | ||
|
|
||
| # Project specific columns by name (Python-specific, more idiomatic!) | ||
| print("\n2. Projection by name ['name', 'score'] (Pythonic):") | ||
| scanner_names = await table.new_log_scanner_with_column_names(["name", "score"]) | ||
| scanner_names.subscribe(None, None) | ||
| df_named = scanner_names.to_pandas() | ||
| print(df_named.head()) | ||
| print(f" Projected {df_named.shape[1]} columns: {list(df_named.columns)}") |
There was a problem hiding this comment.
Should the polling part also be included (as with C++ example)?
There was a problem hiding this comment.
I'm not sure I get what you mean.
to_pandas() polls internally, but if we're talking about adding a separate polling API to Python bindings - we can add it.
Let's file an issue for it, as it's orthogonal to the column projection feature.
There was a problem hiding this comment.
Ah, I didn't realise that to panda polls. Thank you for the clarification
|
@leekeiabstraction Thank you for the review. Addressed the comments. PTAL 🙏 |
2 participants
Adds two new methods for column projection:
Both methods create LogScanner with specified columns only, improving performance by reducing data transfer and processing overhead.
Implementation leverages core scanner.project() and scanner.project_by_name() APIs.
Error handling validates column indices/names before creating scanner.
Closes #149