Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, we enforce a strict limitation that prefix lookup columns must encompass all partition fields when querying partitioned tables. However, in numerous practical scenarios, users require the flexibility to perform prefix lookups across multiple partitions. A common use case is executing delta joins where join keys originate from different partitions.

Solution

• Concurrent Multi-Partition Prefix Lookup: Implement support for concurrent prefix lookups across multiple partitions, with automatic result merging.
• Partition Pruning with Predicate Pushdown in PrefixLookup: Enable partition pruning based on pushed-down predicates, including both JOIN conditions and WHERE conditions, to minimize unnecessary data scanning and improve query performance.
• Currently, to avoid duplicate work related to generic partition filter pushdown based on the predicate system (see [flink]Support partition pushdown for more filters in Flink connector #420), this issue will not implement pushdown of WHERE conditions into PrefixKeyLookuper. This functionality will be addressed after [flink]Support partition pushdown for more filters in Flink connector #420 is merged.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!