[Enhancement] Fulltext index BloomFilter pushdown pre-filter#23833
Open
ck89119 wants to merge 5 commits intomatrixorigin:mainfrom
Open
[Enhancement] Fulltext index BloomFilter pushdown pre-filter#23833ck89119 wants to merge 5 commits intomatrixorigin:mainfrom
ck89119 wants to merge 5 commits intomatrixorigin:mainfrom
Conversation
…rigin#23832) Implement BloomFilter pushdown for fulltext index tables, similar to the existing IVF index pushdown. When a fulltext search query has additional filter predicates (e.g. WHERE match(...) against(...) AND category = 'tech'), the filtered PKs are collected into a BloomFilter and pushed down to the fulltext index table reader to skip irrelevant rows early. Key changes: - plan: applyJoinFullTextIndices builds RuntimeFilterBuildList with BF - compile: scope.go/remoterun.go pass BF from context to FilterHint - reader: tryUpdateColumns uses doc_id column for BF filtering on fulltext tables (independent of pkPos since internal SQL doesn't include __mo_fake_pk_col) - pk_filter_mem: NewMemPKFilter sets BFSeqNum using doc_id column for fulltext tables (fallback when __mo_index_pri_col is absent) - build_ddl: fix fulltext index table relkind from SystemIndexRel to FullTextIndex_TblType so TableType check matches at runtime Approved by: @xxx Ref matrixorigin#23832
Contributor
|
Feng design is to only use post-filtering and no bloomfilter. Keep callling Call() to get more rows until it reach the limit if there is/are filters in SQL. |
cpegeric
requested changes
Mar 10, 2026
…bloom filter pushdown (default off)
cpegeric
approved these changes
Mar 13, 2026
heni02
approved these changes
Mar 13, 2026
aunjgr
approved these changes
Mar 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What type of PR is this?
Which issue(s) this PR fixes:
issue #23832
What this PR does / why we need it:
Implement BloomFilter pushdown for fulltext index tables, similar to the existing IVF index pushdown.
When a fulltext search query has additional filter predicates (e.g.
WHERE match(...) against(...) AND category = 'tech'), the filtered PKs are collected into a BloomFilter and pushed down to the fulltext index table reader to skip irrelevant rows early.current
applyJoinFullTextIndices's query plan:When there is a filter, the doc_id returned by fulltext_index_scan can be massively selected for filters, resulting in wastes significant I/O.
optimized query plan(with filter)
Key changes:
plan (
apply_indices_fulltext.go):applyJoinFullTextIndicesbuildsRuntimeFilterBuildListwith BloomFilter for the inner join between fulltext results and the source table.compile (
scope.go,remoterun.go): Pass BloomFilter from context toFilterHintwhen the target table is a fulltext index table (TableType == "fulltext").reader (
reader.go):tryUpdateColumnsusesdoc_idcolumn for BF filtering on fulltext tables, independent ofpkPossince internal SQL does not include__mo_fake_pk_col.pk_filter_mem (
pk_filter_mem.go):NewMemPKFiltersetsBFSeqNumusingdoc_idcolumn for fulltext tables as fallback when__mo_index_pri_colis absent.build_ddl (
build_ddl.go): Fix fulltext index tablerelkindfromSystemIndexRel("i") toFullTextIndex_TblType("fulltext") so theTableTypecheck matches at runtime.Bug fixes included:
buildFullTextIndexTablewrote"i"tomo_tables.relkindinstead of"fulltext", causing BF context propagation to be skipped.if pkPos != -1block, but fulltext internal SQL never includes the PK column.NewMemPKFilteronly looked for__mo_index_pri_colto setBFSeqNum; fulltext tables usedoc_idinstead.Tests:
NewMemPKFilterwith fulltext table BFSeqNumtryUpdateColumnswith fulltext BF filteringtest/distributed/cases/fulltext/fulltext_pushdown.sqlwith 7 scenarios