ESQL: Type conflict resolution in unmapped-fields load#143693
Draft
GalLalouche wants to merge 108 commits intoelastic:mainfrom
Draft
ESQL: Type conflict resolution in unmapped-fields load#143693GalLalouche wants to merge 108 commits intoelastic:mainfrom
GalLalouche wants to merge 108 commits intoelastic:mainfrom
Conversation
PotentiallyUnmappedKeywordEsField (used for fields loaded from _source when unmapped_fields="load") had isAggregatable=true, which caused isPushableFieldAttribute to short-circuit past the SearchStats check and push filters down to Lucene. On shards where the field is not indexed, the Lucene query returns no results instead of letting the compute engine evaluate the filter on _source-loaded values. Guard isPushableFieldAttribute against PotentiallyUnmappedKeywordEsField so these fields are always filtered in the compute engine. Co-authored-by: Cursor <cursoragent@cursor.com>
The isPushableFieldAttribute fix (rejecting PotentiallyUnmappedKeywordEsField) already covers sort pushdown since PushTopNToSource uses the same method. This test verifies correct sort order when a field is mapped in one index but unmapped in another under unmapped_fields="load". Co-authored-by: Cursor <cursoragent@cursor.com>
This adds the remaining serialization tests. It makes a few production code changes: 1. Registers `TO_DATE_RANGE` for serialization. It was serializable and not registered. 2. Normalizes the serialization of `SCORE` and explains how it used to work. It was not broken but held together strangely. 3. Replaces implementing `SurrogateExpression` with `OnlySurrogateExpression` in a few more cases so we don't expect serialization to work. 4. Adds some package private accessors so we can refer to children by name in serialization tests.
Replace command parsing with regex that checks whole query for METADATA _index. Use static Pattern to avoid recompilation. Made-with: Cursor
Made-with: Cursor
Add tests covering cases where time-series aggregation behaves differently from non-TS when handling unmapped fields: unmapped-load.csv-spec: - statsImplicitLastOverTimeUnmappedTsIndex: max(max_over_time(event_log)) on unmapped metric with load - statsByUnmappedFieldTsIndex: STATS BY unmapped event_log on TS index (uses COUNT(events_received) since TS rejects COUNT(*)) - explicitLastOverTimeUnmappedTsIndex: explicit max_over_time on unmapped unmapped-nullify.csv-spec: - unmappedFieldNullifiedTsIndex: unmapped event_log nullified on TS index - rateOnUnmappedTsIndex: RATE(nonexistent_counter) with nullify - explicitLastOverTimeUnmappedNullifyTsIndex: max_over_time on nullified unmapped field Add k8s_unmapped index (mapping omits event_log, uses k8s.csv) to support these TS unmapped tests. Made-with: Cursor
GalLalouche
added
>feature
Team:Analytics
Collaborator
|
Hi @GalLalouche, I've created a changelog YAML for you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.