Skip to content

Esql dimension aware attributes #131463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Aug 18, 2025
Merged

Esql dimension aware attributes #131463

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
05a4561
abstract function
not-napoleon May 7, 2025
5f8e350
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon May 8, 2025
0c0b5bd
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon Jun 12, 2025
9d79b2f
stub in field attribute
not-napoleon Jun 17, 2025
2f9e7d2
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon Jul 2, 2025
c11fa12
begin wiring up the time series types
not-napoleon Jul 7, 2025
28a3aea
stub in abstract methods
not-napoleon Jul 7, 2025
001047d
explicit metadata type for some benchmarks
not-napoleon Jul 8, 2025
3397960
explicit metadata type
not-napoleon Jul 8, 2025
038f743
stuff
not-napoleon Jul 10, 2025
d726c75
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon Jul 16, 2025
37849ed
explicit default for a bunch of constructors
not-napoleon Jul 16, 2025
cf48733
fix the other constructor
not-napoleon Jul 16, 2025
f246efa
multi-type fields
not-napoleon Jul 16, 2025
3703533
text-field subclass
not-napoleon Jul 17, 2025
47df3b4
keyword fields
not-napoleon Jul 17, 2025
5250dce
date fields
not-napoleon Jul 17, 2025
c7dfe13
equals and hashcode
not-napoleon Jul 17, 2025
ec5dfd4
fix a couple of tests
not-napoleon Jul 17, 2025
2f638e7
Remove unnecessary check
not-napoleon Jul 18, 2025
1a16702
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon Jul 21, 2025
e7b7e7c
revert unexpected docs changes
not-napoleon Jul 22, 2025
1b1afe5
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon Jul 24, 2025
89c2054
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon Aug 14, 2025
3b4dc17
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon Aug 15, 2025
9ab5c0f
Merge branch 'main' into esql-dimension-aware-attributes
not-napoleon Aug 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ...marks/src/main/java/org/elasticsearch/benchmark/_nightly/esql/QueryPlanningBenchmark.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,8 @@ public void setup() {
var fields = 10_000;
var mapping = LinkedHashMap.<String, EsField>newLinkedHashMap(fields);
for (int i = 0; i < fields; i++) {
mapping.put("field" + i, new EsField("field-" + i, TEXT, emptyMap(), true));
// We're creating a standard index, so none of these fields should be marked as dimensions.
mapping.put("field" + i, new EsField("field-" + i, TEXT, emptyMap(), true, EsField.TimeSeriesFieldType.NONE));
}

var esIndex = new EsIndex("test", mapping, Map.of("test", IndexMode.STANDARD));
Expand Down
26 changes: 21 additions & 5 deletions benchmarks/src/main/java/org/elasticsearch/benchmark/compute/operator/EvalBenchmark.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ private static EvalOperator.ExpressionEvaluator evaluator(String operation) {
FieldAttribute timestamp = new FieldAttribute(
Source.EMPTY,
"timestamp",
new EsField("timestamp", DataType.DATETIME, Map.of(), true)
new EsField("timestamp", DataType.DATETIME, Map.of(), true, EsField.TimeSeriesFieldType.NONE)
);
yield EvalMapper.toEvaluator(
FOLD_CONTEXT,
Expand Down Expand Up @@ -321,19 +321,35 @@ private static EvalOperator.ExpressionEvaluator evaluator(String operation) {
}

private static FieldAttribute longField() {
return new FieldAttribute(Source.EMPTY, "long", new EsField("long", DataType.LONG, Map.of(), true));
return new FieldAttribute(
Source.EMPTY,
"long",
new EsField("long", DataType.LONG, Map.of(), true, EsField.TimeSeriesFieldType.NONE)
);
}

private static FieldAttribute doubleField() {
return new FieldAttribute(Source.EMPTY, "double", new EsField("double", DataType.DOUBLE, Map.of(), true));
return new FieldAttribute(
Source.EMPTY,
"double",
new EsField("double", DataType.DOUBLE, Map.of(), true, EsField.TimeSeriesFieldType.NONE)
);
}

private static FieldAttribute intField() {
return new FieldAttribute(Source.EMPTY, "int", new EsField("int", DataType.INTEGER, Map.of(), true));
return new FieldAttribute(
Source.EMPTY,
"int",
new EsField("int", DataType.INTEGER, Map.of(), true, EsField.TimeSeriesFieldType.NONE)
);
}

private static FieldAttribute keywordField() {
return new FieldAttribute(Source.EMPTY, "keyword", new EsField("keyword", DataType.KEYWORD, Map.of(), true));
return new FieldAttribute(
Source.EMPTY,
"keyword",
new EsField("keyword", DataType.KEYWORD, Map.of(), true, EsField.TimeSeriesFieldType.NONE)
);
}

private static Configuration configuration() {
Expand Down
3 changes: 0 additions & 3 deletions docs/reference/query-languages/esql/_snippets/operators/types/is_null.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,6 @@
| boolean | boolean |
| cartesian_point | boolean |
| cartesian_shape | boolean |
| counter_double | boolean |
| counter_integer | boolean |
| counter_long | boolean |
| date | boolean |
| date_nanos | boolean |
| double | boolean |
Expand Down
69 changes: 16 additions & 53 deletions docs/reference/query-languages/esql/kibana/definition/operators/is_null.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions ...lugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/Attribute.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,4 +149,9 @@ public static boolean dataTypeEquals(List<Attribute> left, List<Attribute> right
}
return true;
}

/**
* @return true if the attribute represents a TSDB dimension type
*/
public abstract boolean isDimension();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should actually be ternary, resp. return Boolean (capital B) so that attributes can return null to say "I don't know if I refer to a dimension" rather than claiming not to.

Otherwise 👍

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the behavior associated with the null version? The intention here is that we want to only allow filtering by dimensions, and enable automatic grouping by all dimensions. If a field returns null here, we would then need to decide if "null" behaved as "this is a dimension" or "this is not a dimension" for these cases.

Unless you're proposing that we fail (or null-with-warning) any query where there is ambiguity about the dimensions? I'm not sure I agree with that.

}
5 changes: 5 additions & 0 deletions .../esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/EmptyAttribute.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,11 @@ protected String label() {
return "e";
}

@Override
public boolean isDimension() {
return false;
}

@Override
public boolean resolved() {
return true;
Expand Down
5 changes: 5 additions & 0 deletions .../esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/FieldAttribute.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,11 @@ protected String label() {
return "f";
}

@Override
public boolean isDimension() {
return field.getTimeSeriesFieldType() == EsField.TimeSeriesFieldType.DIMENSION;
}

public EsField field() {
return field;
}
Expand Down
5 changes: 5 additions & 0 deletions ...ql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/MetadataAttribute.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,11 @@ protected String label() {
return "m";
}

@Override
public boolean isDimension() {
return false;
}

@Override
protected NodeInfo<? extends Expression> info() {
return NodeInfo.create(this, MetadataAttribute::new, name(), dataType(), nullable(), id(), synthetic(), searchable);
Expand Down
5 changes: 5 additions & 0 deletions ...l-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/ReferenceAttribute.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,9 @@ protected NodeInfo<ReferenceAttribute> info() {
protected String label() {
return "r";
}

@Override
public boolean isDimension() {
return false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: it might get tricky when dealing with renames; if you have a FROM idx | RENAME dimension_field AS foo, then the reference attribute for foo will claim not to be a dimension - so downstream plan nodes may need to resolve aliases to correctly determine if they're looking at a dimension or not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding from @kkrik-es is that, at least for the initial version, that's an acceptable restriction. I think ReferenceAttributes are used for the output of EVAL and similar commands, and at least for the initial version of this we do not want to treat those as dimensions.

}
}
5 changes: 5 additions & 0 deletions ...-core/src/main/java/org/elasticsearch/xpack/esql/core/expression/UnresolvedAttribute.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,11 @@ protected String label() {
return UNRESOLVED_PREFIX;
}

@Override
public boolean isDimension() {
return false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: maybe this should throw UOE, because we can't determine this on an unresolved attr.

}

@Override
public String nodeString() {
return toString();
Expand Down
4 changes: 2 additions & 2 deletions ...ck/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DateEsField.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@
*/
public class DateEsField extends EsField {

public static DateEsField dateEsField(String name, Map<String, EsField> properties, boolean hasDocValues) {
return new DateEsField(name, DataType.DATETIME, properties, hasDocValues, TimeSeriesFieldType.UNKNOWN);
public static DateEsField dateEsField(String name, Map<String, EsField> properties, boolean hasDocValues, TimeSeriesFieldType tsType) {
return new DateEsField(name, DataType.DATETIME, properties, hasDocValues, tsType);
}

private DateEsField(
Expand Down
Loading
Loading