ESQL: Enable new data types with created version #136327

nik9000 · 2025-10-09T20:45:52Z

Enables dense_vector and aggregate_metric_double if all nodes in all of the target clusters support it.

Relates to #135193

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to elastic#135193

github-actions · 2025-10-09T20:47:28Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

nik9000 · 2025-10-21T22:56:06Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

-        if (Build.current().isSnapshot()) {
-            // We only test behavior in release builds. Snapshot builds will have data types enabled that are still under construction.
-            return List.of();
-        }


Moved this.

elasticsearchmachine · 2025-10-21T22:58:14Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-22T12:57:44Z

Hi @nik9000, I've created a changelog YAML for you.

alex-spies

Nice, thanks @nik9000 ! I have only minor comments, and a suggestion for a follow-up (more tests!) that we can tackle separately.

alex-spies · 2025-10-22T13:21:05Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

    }

+    @Before
+    public void onlyOnSnapshot() throws IOException {


The name seems to suggest the opposite of the actual method body?

Suggested change

public void onlyOnSnapshot() throws IOException {

public void onlyOnProductionBuilds() throws IOException {

alex-spies · 2025-10-22T15:19:04Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

+     * Tests fetching {@code dense_vector} if possible. Uses the {@code dense_vector_agg_metric_double_if_fns}
+     * work around if required.
+     */
    public final void testFetchDenseVector() throws IOException {


We could use a similar test for aggregate metric double, no?

alex-spies · 2025-10-22T15:22:58Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

    }

+    @Before
+    public void onlyOnSnapshot() throws IOException {


Interesting! I thought that running tests in release mode will use release builds of all bwc versions as well. Is that not the case?

Can we add a comment to explain this? Also, should we double check our release tests whether they always should use release builds of bwc versions?

I'll add a comment, yeah. If you run:

./gradlew -p x-pack/plugin/esql/qa/server/multi-clusters/ clean v9.1.5#newToOld -Dbuild.snapshot=false -Dlicense.key="x-pack/license-tools/src/test/resources/public.key" -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.class='*All*'

while 9.1.5 is the tip of the 9.1 branch then we'll compile 9.1 and run it as a snapshot.

I'll ask around if this is intended.

I'll ask around if this is intended.

Well, not "intended", but not something we're likely to fix right now.

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

alex-spies · 2025-10-22T17:40:29Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

+     *                               production but tests need more control
+     * @param useAggregateMetricDoubleWhenNotSupported does the query itself force us to use {@code aggregate_metric_double} fields
+     *                                                 even if the remotes don't report that they support the type? This exists because
+     *                                                 some remotes <strong>do</strong> support {@code aggregate_metric_double} without


We can be more precise here: the support is precisely on 9.2.0; we can link to the data type's supported transport version so that hackers can use git blame to confirm that.

Being precise allows us to refactor and clean this in 9.5, when we don't have to expect a mixed cluster state with 9.2.0 (or only during a rolling upgrade)

alex-spies · 2025-10-22T17:40:38Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

+     *                                                 reporting that they do. And, for a while, we used the query itself to opt into
+     *                                                 reading these fields.
+     * @param useDenseVectorWhenNotSupported does the query itself force us to use {@code dense_vector} fields even if the remotes don't
+     *                                       report that they support the type? This exists because some remotes <strong>do</strong>


alex-spies · 2025-10-22T17:44:04Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

+        boolean useDenseVectorWhenNotSupported
+    ) {
+        /**
+         * The {@link #minTransportVersion}, but if any remote didn't tell us the version we assume


Can we say that, if a remote (cluster) didn't tell us the min version, it's at most on 9.1, because if all nodes where on 9.2.0+, the min transport version would be part of the field caps response?

I could add that to the comment, but this is lowest version it could possibly be, not the highest. At least, that's what I think it should be.

What you say is correct! I meant that it'd just be helpful to understand that the min transport version was added to the FC response in 9.2.0. Which means that if we didn't receive it, there is at least one cluster with at least one node that is on a previous version.

Cool! I've pushed some updated comments.

alex-spies · 2025-10-22T17:44:37Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

+        boolean typeSupported = type.supportedVersion()
+            .supports(fieldsInfo.effectiveMinTransportVersion(), fieldsInfo.currentBuildIsSnapshot)
+            || switch (type) {
+                case AGGREGATE_METRIC_DOUBLE -> fieldsInfo.useAggregateMetricDoubleWhenNotSupported;
+                case DENSE_VECTOR -> fieldsInfo.useDenseVectorWhenNotSupported;
+                default -> false;
+            };
+        if (false == typeSupported) {
+            type = UNSUPPORTED;
+        }


Yeah, this is the main change and looks right!

alex-spies · 2025-10-22T17:46:45Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

This change effectively adds support to agg metric double and dense vector fields in lookup indices and enrich policies.

Let's add a test case for using all kinds of types as lookup/enrich fields, right? (Not match fields, we have separate tests for that.) In a follow-up that is, this should be fine to merge without.

nik9000 · 2025-10-23T15:12:20Z

Follow ups we need:

Let's add a test case for using all kinds of types as lookup/enrich fields, right? (Not match fields, we have separate tests for that.)
We could use a similar test for aggregate metric double, no? ESQL: Handle release of 9.2 in test #137070
Integrate with Include min transport version into FC response #136887
Create a pragma to enable under construction types (sounds kind of difficult, but probably worth it)

…able

elasticsearchmachine · 2025-10-23T20:24:27Z

💔 Backport failed

Status	Branch	Result
❌	9.2	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 136327

I'd made a mistake in elastic#136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from elastic#136327.

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to elastic#135193

nik9000 · 2025-10-23T22:15:22Z

Backport: #137073

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to #135193

ESQL: Enable new data types with created version

57b63c2

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to elastic#135193

nik9000 added auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v9.2.1 v9.3.0 labels Oct 9, 2025

nik9000 added the test-release Trigger CI checks against release build label Oct 10, 2025

nik9000 added 5 commits October 10, 2025 09:51

Fixup

3b5c4e2

Merge branch 'main' into version_enable

ac6efb9

Merge branch 'main' into version_enable

ad8b9b9

Merge

fb665c1

Rename and fix check

8b22742

nik9000 commented Oct 21, 2025

View reviewed changes

nik9000 marked this pull request as ready for review October 21, 2025 22:57

Merge branch 'main' into version_enable

a7d19df

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Oct 21, 2025

nik9000 requested a review from alex-spies October 21, 2025 22:58

Merge branch 'main' into version_enable

5016b73

nik9000 added the >enhancement label Oct 22, 2025

Update docs/changelog/136327.yaml

8e2d031

alex-spies approved these changes Oct 22, 2025

View reviewed changes

nik9000 added 5 commits October 23, 2025 11:13

Merge branch 'main' into version_enable

47590a1

Comments

0527749

Merge remote-tracking branch 'nik9000/version_enable' into version_en…

e9b20c8

…able

More rename

dd895a1

Merge branch 'main' into version_enable

b86a096

nik9000 merged commit b311383 into elastic:main Oct 23, 2025
35 checks passed

elasticsearchmachine added the backport pending label Oct 23, 2025

nik9000 mentioned this pull request Oct 23, 2025

ESQL: Handle release of 9.2 in test #137070

Open

nik9000 added a commit that referenced this pull request Oct 24, 2025

ESQL: Enable new data types with created version (#136327) (#137073)

247d9dc

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to #135193

	public void onlyOnSnapshot() throws IOException {
	public void onlyOnProductionBuilds() throws IOException {

Uh oh!

ESQL: Enable new data types with created version #136327

ESQL: Enable new data types with created version #136327

Uh oh!

Conversation

nik9000 commented Oct 9, 2025

Uh oh!

github-actions bot commented Oct 9, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 21, 2025

Uh oh!

elasticsearchmachine commented Oct 22, 2025

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 23, 2025

💔 Backport failed

Uh oh!

nik9000 commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nik9000 commented Oct 23, 2025 •

edited

Loading