QueryPhaseExecutionException: Query Failed [Failed to execute main query] Caused by: java.util.concurrent.ExecutionException: org.elasticsearch.tasks.TaskCancelledException: task cancelled

[fang-xing-esql]

Elasticsearch Version

main

Installed Plugins

No response

Java Version

bundled

OS Version

20.04.6 LTS (Focal Fossa)

Problem Description

org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:237)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:143)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:70)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:691)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:896)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:729)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1067)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	at org.elasticsearch.server@9.2.0/org.elasticsearch.ExceptionsHelper.convertToElastic(ExceptionsHelper.java:60)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.index.cache.bitset.BitsetFilterCache$QueryWrapperBitSetProducer.getBitSet(BitsetFilterCache.java:239)
	at org.apache.lucene.join@10.2.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:152)
	at org.apache.lucene.core@10.2.2/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:290)
	at org.apache.lucene.join@10.2.2/org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:145)
	at org.apache.lucene.core@10.2.2/org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:290)
	at org.apache.lucene.core@10.2.2/org.apache.lucene.search.Weight.bulkScorer(Weight.java:171)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:454)
	at org.apache.lucene.core@10.2.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at org.apache.lucene.core@10.2.2/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
	at org.apache.lucene.core@10.2.2/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
	... 6 more
	Suppressed: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	Suppressed: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	Suppressed: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	Suppressed: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	Suppressed: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	Suppressed: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	Suppressed: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	Suppressed: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
Caused by: java.util.concurrent.ExecutionException: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]
	at org.elasticsearch.server@9.2.0/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:443)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.index.cache.bitset.BitsetFilterCache.getAndLoadIfNotPresent(BitsetFilterCache.java:180)
	at org.elasticsearch.server@9.2.0/org.elasticsearch.index.cache.bitset.BitsetFilterCache$QueryWrapperBitSetProducer.getBitSet(BitsetFilterCache.java:237)
	... 18 more
Caused by: org.elasticsearch.tasks.TaskCancelledException: task cancelled [http channel [Netty4HttpChannel{localAddress=/100.64.52.76:9200, remoteAddress=/100.64.29.13:53472}] closed]

Steps to Reproduce

N/A

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[javanna]

Hey @fang-xing-esql , what's the issue here? You posted a stacktrace, can you also share some context on it? As far as I can tell, the search has been cancelled, and that is why the query phase on the shard gets interrupted with TaskCancelledException . What is the bug?

[iverase]

My two cents here:

The issue is in this line where we wrap the TaskCancelledException as an ElasticsearchException. That ends up being logged as a suppressed rest error which is wrong.

I thin we need to add logic here to handle with this case (and maybe CCB too).

[javanna]

Good point, the exception itself is fine, it is already a user error. The problem is that when it gets wrapped its error code may get lost. Thanks @iverase !

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)