[CI] SetSingleNodeAllocateStepTests testPerformActionSomeShardsOnlyOnNewNodes failing

[ywangd]

CI Link

https://gradle-enterprise.elastic.co/s/yxahyrmbwdjag/tests/task/:x-pack:plugin:core:test/details/org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStepTests/testPerformActionSomeShardsOnlyOnNewNodes?top-execution=1

Repro line

./gradlew ":x-pack:plugin:core:test" --tests "org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStepTests.testPerformActionSomeShardsOnlyOnNewNodes" -Dtests.seed=2E2DDF7F31266EF7 -Dtests.locale=tn-Latn-ZA -Dtests.timezone=Jamaica -Druntime.java=25

Does it reproduce?

Yes

Applicable branches

main

Failure history

No response

Failure excerpt

It failed on my PR. But reproducible on the current main.

  2> java.lang.AssertionError:
    Expected: ("new_node_id_1" or "new_node_id_0" or "new_node_id_3" or "new_node_id_2")
         but: was "old_node_id_2"
        at __randomizedtesting.SeedInfo.seed([2E2DDF7F31266EF7:277C913A2B067531]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
        at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2718)
        at org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStepTests.assertSettingsRequestContainsValueFrom(SetSingleNodeAllocateStepTests.java:92)
        at org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStepTests.lambda$assertNodeSelected$7(SetSingleNodeAllocateStepTests.java:655)
        at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42)
        at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103)
        at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
        at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
        at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
        at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:56)
        at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptSuperCallable(MockMethodInterceptor.java:145)
        at org.elasticsearch.client.internal.IndicesAdminClient$MockitoMock$8HnlxiDg.updateSettings(Unknown Source)
        at org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStep.performAction(SetSingleNodeAllocateStep.java:118)
        at org.elasticsearch.xpack.core.ilm.AbstractStepTestCase.performActionAndWait(AbstractStepTestCase.java:83)
        at org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStepTests.assertNodeSelected(SetSingleNodeAllocateStepTests.java:666)
        at org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStepTests.testPerformActionSomeShardsOnlyOnNewNodes(SetSingleNodeAllocateStepTests.java:458)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

Thanks for opening this, @ywangd. Looking at the failure history of this whole class of the last year, it looks like the first failure dates back to Jan 31, 2025: https://es-delivery-stats.elastic.dev/app/r/s/nFY1Q. The failure message of the issue description occurs fairly rarely (only a handful of times over the past year), but there are other failures. The failures are also not unique to this one test (testPerformActionSomeShardsOnlyOnNewNodes); other tests in the class fail from time to time as well. All in all, definitely worth having a look.