Migrate x-pack-deprecation REST tests

[ldematte]

This PR migrates legacy rest tests in the x-pack deprecation module.

I tried to use the INTEG_TEST distribution and add just the basic modules needed (ml, data-streams, deprecation) but this way the indices are not persisted and the tests fails. Need more investigation to identify the set of required modules.

Also, I preferred not to move and "flatten" the project here, but maintain a separate "qa" sub-project, as the IT tests use two different test plugins, specific to this module.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mark-vieira]

LGTM but one comment:

Also, I preferred not to move and "flatten" the project here, but maintain a separate "qa" sub-project, as the IT tests use two different test plugins, specific to this module.

Could we at least combine to two separate QA projects into one? I don't see any reason to keep them separate.

[rjernst]

as the IT tests use two different test plugins

FWIW this is a big advantage of the new IT tests, that we can control plugins for the cluster per test suite. Moving the qa tests into one sourceset doesn't affect that, but does make the entire set of tests more discoverable together.

[ldematte]

Could we at least combine to two separate QA projects into one? I don't see any reason to keep them separate.

Oh yes, definitely, I can do that.

FWIW this is a big advantage of the new IT tests, that we can control plugins for the cluster per test suite. Moving the qa tests into one sourceset doesn't affect that, but does make the entire set of tests more discoverable together.

What I cannot (or don't want) to do is to remove the QA sub-project altogether: keep me honest here and correct me if I'm wrong, but if I remove it and move everything under :x-pack:deprecation I have to find a new "home" for the plugins source. And I'm reluctant to do that in this case, because the plugins are very specific to these tests.

[ldematte]

Could we at least combine to two separate QA projects into one? I don't see any reason to keep them separate.

Merged them. @mark-vieira still LG to you?

[mark-vieira]

I'm pretty sure the module you need here is x-pack-stack, since it's what registers the logs-* index templates.

[ldematte]

I'm pretty sure the module you need here is x-pack-stack, since it's what registers the logs-* index templates.

Tried to add it, still failing :/

[mark-vieira]

I'm pretty sure the module you need here is x-pack-stack, since it's what registers the logs-* index templates.

Tried to add it, still failing :/

Same error or a different one? It's often a bit of a whack-a-mole with these things.

[ldematte]

Exact same indexnotfoud exception: https://gradle-enterprise.elastic.co/s/ef6jzgb7p337w

[mark-vieira]

We should be installing this as a module not a plugin. Also, it should already be included in the default distro, so installing it as a plugin here is unnecessary. In fact, I'm surprised this even works and doesn't result in an error since we basically have this thing loaded twice.

[mark-vieira]

Exact same indexnotfoud exception: https://gradle-enterprise.elastic.co/s/ef6jzgb7p337w

Hmm. Yeah, there's something registering that index. I would expect it to be the deprecation plugin itself. Perhaps the test is implicitly relying on something to trigger a deprecation and therefore create the index? I'm not sure.

[ldematte]

After a lot of trial and error, I narrowed down the plugin (module) set:

clusterModules project(':modules:ingest-common')
  clusterModules project(':modules:mapper-extras')
  clusterModules project(':modules:data-streams')
  clusterModules project(xpackModule('stack'))
  clusterModules project(xpackModule('deprecation'))
  clusterModules project(xpackModule('ilm'))
  clusterModules project(xpackModule('ml'))
  clusterModules project(xpackModule('mapper-constant-keyword'))
  clusterModules project(xpackModule('wildcard'))
  clusterModules project(xpackModule('transform'))

Not trivial, but better than using the default distribution I think.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
✅	9.1
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 131444