[Nightshift][Investigations] Rank investigation recommendations and blind spots by confidence

[mykolaharmash]

Closes https://github.com/elastic/nightshift-program/issues/1328

Depends on #289984.

Require investigation recommendations and blind spots to carry relevance confidence, rank them by that confidence, and retain only the three strongest items. Output that does not satisfy the current investigation state schema is ignored (explicitly ignoring old investigation results).

Notable changes

• Uses one strict investigation state schema that requires bounded confidence and stably sorts recommendation and blind-spot lists.
• Applies the same schema to live progress, persistence updates, and reads; legacy unscored arrays are omitted.
• Displays each recommendation's confidence as a percentage in the Nightshift flyout.

[kibanamachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 5eb04bd
• Build duration: 47 mins
• Storybooks Preview

Failed CI Steps

• Jest Tests #4
• Scout Lane #3 - stateful-classic / default

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
nightshift	110.5KB	110.4KB	-20.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
shared-packages	4.6MB	4.6MB	+135.0B

Unknown metric groups

total optimizer output size

id	before	after	diff
all	63.7MB	63.7MB	+115.0B

warm start memory

id	before	after	diff
post forced gc heap baseline	-	833397674	+833397674
post forced gc heap delta	-	-1815724	-1815724
post forced gc heap delta standard deviation	-	1319682	+1319682
post forced gc heap target	-	831581950	+831581950
tail heap delta	-	132173	+132173
total			+1664615755

Test Failures

• [job] [logs] Scout Lane #3 - stateful-classic / default / local-stateful-classic - Discover tabs - tab preview - should preserve content after refresh
• [job] [logs] Scout Lane #3 - stateful-classic / default / local-stateful-classic - Discover tabs - time range - should save different time ranges when the switch is on

History

• 💔 Build #498848 failed dc81aea
• 💔 Build #498832 failed ba99efb

[jasonrhodes]

Mostly a philosophical question at this point, but is a hard limit going to cause problems for LLM work? I am imagining an outage where 4+ services need to be restarted, the LLM splits them into one action per service -- but has to choose the 3 strongest, so leaves off a critical action. Hopefully we can account for this with other prompt instructions.

[jasonrhodes]

LGTM, let's merge and iterate after we see how it performs on some real data