[Hack] Change EKS to use Bottlerocket OS

[rhr323]

Temporary change to test ECK on EKS+Bottlerocket OS.

When amiFamily: Bottlerocket is set, ami setting isn't supported:

eksctl-io/eksctl#4998 (comment)

You're using the auto AMI resolution method which we do not support for Bottlerocket. The error message here could've been more descriptive though. Please remove the ami: auto field from your nodegroup config and it'll work.

Rel: https://github.com/elastic/sdh-control-plane/issues/10165

[prodsecmachine]

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

[rhr323]

buildkite test this -f p=eks -m s=8.19.2

[rhr323]

=> https://buildkite.com/elastic/cloud-on-k8s-operator/builds/11434#0198c6d6-b9fa-41fd-bd84-374a6c5052cd

EKS cluster creation fails with:

Error: unable to determine AMI to use: error getting AMI from SSM Parameter Store: operation error SSM: GetParameter, https response error StatusCode: 400, RequestID: 76bd2007-4622-4d24-a763-ae2fc9955f27, api error AccessDeniedException: User: arn:aws:iam::444732909647:user/services/eks_devops_ci is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:ap-northeast-3::parameter/aws/service/bottlerocket/aws-k8s-1.33/x86_64/latest/image_id because no identity-based policy allows the ssm:GetParameter action. please verify that AMI Family is supported

The IAM role/policy is missing the ssm:GetParameter permission, it seems:

https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id-bottlerocket.html

For using ami: ..., get the latest Bottlerocket AMI with this command:

aws ssm get-parameter --name /aws/service/bottlerocket/aws-k8s-1.33/x86_64/latest/image_id --region ap-northeast-3 --query "Parameter.Value" --output text

ami-0f3220830bd022670

[rhr323]

buildkite test this -f p=eks -m s=8.19.2

[rhr323]

=> https://buildkite.com/elastic/cloud-on-k8s-operator/builds/11468

Error: test run failed: Test job timeout exceeded (15h0m0s)
--
  | exit status 1
  | make: *** [Makefile:439: e2e-run] Error 1

Realized we need the #8795 for EKS to work.

[rhr323]

buildkite test this -f p=eks t=TestSmoke

[rhr323]

=> https://buildkite.com/elastic/cloud-on-k8s-operator/builds/11509

✅ all tests passed

[rhr323]

buildkite test this -f p=eks -m s=8.19.2

[rhr323]

=> https://buildkite.com/elastic/cloud-on-k8s-operator/builds/11520

✅ all tests passed