`failed to refresh cached credentials` when using aws provider to connect to object storage

[mardonner]

Hey,

I'm having issues using the aws provider to connect to the object storage. Don't know if bug or user error.
I'm trying something like in the example.
I'm getting this error during tofu init:

╷
│ Error: No valid credential sources found
│ 
│   with provider["registry.opentofu.org/hashicorp/aws"],
│   on providers.tf line 28, in provider "aws":
│   28: provider "aws" {
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, no EC2 IMDS role found,
│ operation error ec2imds: GetMetadata, failed to get API token, operation
│ error ec2imds: getToken, http response error StatusCode: 400, request to
│ EC2 IMDS failed
│ 
╵

Here is my opentofu code snippet:

terraform {
  required_version = "~> 1.10.0"
  required_providers {
    # for creating STACKIT resources
    stackit = {
      source  = "stackitcloud/stackit"
      version = "~> 0.58.0"
    # for writing the ACL policy on the STACKIT Object Storage Bucket
    aws = {
      source  = "hashicorp/aws"
      version = "6.4.0"
    }
  }
}
provider "stackit" {
  default_region        = var.stackit_default_region
  enable_beta_resources = true
  # authentication via environment variable
}

provider "aws" {
  region                      = var.stackit_default_region
  skip_credentials_validation = true
  skip_region_validation      = true
  skip_requesting_account_id  = true
  # skip_metadata_api_check = true
  access_key                  = stackit_objectstorage_credential.usage_credential.access_key
  secret_key                  = stackit_objectstorage_credential.usage_credential.secret_access_key
  endpoints {
    s3 = "https://object.storage.${var.stackit_default_region}.onstackit.cloud"
  }
}

resource "stackit_objectstorage_bucket" "usage-bucket" {
  project_id = var.stackit_project_id
  name       = local.obj_str_bucket_name
}

resource "stackit_objectstorage_credentials_group" "usage-group" {
  project_id = var.stackit_project_id
  name       = local.obj_str_creds_grp_name
}

resource "stackit_objectstorage_credential" "usage_credential" {
  project_id           = var.stackit_project_id
  credentials_group_id = stackit_objectstorage_credentials_group.usage-group.credentials_group_id
  lifecycle {
    create_before_destroy = true
    replace_triggered_by  = [null_resource.credential_rotation_trigger]
  }
}

# ACLs on Object Storage need to be set up with aws provider
resource "aws_s3_bucket_policy" "acl_policy" {
  bucket = stackit_objectstorage_bucket.usage-bucket.name
  policy = <<EOF
  {
    "Statement":[
      {
        "Sid":"Restrict-IP-Range",
        "Effect":"Deny",
        "Principal":"*",
        "Action": [
          "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::${stackit_objectstorage_bucket.usage-bucket.name}/*",
            "arn:aws:s3:::${stackit_objectstorage_bucket.usage-bucket.name}"
        ],
        "Condition": {
          "NotIpAddress": {
            "aws:SourceIp": ${local.acls}
          }
        }
      }
    ]
  }
  EOF
}

When setting skip_metadata_api_check = true, I get:

╷
│ Error: No valid credential sources found
│ 
│   with provider["registry.opentofu.org/hashicorp/aws"],
│   on providers.tf line 28, in provider "aws":
│   28: provider "aws" {
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, no EC2 IMDS role found,
│ operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via
│ client option, or "AWS_EC2_METADATA_DISABLED" environment variable
│ 
╵

Currently don't see what I'm doing wrong here. Any advice?

Also, just noticed that the object storage aws provider example has a formatting issue:

[Vijayanagendrakadimi]

You're telling Terraform not to attempt using EC2 Instance Metadata (IMDS), but it still can't use your dynamic credentials because they are not known yet.

The aws provider is only used after the credentials are created, so no IMDS fallback or failure happens.

You need to alias the AWS provider and use it only after credentials are available.
Create the AWS provider with alias:

provider "aws" {
alias = "stackit"
region = var.stackit_default_region
skip_credentials_validation = true
skip_region_validation = true
skip_requesting_account_id = true
access_key = stackit_objectstorage_credential.usage_credential.access_key
secret_key = stackit_objectstorage_credential.usage_credential.secret_access_key
endpoints {
s3 = "https://object.storage.${var.stackit_default_region}.onstackit.cloud"
}
}

Reference the aliased provider in the ACL policy:

resource "aws_s3_bucket_policy" "acl_policy" {
provider = aws.stackit
bucket = stackit_objectstorage_bucket.usage-bucket.name
policy = <<EOF
{
"Statement":[
{
"Sid":"Restrict-IP-Range",
"Effect":"Deny",
"Principal":"",
"Action": [
"s3:"
],
"Resource": [
"arn:aws:s3:::${stackit_objectstorage_bucket.usage-bucket.name}/*",
"arn:aws:s3:::${stackit_objectstorage_bucket.usage-bucket.name}"
],
"Condition": {
"NotIpAddress": {
"aws:SourceIp": ${local.acls}
}
}
}
]
}
EOF
}

Run it

[mardonner]

Run it

Getting the exact same errors. Also, why would I need the workaround with the provider alias? The vault example doesn't need this and it works as expected. Provider configurations are evaluated lazily. Creating a secrets manager and secrets in the same apply is possible.