Intermittent ExAws OIDC credential refresh error on Fly.io

[tofran]

Hello, I'm reposting the post I made on fly.io community but did not get any response there.

TLDR:
I'm using OIDC and I get this error time to time:

RuntimeError: Could't get credentials from auth adapter after 6 retries
  Value at 'webIdentityToken' failed to satisfy constraint: Member must have length greater than or equal to 4
GenServer ExAws.Config.AuthCache terminating

• OIDC seems properly configured (It works).
• It throws this error time-to-time regardless of traffic/activity.

Is it a flyio issue? Has anyone encountered this error before?
Any ideas on how to debug this?

Thank you in advance.

Logs

``` Last message: {:refresh_awscli_config, "default", 30000} line=2646 pid=<0.3070.0> registered_name=ExAws.Config.AuthCache file=gen_server.erl domain=otp mfa=:gen_server.error_info/8 (stdlib 6.2.2) proc_lib.erl:329: :proc_lib.init_p_do_apply/3 (stdlib 6.2.2) gen_server.erl:2433: :gen_server.handle_msg/6 (stdlib 6.2.2) gen_server.erl:2345: :gen_server.try_handle_info/3 (ex_aws 2.5.10) lib/ex_aws/config/auth_cache.ex:60: ExAws.Config.AuthCache.handle_info/2 (ex_aws 2.5.10) lib/ex_aws/config/auth_cache.ex:73: ExAws.Config.AuthCache.refresh_awscli_config/3 (ex_aws 2.5.10) lib/ex_aws/config/auth_cache.ex:87: ExAws.Config.AuthCache.attempt_credentials_refresh/5 ** (RuntimeError) Could't get credentials from auth adapter after 6 retries, last error was {:http_error, 400, %{code: "ValidationError", message: "1 validation error detected: Value at 'webIdentityToken' failed to satisfy constraint: Member must have length greater than or equal to 4", type: "Sender", detail: "", request_id: "c0e5da71-d37c-4c59-937b-4977e130a69d"}} 17:19:02.144 [error] GenServer ExAws.Config.AuthCache terminating ```

[bernardd]

Hi @tofran. Short answer is that that error message ( Value at 'webIdentityToken' failed to satisfy constraint: Member must have length greater than or equal to 4) is coming from the server, so either ex_aws is sending something invalid for some reason, or the server's rejecting a valid request. I haven't really had any experience with fly.io, so I'm not really across the details of what might be going on. In terms of debugging, my suggestion would be to log the message being sent at ExAws.STS.AuthCache.AssumeRoleWEebIdentityAdapter.get_security_redentials/2 around line 28. If the token has a length >= 4, then it seems like the server is getting confused somehow. If the token is empty or too short or whatever, it's possibly ExAws not sending the right thing.
I did just notice once thing, though - ExAws sends the value with the name WebIdentityToken whereas that error message implies it's expecting webIdentityToken. Could be that the lowercase first letter is the problem. Might be worth changing it and seeing if that helps.

[tofran]

Thank you for the reply, since this is not constant I will add some logging and see if I create more visibility