VAULT-30877: Repopulate AWS static creds queue in initialize

[miagilepner]

Description

Closes #21935

Use the initialize function to re-populate the rotation queue on an active node. Store the credential's expiration value in storage and update it when it changes.

TODO only if you're a HashiCorp employee

• Backport Labels: If this PR is in the ENT repo and needs to be backported, backport
  to N, N-1, and N-2, using the backport/ent/x.x.x+ent labels. If this PR is in the CE repo, you should only backport to N, using the backport/x.x.x label, not the enterprise labels.
  • If this fixes a critical security vulnerability or severity 1 bug, it will also need to be backported to the current LTS versions of Vault. To ensure this, use all available enterprise labels.
• ENT Breakage: If this PR either 1) removes a public function OR 2) changes the signature
  of a public function, even if that change is in a CE file, double check that
  applying the patch for this PR to the ENT repo and running tests doesn't
  break any tests. Sometimes ENT only tests rely on public functions in CE
  files.
• Jira: If this change has an associated Jira, it's referenced either
  in the PR description, commit message, or branch name.
• RFC: If this change has an associated RFC, please link it in the description.
• ENT PR: If this change has an associated ENT PR, please link it in the
  description. Also, make sure the changelog is in this PR, not in your ENT PR.

[github-actions]

CI Results:
All Go tests succeeded! ✅

[github-actions]

Build Results:
All builds succeeded! ✅

[fairclothjm]

LGTM! Thanks @miagilepner for picking this up!

[fairclothjm]

nit: Is there a way to make this error point to the test file? Instead of the testhelpers.go file:

    testhelpers.go:726: did not complete before deadline, err: secrets haven't been rotated

[miagilepner]

That's a good idea! I added a t.Helper() to the testhelpers function, so now we see:

    rotation_test.go:554: did not complete before deadline, err: secrets haven't been rotated

[fairclothjm]

I was testing this today and after stopping and restarting Vault (with vcm) I am getting an error:

2024-10-30T16:38:50.504-0500 [ERROR] secrets.aws.aws_25fb669c: failed to create credential, re-queueing:
  error=
  | iam user didn't exist, or username/userid didn't match: unable to validate username "[email protected]": NoCredentialProviders: no valid providers in chain. Deprecated.
  | \tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors

2024-10-30T16:38:50.504-0500 [ERROR] rollback: error rolling back: path=aws/
  error=
  | 1 error occurred:
  | \t* error(s) occurred while rotating expired static credentials: 1 error occurred:
  | \t* iam user didn't exist, or username/userid didn't match: unable to validate username "[email protected]": NoCredentialProviders: no valid providers in chain. Deprecated.
  | \tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors

I am not sure if this is just my setup though.

[miagilepner]

@fairclothjm interesting, how are you setting this up? I tested it now with vcm using:

$ vault write aws/config/root \
  access_key=<redacted> \
  secret_key=<redacted>     # this is an IAM user with username [email protected] 

$ vault write aws/static-roles/test-role \
  [email protected] \
  rotation_period=1m        # this is a separate IAM user 

I didn't see any errors in the server logs from the rollbacks.

[fairclothjm]

@miagilepner Thanks, I was using the same user for the vault admin and static user. 😅 Now I am working through a policy issue with my vault admin user.

[fairclothjm]

@miagilepner I got it working. Apparently the permission boundary I am using requires a certain naming convention to function properly.