Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Vault entity aliases #12449

Merged
merged 8 commits into from
Apr 5, 2022
Merged

Support Vault entity aliases #12449

merged 8 commits into from
Apr 5, 2022

Conversation

lgfa29
Copy link
Contributor

@lgfa29 lgfa29 commented Apr 4, 2022

This PR adds support for deriving task tokens using an entity alias. This allows operators to better control the identity of the dynamic tokens that Nomad generates, which provide better control on billing and audit log parsing.

The entity alias can be set in the job and also a default value in the server configuration. If the server has a default alias, it will be used for tasks that don't define one.

In order to facilitate development, this PR starts by refactoring some of the Vault client internals. It may be easier to review this per commit.

To test this, you can use this script that will spin-up a Vault and Nomad agent preconfigured to support entity aliases.

Sample script run
$ ./demo.sh
==> Starting Vault...
Success! Uploaded policy: nomad-server
Success! Data written to: auth/token/roles/nomad-cluster
Success! Uploaded policy: app
Success! Data written to: auth/token/roles/nomad-user
Success! Data written to: auth/token/roles/nomad-user-not-allowed

You can use the following tokens to test job submission in different
scenarios:
    Token allowed:      s.cj1a6S4sOkvQOSy4cBBLQocu
    Token not allowed:  s.wjAie9VT0oWrrqPt4B1maty4
    Token without role: s.6UmUiXMNVMzIDJbJvjG6PjRs

==> Starting Nomad...

You can now run these two jobs, picking a token from the list above:
    Job that only runs if the token allows:
         $ VAULT_TOKEN=<TOKEN> nomad run demo/example.nomad

    Job that will never run because it uses an entity alias that the Nomad
    server can't access:
        $ VAULT_TOKEN=root nomad run demo/example-not-allowed.nomad

The first job has two tasks, but only one of them uses entity alias. You
can check the tokens information by running:
    $ ./demo.sh token-info

$ VAULT_TOKEN=s.cj1a6S4sOkvQOSy4cBBLQocu nomad run demo/example.nomad
==> 2022-04-04T14:16:53-04:00: Monitoring evaluation "289f0db0"
    2022-04-04T14:16:53-04:00: Evaluation triggered by job "example"
    2022-04-04T14:16:53-04:00: Allocation "91bfc88f" created: node "764ff4e9", group "app"
==> 2022-04-04T14:16:54-04:00: Monitoring evaluation "289f0db0"
    2022-04-04T14:16:54-04:00: Evaluation within deployment: "0aa0cb7a"
    2022-04-04T14:16:54-04:00: Allocation "91bfc88f" status changed: "pending" -> "running" (Tasks are running)
    2022-04-04T14:16:54-04:00: Evaluation status changed: "pending" -> "complete"
==> 2022-04-04T14:16:54-04:00: Evaluation "289f0db0" finished with status "complete"
==> 2022-04-04T14:16:54-04:00: Monitoring deployment "0aa0cb7a"
  ✓ Deployment "0aa0cb7a" successful

    2022-04-04T14:17:05-04:00
    ID          = 0aa0cb7a
    Job ID      = example
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    app         1        1       1        0          2022-04-04T14:27:03-04:00

$ ./demo.sh token-info
==> Token with entity
Key                  Value
---                  -----
accessor             uBPEG7WDciZxPXVKTWqYGTGY
creation_time        1649096213
creation_ttl         72h
display_name         token-91bfc88f-9c23-cf7b-52db-00334bf4be40-with-entity
entity_id            f9ac27cf-1135-435e-2401-6d617d0803d2
expire_time          2022-04-07T14:16:53.132469-04:00
explicit_max_ttl     0s
id                   s.ji1RGPmyHPVTmfY0757khESE
issue_time           2022-04-04T14:16:53.12894-04:00
last_renewal         2022-04-04T14:16:53.132469-04:00
last_renewal_time    1649096213
meta                 map[AllocationID:91bfc88f-9c23-cf7b-52db-00334bf4be40 JobID:example Namespace: NodeID:764ff4e9-fe78-0789-403b-967c22f241a4 Task:with_entity TaskGroup:app]
num_uses             0
orphan               true
path                 auth/token/create/nomad-cluster
policies             [app default]
renewable            true
role                 nomad-cluster
ttl                  71h59m41s
type                 service

==> Token without entity
Key                  Value
---                  -----
accessor             AGpy7Dz7O0gM67sIQbRNYdK4
creation_time        1649096213
creation_ttl         72h
display_name         token-91bfc88f-9c23-cf7b-52db-00334bf4be40-without-entity
entity_id            n/a
expire_time          2022-04-07T14:16:53.131276-04:00
explicit_max_ttl     0s
id                   s.5yLAUMknIAz62nlBjQoWYecg
issue_time           2022-04-04T14:16:53.124201-04:00
last_renewal         2022-04-04T14:16:53.131276-04:00
last_renewal_time    1649096213
meta                 map[AllocationID:91bfc88f-9c23-cf7b-52db-00334bf4be40 JobID:example Namespace: NodeID:764ff4e9-fe78-0789-403b-967c22f241a4 Task:without_entity TaskGroup:app]
num_uses             0
orphan               true
path                 auth/token/create/nomad-cluster
policies             [app default]
renewable            true
role                 nomad-cluster
ttl                  71h59m41s
type                 service

Notice how only one token as entity_id. Uncommenting line 218 of the script would use a default entity alias for the second token.

lgfa29 added 5 commits April 4, 2022 14:22
Move some common Vault API data struct decoding out of the Vault client
so it can be reused in other situations.

Make Vault job validation its own function so it's easier to expand it.

Rename the `Job.VaultPolicies` method to just `Job.Vault` since it
returns the full Vault block, not just their policies.

Set `ChangeMode` on `Vault.Canonicalize`.

Add some missing tests.
Allows specifying an entity alias that will be used by Nomad when
deriving the task Vault token.

An entity alias assigns an indentity to a token, allowing better control
and management of Vault clients since all tokens with the same indentity
alias will now be considered the same client. This helps track Nomad
activity in Vault's audit logs and better control over Vault billing.
Add support for a new Nomad server configuration to define a default
entity alias to be used when deriving Vault tokens. This default value
will be used if the task doesn't have an entity alias defined.
Copy link
Contributor

@mikenomitch mikenomitch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 from me on the UX!

Copy link
Contributor

@shoenig shoenig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great @lgfa29! Just the usual suggestions ~

}
expect := []string{"foo", "bar", "baz"}
got := SetToSliceString(set)
require.ElementsMatch(t, expect, got)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

neato, was gonna comment about iteration order but this gets around that!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup! It's a handy function, the problem is that I keep forgetting how it's called 😅

}
}
}
if err := j.validateVault(args.Job); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice refactoring! in the long term I'd like to get these all into the admissions controller plumbing

func (j *Job) admissionControllers(job *structs.Job) (out *structs.Job, warnings []error, err error) {

#7020

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Since there are a few refactorings going on already, I may as well do this now 🙂

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shoenig moved to admission controller in this commit: fb6a65f

I skipped unit tests because the Job registration tests were already covering them in general. But we may want to move them in the future.

}
}

// Check entity aliases.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the entity alias validation, can we move this chunk into a helper method? similar to multiVaultNamespaceValidation above.

// - the token used to submit the job must be allowed to use the default
// entity alias
// - except if all Vault blocks in the job define an alias, since in this
// case the server alias would not be used.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is implied from the summary above, but an explicit comment here saying:

"// Assign the default entity alias to any vault block with no entity alias already set"

would be helpful

Comment on lines 140 to 143
To allows tasks to receive tokens that are associated with an entity, the role
must have a list of `allowed_entity_aliases` that includes this entity aliases
that are expected to be used. This field supports globbing to cover multiple
aliases.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trying to make it more succinct

For tasks to receive tokens associated with a Vault entity, the role must include a list of
allowed_entity_aliases indicating the entity aliases allowed for use. This field supports
globbing to cover multiple entity aliases.

also is there vault docs to link to for the globbing rules?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

The most comprehensive doc I found for globbing was this: https://www.vaultproject.io/docs/concepts/policies#policy-syntax

But I don't know if it applies to allowed_entity_aliases so I kept it kind of vague 😅

@lgfa29 lgfa29 merged commit d412f7b into main Apr 5, 2022
@lgfa29 lgfa29 deleted the f-vault-entity-alias branch April 5, 2022 18:18
lgfa29 added a commit that referenced this pull request Apr 20, 2022
After a more detailed analysis of this feature, the approach taken in
PR #12449 was found to be not ideal due to poor UX (users are
responsible for setting the entity alias they would like to use) and
issues around jobs potentially masquerading itself as another Vault
entity.
lgfa29 added a commit that referenced this pull request Apr 22, 2022
After a more detailed analysis of this feature, the approach taken in
PR #12449 was found to be not ideal due to poor UX (users are
responsible for setting the entity alias they would like to use) and
issues around jobs potentially masquerading itself as another Vault
entity.
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants