Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage Operator can use SaltAPI without RBAC in 2.5 #2634

Closed
gdemonet opened this issue Jun 18, 2020 · 12 comments · Fixed by #2642
Closed

Storage Operator can use SaltAPI without RBAC in 2.5 #2634

gdemonet opened this issue Jun 18, 2020 · 12 comments · Fixed by #2642
Assignees
Labels
release:blocker An issue that blocks a release until resolved severity:major Major impact on live deployments (e.g. some non-critical feature is not working at all) state:question Further information is requested topic:salt Everything related to SaltStack in our product topic:security Security-related issues topic:tests What's not tested may be broken

Comments

@gdemonet
Copy link
Contributor

Component: salt

What happened:

When implementing #2625, we found that forgetting the permission for a newly added Salt execution function (namely, metalk8s_volumes.device_name) didn't prevent storage-operator from using it, which is a serious security hole.
This appeared in 2.5.1-dev, whereas the RBAC was properly enforced in 2.6.0-dev.

What was expected:

Without the relevant permissions, one shouldn't be able to use SaltAPI freely.
If there is a hole in our setup in 2.5 around RBAC with Salt(API), we need it fixed ASAP.

Steps to reproduce: TBD

Resolution proposal (optional): TBD (we need a reproducing example, and tests to ensure this behaviour is fixed)

@gdemonet gdemonet added state:question Further information is requested topic:security Security-related issues topic:tests What's not tested may be broken severity:major Major impact on live deployments (e.g. some non-critical feature is not working at all) topic:salt Everything related to SaltStack in our product labels Jun 18, 2020
@gdemonet
Copy link
Contributor Author

First observation: in salt/_auth/kubernetes_rbac.py, the auth method is not checking that the username provided matches the ServiceAccount for which the token was issued. That in itself is a security hole (cc @NicolasT)

@NicolasT
Copy link
Contributor

There's no difference in the eauth code between 2.5 and 2.6, so it's odd as to why it worked in 2.5 and not 2.6.

@NicolasT
Copy link
Contributor

Actually not sure there's something wrong here. When looking at storage-operator/deploy/role.yaml, the storage-operator ClusterRole allows create/delete/get/list/patch/update/watch on node objects. As such, it's deemed to be part of the Salt node-admins group (which requires * on nodes in K8s, see the eauth code).

This node-admins group can un .* on * (see salt/metalk8s/salt/master/files/master-99-metallk8s.conf.j2).

So everything works as designed (except that maybe the storage-operator should not have * rights on node objects, of course!)

@NicolasT
Copy link
Contributor

First observation: in salt/_auth/kubernetes_rbac.py, the auth method is not checking that the username provided matches the ServiceAccount for which the token was issued. That in itself is a security hole (cc @NicolasT)

This is indeed a bug and needs to be fixed.

@NicolasT
Copy link
Contributor

Seems introduced in 56eda6d @slaperche-scality. Giving the storage-operator role access to nodes gives it node-admins permissions in Salt.

@NicolasT
Copy link
Contributor

Actually, in 2.6 the * verbs on nodes was turned into CRUDLPW, hence the storage-operator account getting node-admins in Salt in 2.5 but no longer in 2.6.

Now, storage-operator shouldnt have CRUDLPW on Node (I think) in the first place (plus a whole bunch of other rights in the ClusterRole for it it doesn't really need).

@slaperche-scality
Copy link
Contributor

Now, storage-operator shouldnt have CRUDLPW on Node (I think) in the first place (plus a whole bunch of other rights in the ClusterRole for it it doesn't really need).

Yup, see #2084

@slaperche-scality
Copy link
Contributor

Giving the storage-operator role access to nodes gives it node-admins permissions in Salt.

That's really a weird side-effect, I've totally missed it :/

@gdemonet
Copy link
Contributor Author

That's really a weird side-effect

I can't agree more. Could we imagine building another RBAC model for Salt in K8s? I'm thinking, just as an example, we could create dummy objects (empty ConfigMaps) in some metalk8s-salt namespace, and use the SubjectAccessReviews to control which "groups" (in Salt world) a user belongs to.

An example Role could then be:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: metalk8s-salt
  name: salt-admin
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  resourceNames:
  - salt-rbac-admin
  verbs:
  - get

If we really wanted to (not sure though), we could even use the contents of these ConfigMaps as a source for templating the RBAC rules in the Salt master configuration.

gdemonet added a commit that referenced this issue Jun 18, 2020
We don't want too open permissions for security reasons that were made
obvious in #2634. We thus reduce the rules deployed to the bare minimum,
and add some inline comments to better explain why each ruleset is
needed.

Fixes: #2084
gdemonet added a commit that referenced this issue Jun 18, 2020
This checks that if not allowed in Salt eauth perms, an authenticated
user (using 'storage-operator' as a practical example) cannot run
any module (using 'test.ping' as an example).

See: #2634
@gdemonet gdemonet added the release:blocker An issue that blocks a release until resolved label Jun 18, 2020
@gdemonet gdemonet added this to the MetalK8s 2.5.1 milestone Jun 18, 2020
gdemonet added a commit that referenced this issue Jun 19, 2020
We don't want too open permissions for security reasons that were made
obvious in #2634. We thus reduce the rules deployed to the bare minimum,
and add some inline comments to better explain why each ruleset is
needed.

Fixes: #2084
gdemonet added a commit that referenced this issue Jun 19, 2020
This checks that if not allowed in Salt eauth perms, an authenticated
user (using 'storage-operator' as a practical example) cannot run
any module (using 'test.ping' as an example).

See: #2634
gdemonet added a commit that referenced this issue Jun 19, 2020
This checks that if not allowed in Salt eauth perms, an authenticated
user (using 'storage-operator' as a practical example) cannot run
any module (using 'test.ping' as an example).

See: #2634
@gdemonet gdemonet self-assigned this Jun 22, 2020
@gdemonet
Copy link
Contributor Author

Could we imagine building another RBAC model for Salt in K8s?

Another idea: when using the TokenReview API (which only supports Bearer tokens, not Basic), we can get all the groups this user is in (under status.user.groups). Not entirely sure how, but we could try to use those.

@NicolasT
Copy link
Contributor

Could we imagine building another RBAC model for Salt in K8s?

Another idea: when using the TokenReview API (which only supports Bearer tokens, not Basic), we can get all the groups this user is in (under status.user.groups). Not entirely sure how, but we could try to use those.

We can't, I believe, and shouldn't:

  • A ServiceAccount token will likely not have any groups defined here.
  • This would require to configure Salt with intel about the groups a human user may be part of in the directory service of the customer, which is 'meh'. Also, would depend on Dex groups support (which, granted, we may have).

@gdemonet
Copy link
Contributor Author

Mmh right, ServiceAccounts only have system: groups, like system:serviceaccounts:<namespace> or system:authenticated. Good point. So WDYT about synthetic objects we could use to perform SubjectAccessReviews?

gdemonet added a commit that referenced this issue Jun 22, 2020
When using ServiceAccount, we were not checking if the provided Bearer
`token` actually belonged to the claimed `username`. This is now done,
and we add tests to prevent regressions.

NOTE: storage-operator username, from SaltAPI PoV, is now
'system:serviceaccount:kube-system:storage-operator'.

Fixes: #2634
gdemonet added a commit that referenced this issue Jun 23, 2020
When using ServiceAccount, we were not checking if the provided Bearer
`token` actually belonged to the claimed `username`. This is now done,
and we add tests to prevent regressions.

NOTE: storage-operator username, from SaltAPI PoV, is now
'system:serviceaccount:kube-system:storage-operator'.

Fixes: #2634
gdemonet added a commit that referenced this issue Jun 23, 2020
We only assessed if an authentication attempt failed, while tried to
check the actual permissions for an authn assumed successful.
To make errors more explicit, we add a `then` check that the
"authentication succeeds".

See: #2634
gdemonet added a commit that referenced this issue Jun 23, 2020
When using ServiceAccount, we were not checking if the provided Bearer
`token` actually belonged to the claimed `username`. This is now done,
and we add tests to prevent regressions.

NOTE: storage-operator username, from SaltAPI PoV, is now
'system:serviceaccount:kube-system:storage-operator'.

Fixes: #2634
gdemonet added a commit that referenced this issue Jun 23, 2020
We only assessed if an authentication attempt failed, while tried to
check the actual permissions for an authn assumed successful.
To make errors more explicit, we add a `then` check that the
"authentication succeeds".

See: #2634
gdemonet added a commit that referenced this issue Jun 23, 2020
When using ServiceAccount, we were not checking if the provided Bearer
`token` actually belonged to the claimed `username`. This is now done,
and we add tests to prevent regressions.

NOTE: storage-operator username, from SaltAPI PoV, is now
'system:serviceaccount:kube-system:storage-operator'.

Fixes: #2634
gdemonet added a commit that referenced this issue Jun 23, 2020
We only assessed if an authentication attempt failed, while tried to
check the actual permissions for an authn assumed successful.
To make errors more explicit, we add a `then` check that the
"authentication succeeds".

See: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
When using ServiceAccount, we were not checking if the provided Bearer
`token` actually belonged to the claimed `username`. This is now done,
and we add tests to prevent regressions.

NOTE: storage-operator username, from SaltAPI PoV, is now
'system:serviceaccount:kube-system:storage-operator'.

Fixes: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
We only assessed if an authentication attempt failed, while tried to
check the actual permissions for an authn assumed successful.
To make errors more explicit, we add a `then` check that the
"authentication succeeds".

See: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
When using ServiceAccount, we were not checking if the provided Bearer
`token` actually belonged to the claimed `username`. This is now done,
and we add tests to prevent regressions.

NOTE: storage-operator username, from SaltAPI PoV, is now
'system:serviceaccount:kube-system:storage-operator'.

Fixes: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
We only assessed if an authentication attempt failed, while tried to
check the actual permissions for an authn assumed successful.
To make errors more explicit, we add a `then` check that the
"authentication succeeds".

See: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
Since our eauth module for Salt doesn't accept Basic auth anymore, we
remove this logic from the storage-operator entirely.
Since we don't plan on adding this support back, nor on adding other
authentication methods (everything should work with Bearer tokens), we
also remove the multi-method handling from our eauth module.

See: #2634
See: #2642
gdemonet added a commit that referenced this issue Jun 24, 2020
When using ServiceAccount, we were not checking if the provided Bearer
`token` actually belonged to the claimed `username`. This is now done,
and we add tests to prevent regressions.

NOTE: storage-operator username, from SaltAPI PoV, is now
'system:serviceaccount:kube-system:storage-operator'.

Fixes: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
We only assessed if an authentication attempt failed, while tried to
check the actual permissions for an authn assumed successful.
To make errors more explicit, we add a `then` check that the
"authentication succeeds".

See: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
We only assessed if an authentication attempt failed, while tried to
check the actual permissions for an authn assumed successful.
To make errors more explicit, we add a `then` check that the
"authentication succeeds".

See: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
Since our eauth module for Salt doesn't accept Basic auth anymore, we
remove this logic from the storage-operator entirely.
Since we don't plan on adding this support back, nor on adding other
authentication methods (everything should work with Bearer tokens), we
also remove the multi-method handling from our eauth module.

See: #2634
See: #2642
gdemonet added a commit that referenced this issue Jun 24, 2020
The fact that any authenticated user can use the SelfSubjectAccessReview
API was not self-evident. We add a comment to clarify this.

See: #2634
gdemonet added a commit that referenced this issue Jun 24, 2020
We only need this token when interacting with K8s API, and it's already
stored in LocalStorage. We don't need to save it in the Redux store.

See: #2634
@bert-e bert-e closed this as completed in 1ebda6e Jun 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release:blocker An issue that blocks a release until resolved severity:major Major impact on live deployments (e.g. some non-critical feature is not working at all) state:question Further information is requested topic:salt Everything related to SaltStack in our product topic:security Security-related issues topic:tests What's not tested may be broken
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants