KBS | Refactoring the codebase / update config file format / bring in plugin mechanism #514

Xynnn007 · 2024-09-27T07:42:58Z

This PR includes the following work

Refactoring the KBS codebase to have plugin mechanism in, including attestation/resource/policy engine
Update the config file format to make it more tidy and scalable.
Combine CoCo/ITA Token verification logic.
Add client plugin mechanism in. All client plugin calls will be filtered by attestation token.

I am sorry to have such a huge PR and did some disruptive changes. I originally want to add a simple plugin mechanism but soon found that some architectural problems and hidden bugs, thus fixed them by the way. At last resulting such many changed lines.

This PR still works fine with kbs-client on the guest-components side and did not change the protocol.

Needs some attestion from ITA folks @mythi @jodh-intel to see if the changes upon ITA config need any change.
Nebula folks @cclaudio about #451

cc @fitzthum @jialez0 @mkulke

Fixes #486
Related to #502

kbs/src/admin/mod.rs

kbs/src/api_server.rs

kbs/src/policy_engine/opa/mod.rs

kbs/src/policy_engine/mod.rs

kbs/src/attestation/backend.rs

fitzthum

Great work. Lots of positive changes here.

I mention one problem in a couple of my comments. Let me expand on it here.

For some of our endpoints, we have GET and POST functionality. The GET request is usually called by the client in the guest, but the POST request has too different cases. Sometimes the POST request is a privileged admin function that is called by the kbs-client running in a trusted environment. Sometimes it is called by the client in the guest.

In some cases we could actually support both. For resources and plugins, for instance, we might want to have an admin be able to configure the resource/plugin, but also have the guest be able to make a POST request. Unfortunately we only have one endpoint for these two things. Maybe we should decide on a convention for how these two different things can be implemented.

I think this will also help resolve some confusion around the so-called client and admin APIs, which are currently intermixed.

kbs/src/resource/backend.rs

kbs/src/api_server.rs

kbs/src/plugins/sample.rs

kbs/src/api_server.rs

fitzthum · 2024-09-27T18:02:12Z

btw I wonder if we should implement the resource backend as a plugin now

Xynnn007 · 2024-09-28T00:06:18Z

@fitzthum Thanks for the review and nice suggestion. About admin and client api, my idea is to move old admin APIs like POST kbs/v0/<plugin> to POST kbs/v0/admin/<plugin>. All admin APIs should be under kbs/v0/admin, no matter it is resource setting or policy setting. This would help us to have a clear border. ~~Let me put a separate commit for review to see if it is good.~~ Please see https://github.com/confidential-containers/trustee/compare/bddfcf5241d71fa1f3263648ac8cfd311cac9df4..91b69d06d33c714de6c591d8f1013feb1e1c686b

btw, about the policy filter for APIs. Now it still works fine with legacy ways, s.t. GET resources. I want to also do a re-design for the policy engine module when refactoring the code. But it seems like an independent work thus not included inside this PR. Mainly for

change the policy format to allow filter both client plugins (including resource) and admin plugins
new point from yours, to add support for GET/POST requests
To have relative static policy (defines roles and privileges) and data (defines how to be a role) and dynamic input (claims to be checked to be a role) to leverage policy engine like the example.

mythi

I checked 542f64c only. Can that be split to a PR of its own? it's a bit too much content in one PR.

kbs/src/token/mod.rs

fitzthum · 2024-09-30T15:18:37Z

@Xynnn007

All admin APIs should be under kbs/v0/admin, no matter it is resource setting or policy setting. This would help us to have a clear border.

I think this is good. Maybe we should also have a generic admin endpoint for each plugin. Like I said, I think it would be good to move our existing resource implementation to a plugin. This would prove that the plugin interface is robust. That might be too much work for this PR, though.

One other thing, do you think we should plugin as part of the route for plugins. As it is people can't implement a plugin named attest because it will collide with the attest endpoint. Maybe that's not too big of a deal. Having a specific plugin route would make the function serving it a bit smaller.

cclaudio · 2024-10-01T18:40:24Z

kbs/src/plugins/sample.rs

+
+pub struct Sample {
+    _item: String,
+}


Can you update the kbs-client to take requests for plugins (e.g. Sample plugin)?

Yes we can! This would give a good implementation example

kbs/src/plugins/sample.rs

cclaudio · 2024-10-04T14:46:48Z

kbs/src/api_server.rs

+                })?;
+            let body = body.to_vec();
+            let response = plugin
+                .handle(body, query.into(), sub_path.into(), request.method())


I think the plugin may/should want to encrypt the response payload. If so, the handle() should be able to access the TEE public key and call jwe(). You could pass the TEE public key via parameter.

I added a encrypted() method for plugin, that will tell KBS if the response needs to be encrypted via JWE.

encrypted() gives some flexibility to the plugin in terms of encrypting the payload. However, I can't think of a plugin that would want its payload to be sent back unencrypted. If we don't have a use case for that we could just assume that all plugins want its payload to be encrypted.

There is one in my mind is to return the public cert whose confidentiality does not need to be protected. The request from guest is the evidence and the CSR of tee public key.

Xynnn007 · 2024-10-08T07:14:13Z

btw I wonder if we should implement the resource backend as a plugin now

Resource backend is somehow different. If we want to implement it as a backend plugin, we would use a config item like

...
[[plugins]]
name = "ResourceStorage"
type = "LocalFs"
dir_path = "/tmp/kbs-resource"

While what we are using now is

...
[repository]
type = "LocalFs"
dir_path = "/tmp/kbs-resource"

The difference

If we use plugin mechanism, if and only if the ResourceStorage is explicitly claimed in the configuration file, the API will be open. This hints that delete feature resource and include that may be a good choice.
If we use legacy way, whether the API is open is decided during compilation time.

Xynnn007 · 2024-10-08T09:53:58Z

@fitzthum I added one commit that tries to make resource a plugin. Also, combine admin and client plugins into one plugins system. The concrete plugin would decide whether it could be accessed upon client auth (attestation) or admin auth. Please take a look if it is good, then I can do some commit tidy work

fitzthum · 2024-10-08T16:35:42Z

I added one commit that tries to make resource a plugin. Also, combine admin and client plugins into one plugins system. The concrete plugin would decide whether it could be accessed upon client auth (attestation) or admin auth. Please take a look if it is good, then I can do some commit tidy work

I really like moving the resource stuff to a plugin. I think it makes sense to get rid of the resource feature. In the future there might be some complex plugins that are behind feature flags (in addition to being enabled at runtime), but imo we should always build the resource support.

Like I said, I could go either way on the admin routes. Maybe another reviewer has an opinion there.

Otherwise I think this is good.

Xynnn007 · 2024-10-09T01:32:46Z

Like I said, I could go either way on the admin routes. Maybe another reviewer has an opinion there.

Otherwise I think this is good.

Nice. I wonder if @cclaudio could give some suggestions upon the design.

~~Also, let me put another PR for token affairs as @mythi mentioned at first.~~

I added #524 about Token parts. Hopefully that makes sense and get merged. Then we can go back to this PR to do some tidy work.

cclaudio · 2024-10-11T06:01:26Z

kbs/src/api_server.rs

+                .await?
+            {
+                // Plugin calls needs to be authorized by the admin auth
+                core.admin_auth.validate_auth(&request)?;


I am wondering what would be the scope of the admin auth (or any other auth) for plugins. The way it is implemented now, the plugin could use the admin auth for anything.

I can think of some ways to have a better separation of duties in order to reduce damage if an auth gets compromised. E.g. we could have one auth only for plugins; or picking a crazy idea from the jar, we could have a wallet maintained by admin and its keys/auth could be selected for plugins via rego policies.

Anyways, the design proposed in the PR looks good to me. We can make adjustments later as needed.

I am not quite sure about the design for now. Before this PR, we have set resource, set policy like APIs who should be accessed via so-called "user public key auth". Also, we have APIs like get resource for clients with Attestation tokens. This PR, I just bring them together and rename to "admin auth" who needs old "user public key auth" and those who need attestation token as auth -- which I think better to decribe the two different roles up to now -- admin and client.

Also, the new design makes it easier for developers to integrate any new APIs that needs to be filtered by "admin auth" without changing too many codes.

Your opintion considers this issue more comprehensively and inspiringly. I think in future we will meet some new demands that require more roles for KBS and we can do some more extension upon the design then.

I am not quite sure about the design for now.

Do you want to expand on that? We could try to improve the parts you are not sure or find a way to move on only with the parts we are good. Honestly, like other reviewers, I also see many good changes in the PR. The auth thing I mentioned before is not introduced here, it is just something that may hit us in the future as we evolve trustee.

I am finishing to implement the nebula plugin on top of this PR, next week I should be able to post it. If the PR is not merged until then 😄, the nebula plugin could be used to help the discussion of lingering questions.

So if I understand correctly, the idea is that it is the admin's policy that defines which API needs admin auth, not the code of plugin, right?

It seems a good idea to make use of policies to control which APIs (or plugin services) need admin auth. The policy could be defined by the plugin or the admin, there are tradeoffs in both cases, but I like the current design.

We don't know yet, but in the future we may have a plugin that wants to provide multiple services. IIUC, the current design allows the plugin to determine which of the services require admin_auth. Additionally, it is up to the plugin to make that decision based a policy e.g. loaded from a config file. Each plugin has its own requirements.

if plugin .validate_auth(&body, query, sub_path, request.method()) .await? {

The admin does not need to know much about the business of the plugin (which plugin service should require admin_auth). That seems good to me. Later, if we think that's not enough, we can extend the admin to override some of the plugin's decision (not with the same granularity).

Ok. So my understanding is to keep the logic as-is. That is, whether admin auth is needed is defined by plugin code implementation validate_auth(). Thanks for the explaination and if you anything misunderstanding please correct me

Yep, looks good to me. Thanks

kbs/src/plugins/error.rs

mkulke

some nits

kbs/src/api_server.rs

kbs/src/plugins/implementations/resource/backend.rs

ChengyuZhu6

LGTM Thanks @Xynnn007

fitzthum

LGTM

cclaudio

Looks good to me. Thanks @Xynnn007

fitzthum · 2024-11-01T22:16:18Z

ok are we ready to merge this?

Xynnn007 · 2024-11-02T03:59:54Z

ok are we ready to merge this?

hi are you ok with this? @mkulke @mythi

mkulke

lgtm

mythi

Looks great work to me and I could not spot anything that would block merging. I made some suggestions about "Intel TA" wording usage.

My biggest open is about keeping the KBS session timeout configurable but that logic isn't changing in this PR so it's not a blocker either, I guess.

kbs/config/kbs-config-intel-trust-authority.toml

kbs/docs/config.md

kbs/test_data/configs/intel-ta-1.toml

mythi · 2024-11-05T07:08:43Z

kbs/src/attestation/backend.rs

+    /// Max timeout between `auth` and `attest` request of a client
+    timeout: i64,


in #162 I raised the question about this timeout and IIRC @fitzthum mentioned it somewhere too. I'm asking the question again since we are making big config changes: do we need to have this configurable for KBS or should it just follow the token exp.

With the current CDH setup, is it possible, for example to set 1 minute here but CDH can continue to pull secrets as long as the token is valid (5min by default)?

This timeout as noted here is not the expiration of the token. It is just the maximum delay between an auth request and an attest request. That means if the attest comes too late after auth, the RCAR handshake will fail. You raised a good idea about the inner timeout for session. I think we should distinguish the two different timeouts. Btw, do you think we need to add another option to specify the timeout of the session time, or use a hardcode one? cc @fitzthum

Updated in the following comments

With the current CDH setup, is it possible, for example to set 1 minute here but CDH can continue to pull secrets as long as the token is valid (5min by default)?

Actaully it can for now : | The CDH uses the token (via HTTP Auth header of request) to pull secrets rather than reuse the session.

This is a little tricky. Actually the both timeout are the session timeout. What it will influence is when a secret access comes, the token will firstly be looked up from the session. If the session is expired, the http auth header will be checked to get the token.

As for whether the resource retrievement is allowed, it is the checking against the token that matters. Some different cases

timeout > token expiration time: the token can be got. When checking against the token, kbs will realize the token is expired then failed to pull secret.

timeout < token expiration time: when trying to get token from session, the kbs will realize the session is expired. Then try to get token from http auth header. If a token is found, the secret pulling will succeed. If no tokens are found, the secret pulling will fail.

The core thing is -- No secrets can be accessed with only a session. It must rely on a token and its passing policy enforcement.

I think we should consider getting rid of the session timeout and only using the token expiration (or making sure they always end up being the same), but this is a question for another PR.

Yup. I added a commit for Token verifier to check the expiration time of the token. Hopefully things can be clear and we can discuss more back to #162

that is not necessary since jsonwebtoken handles that automatically when using the default Validation. It's also possible to make that explicit.

Oh yes. nice. Let me delete the last commit

This refactoring combines all RCAR (attestation) related code into one module. This would help to better modularization and error handling. Signed-off-by: Xynnn007 <[email protected]>

This commit does some refactoring upon policy engine module. Including 1. Change ResourcePolicyError to PolicyEngineError. This is because in future, different client plugins would share same policy engine thus the new name will match better. 2. add a new `set_policy` api for PolicyEngine. This api will handle SetPolicyInput format request rather than the plaintext of policy. This would help to integrate into the KBS server. The plugin mechanism is by default enabled, thus we delete `opa` and `policy` feature. By default integrate `regorus` crate for policy. Signed-off-by: Xynnn007 <[email protected]>

This module brings all admin authentication logic together. Currently it allows to use a public key to verify the admin access. Signed-off-by: Xynnn007 <[email protected]>

The Plugins module could provide a plugin way for developers to extend the ability of KBS client APIs. This also provides a Sample implementation for example. The oroginal resource is refactored into plugins as a plugin. Also, we change both `read_secret_resource` and `write_secret_resource` to Fn rather than FnMut. This leaves the synchronization handling to concrete underlying plugins, thus promote the performance because we can avoid a global Mutex. Signed-off-by: Xynnn007 <[email protected]>

This is mostly a refactoring patch for KBS. It brings API serving into one function, and will perform different sub-function due to the requested plugin name. This also changes all configuration codes to have a default value. This patch would have some compatibility issue as it changes the old configuration format. The old configuration format is not well classified. This patch tidies the configuration items. Signed-off-by: Xynnn007 <[email protected]>

This patch fixes example configurations of KBS inside this codebase. Also, it fixes the CI test and the docs. Signed-off-by: Xynnn007 <[email protected]>

Signed-off-by: Xynnn007 <[email protected]>

Now the KBS could be built with support for all backend ASes and enable one of them runtimely due to configuration file. Signed-off-by: Xynnn007 <[email protected]>

This patch uses a single logic to handle all different requests built upon plugin system. Also, resource module is by default a plugin and will be enabled by default. Signed-off-by: Xynnn007 <[email protected]>

Signed-off-by: Xynnn007 <[email protected]>

mythi

LGTM, thanks!

Xynnn007 marked this pull request as ready for review September 27, 2024 08:05

Xynnn007 requested a review from a team as a code owner September 27, 2024 08:05

mkulke reviewed Sep 27, 2024

View reviewed changes

Xynnn007 force-pushed the refactor-kbs branch from 54c5b4d to bddfcf5 Compare September 27, 2024 09:07

fitzthum reviewed Sep 27, 2024

View reviewed changes

Xynnn007 force-pushed the refactor-kbs branch 2 times, most recently from 91b69d0 to db46ea2 Compare September 28, 2024 07:36

mythi suggested changes Sep 30, 2024

View reviewed changes

kbs/src/token/mod.rs Outdated Show resolved Hide resolved

kbs/src/token/mod.rs Outdated Show resolved Hide resolved

mythi mentioned this pull request Oct 1, 2024

ita: updated trusted_certs_paths property documentation #515

Merged

cclaudio reviewed Oct 1, 2024

View reviewed changes

kbs/src/plugins/sample.rs Outdated Show resolved Hide resolved

cclaudio reviewed Oct 4, 2024

View reviewed changes

Xynnn007 mentioned this pull request Oct 9, 2024

KBS: combine CoCo Token and Jwk Token verifier #524

Merged

cclaudio reviewed Oct 11, 2024

View reviewed changes

cclaudio reviewed Oct 18, 2024

View reviewed changes

kbs/src/plugins/error.rs Outdated Show resolved Hide resolved

cclaudio mentioned this pull request Oct 19, 2024

Add nebula_ca plugin #539

Open

Xynnn007 force-pushed the refactor-kbs branch 3 times, most recently from 3881a37 to 492251c Compare October 23, 2024 07:53

Xynnn007 requested review from mythi, cclaudio, fitzthum and mkulke October 23, 2024 08:28

mkulke reviewed Oct 29, 2024

View reviewed changes

kbs/src/api_server.rs Outdated Show resolved Hide resolved

kbs/src/plugins/implementations/resource/backend.rs Show resolved Hide resolved

Xynnn007 force-pushed the refactor-kbs branch 2 times, most recently from b11b9c0 to 16739c6 Compare October 30, 2024 02:38

ChengyuZhu6 approved these changes Oct 30, 2024

View reviewed changes

Xynnn007 requested a review from mkulke October 30, 2024 13:32

fitzthum approved these changes Oct 30, 2024

View reviewed changes

cclaudio approved these changes Oct 30, 2024

View reviewed changes

Xynnn007 mentioned this pull request Nov 1, 2024

AS: fix build with rvps features #553

Merged

mkulke approved these changes Nov 2, 2024

View reviewed changes

Xynnn007 force-pushed the refactor-kbs branch from 16739c6 to 06c4599 Compare November 5, 2024 04:15

mythi reviewed Nov 5, 2024

View reviewed changes

Xynnn007 added 10 commits November 5, 2024 17:35

KBS: refactor attestation module

d0b08fc

This refactoring combines all RCAR (attestation) related code into one module. This would help to better modularization and error handling. Signed-off-by: Xynnn007 <[email protected]>

KBS: add Admin auth module

a773544

This module brings all admin authentication logic together. Currently it allows to use a public key to verify the admin access. Signed-off-by: Xynnn007 <[email protected]>

KBS: fix CI, docs and exampled configurations

71573e2

This patch fixes example configurations of KBS inside this codebase. Also, it fixes the CI test and the docs. Signed-off-by: Xynnn007 <[email protected]>

AS: reorder the dep in lexicographic order

d78dbe1

Signed-off-by: Xynnn007 <[email protected]>

KBS: change default feature to all backend AS and resource

4ef93f5

Now the KBS could be built with support for all backend ASes and enable one of them runtimely due to configuration file. Signed-off-by: Xynnn007 <[email protected]>

KBS: Use one API to serve both admin and user requests

be3c836

This patch uses a single logic to handle all different requests built upon plugin system. Also, resource module is by default a plugin and will be enabled by default. Signed-off-by: Xynnn007 <[email protected]>

toml: add extra line to all toml files

85b7ab8

Signed-off-by: Xynnn007 <[email protected]>

Xynnn007 force-pushed the refactor-kbs branch from 06c4599 to 85b7ab8 Compare November 5, 2024 09:36

Xynnn007 mentioned this pull request Nov 6, 2024

KBS implementation observations #162

Closed

Xynnn007 force-pushed the refactor-kbs branch from c62d109 to 85b7ab8 Compare November 6, 2024 05:47

mythi approved these changes Nov 6, 2024

View reviewed changes

Xynnn007 merged commit a2c804a into confidential-containers:main Nov 6, 2024
31 checks passed

Xynnn007 deleted the refactor-kbs branch November 6, 2024 06:13

Xynnn007 mentioned this pull request Nov 6, 2024

KBS | Plugin system design #502

Closed

		/// Max timeout between `auth` and `attest` request of a client
		timeout: i64,

KBS | Refactoring the codebase / update config file format / bring in plugin mechanism #514

KBS | Refactoring the codebase / update config file format / bring in plugin mechanism #514

Conversation

Xynnn007 commented Sep 27, 2024 • edited Loading

fitzthum left a comment

Choose a reason for hiding this comment

fitzthum commented Sep 27, 2024

Xynnn007 commented Sep 28, 2024 • edited Loading

mythi left a comment

Choose a reason for hiding this comment

fitzthum commented Sep 30, 2024

cclaudio Oct 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cclaudio Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xynnn007 commented Oct 8, 2024

Xynnn007 commented Oct 8, 2024

fitzthum commented Oct 8, 2024

Xynnn007 commented Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cclaudio Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Xynnn007 Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkulke left a comment

Choose a reason for hiding this comment

ChengyuZhu6 left a comment

Choose a reason for hiding this comment

fitzthum left a comment

Choose a reason for hiding this comment

cclaudio left a comment

Choose a reason for hiding this comment

fitzthum commented Nov 1, 2024

Xynnn007 commented Nov 2, 2024

mkulke left a comment

Choose a reason for hiding this comment

mythi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xynnn007 Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Xynnn007 Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

fitzthum Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mythi Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mythi left a comment

Choose a reason for hiding this comment

Xynnn007 commented Sep 27, 2024 •

edited

Loading

Xynnn007 commented Sep 28, 2024 •

edited

Loading

cclaudio Oct 1, 2024 •

edited

Loading

cclaudio Oct 4, 2024 •

edited

Loading

Xynnn007 commented Oct 9, 2024 •

edited

Loading

cclaudio Oct 28, 2024 •

edited

Loading

Xynnn007 Oct 29, 2024 •

edited

Loading

Xynnn007 Nov 5, 2024 •

edited

Loading

Xynnn007 Nov 5, 2024 •

edited

Loading

fitzthum Nov 5, 2024 •

edited

Loading

mythi Nov 6, 2024 •

edited

Loading