FUSE key protection scheme

[jiazhang0]

Background

We have discussed the FUSE key provisioning approach for a long time with the approaches #11 and #3.

Actually, all approaches surrounds corresponding scenarios. Without the background of scenario, it is inefficient to decide next actions.

There are 2 scenarios with different approaches to deploy enclave-cc, which heavily affects the approach of FUSE key provisioning. I recognized we don't need to seek one unified approach to cover both scenarios.

Scenario 1: Tenant owning platform

In order to run confidential containers, a tenant would like to pay for a VM or bare-metal as a platform/node to set up enclave-cc. In this scenario, a tenant has to also set up K8s and take the responsibility of platform maintenance. In this scenario, tenant hopes to have strong control on the platform.

Here is the workflow for this scenario:

• App-Enclave and Agent-Enclave are signed by tenant as signer.
• Use sealing key with MRSIGNER policy is much simpler than other approaches.
• The FUSE key is protected by the sealing key which acts as a wrapping key.
• By the way, LA protocol can still support this scenario.

Scenario 2: CSP owing platform

In this scenario, a tenant only needs to pay for a Pod to deploy a protected container image to enclave-cc provided by CSP. Obviously this is mainstream PaaS use model with efficient cost reduction, sacrificing the control of platform.

v1

Here is the workflow for this scenario 2 v1 approach:

• Sealing key with MRSIGNER policy is not useful.
• Even the enclave binaries used to host app-enclave and agent-encalve are same, their contents of configuration data or manifest are different and are reflected in MRENCLAVE so sealing key with MRENCLAVE policy plus other factors are not useful.
• Launching local attestation protocol as described in [Future Work] Adding LA design for Filesystem Decryption Key Provisioning #11
• Relying Party authenticates the identity of Agent-Enclave as attester, and then provisions App-Enclave's MRENCLAVE to Agent-Enclave as reference value.
• Agent-Enclave as verifier this time can authenticate the identity of App-Enclave with App-Enclave's MRENCLAVE as reference value.
• This approach assumes App-Enclave doesn't need to authenticate the identity of Agent-Enclave.

v2

• It is more flexible to support decoupling the configuration data or manifest and enclave binary in MRENCLAVE. This is what KSS can do. Here is a good material describing the details of using KSS.
• V2 approach additionally needs the authentications to CONFIGID in remote and local attestation.
• Occlum can leverage KSS to load a runtime configurable configuration data or manifest.
• Still, this approach assumes App-Enclave doesn't need to authenticate the identity of Agent-Enclave.

v3

• In order to allow App-Enclave to authenticate the identity of Agent-Enclave, the Agent-Enclave's MRENCLAVE is recorded in the configuration data or manifest of App-Enclave.
• When launching App-Enclave, the configuration data or manifest of App-Enclave is provisioned to App-Enclave.
• App-Enclave then uses Agent-Enclave's MRENCLAVE to authenticiate the identity of Agent-Enclave.
• The integrity of App-Enclavev's configuration data or manifest containing Agent-Enclave's MRENCLAVE is verified by Agent-Enclave during local attestation.
• Open: is there really necessary to authenticate agent-enclave by app-enclave? Even if the agent-enclave is disguised, it has no ability to spoof app-enclave to retrieve the genuine FUSE key.

Conclusions

• Code sealing key with MRSIGNER policy as the first supported approach.
• Add the details of identity authentication in [Future Work] Adding LA design for Filesystem Decryption Key Provisioning #11 and merge this PR, driving the continuous development work to support the scenario 2.

[mythi]

Occlum can leverage KSS to load a runtime configurable configuration data or manifest.

Is this Occlum.json or something else?

[qzheng527]

Occlum can leverage KSS to load a runtime configurable configuration data or manifest.

Is this Occlum.json or something else?

No, use KSS to pass application rootfs hash or other useful information.
https://github.com/occlum/occlum/blob/master/docs/remote_attestation.md#sgx-kss-key-separation-and-sharing-feature-support

[dcmiddle]

Arguably there are not two scenarios. CoCo should not trust the infrastructure and orchestration actors / personas regardless of who owns them. This issue does highlight though the importance that the Workload Provider should be able to provide or inspect the CoCo enclaves (this is equally true for the CoCo sandbox guest).

This issue nicely describes options based on using MRENCLAVE to know that a 3rd party (perhaps the CSP) has provided a known copy of the enclaves. That implies though a number of other things including that the Workload Provider can reproduce MRENCLAVE from sources provided by the CSP.

It may be even more beneficial if we can find patterns to enable the Workload Provider to easily provide their own builds of the enclaves. This will have the added benefit of enabling trust policies with reduced Break Once Run Everywhere risks.

One intermediate approach would be for CoCo to provide trusted builds of the enclaves. CSPs or other infrastructure/orchestration operators would deploy those without inserting themselves into the trust boundary.

A further approach could be for CSPs to facilitate enclave builds by their customers. For example, as part of enrolling in a confidential container offering the customer designates a key under their own control for signing enclaves provided through a verifiable service. That service would use CoCo sources and reproducible build processes such that a customer can independently audit and verify them. In this example the service adoption is still easy for a customer (they don't have to manually produce builds) but also offers greater protection and audit-ability.

These are just two quick examples. I do think that this line of exploration - customer owned/influenced CoCo enclaves or guests - is important to the long term value of CoCo.

[jiazhang0]

Arguably there are not two scenarios. CoCo should not trust the infrastructure and orchestration actors / personas regardless of who owns them. This issue does highlight though the importance that the Workload Provider should be able to provide or inspect the CoCo enclaves (this is equally true for the CoCo sandbox guest).

Yes. The scenario 1 doesn't assume the platform is trusted even it is owned by tenant, because the adversary can still exploit the possible vulnerabilities to break into the platform.

This issue nicely describes options based on using MRENCLAVE to know that a 3rd party (perhaps the CSP) has provided a known copy of the enclaves. That implies though a number of other things including that the Workload Provider can reproduce MRENCLAVE from sources provided by the CSP.

Yes. This is what RVPS (confidential-containers/confidential-containers#122) with reproducible build feature wants to address.

It may be even more beneficial if we can find patterns to enable the Workload Provider to easily provide their own builds of the enclaves. This will have the added benefit of enabling trust policies with reduced Break Once Run Everywhere risks.

It sounds like a trusted supply chain owner is responsible for dong this. It would efficiently convince Workload Provider to trust the signed enclaves with MRSIGNER identity. Even Workload Provider intends to go further, e.g, RVPS with reproducible build support can establish the relationship between MRENCLAVE identity and source codes.

One intermediate approach would be for CoCo to provide trusted builds of the enclaves. CSPs or other infrastructure/orchestration operators would deploy those without inserting themselves into the trust boundary.

Agreed. I think CoCo community should employ a in-toto supply chain or similar to provide the artifacts for the next official release.

A further approach could be for CSPs to facilitate enclave builds by their customers. For example, as part of enrolling in a confidential container offering the customer designates a key under their own control for signing enclaves provided through a verifiable service. That service would use CoCo sources and reproducible build processes such that a customer can independently audit and verify them. In this example the service adoption is still easy for a customer (they don't have to manually produce builds) but also offers greater protection and audit-ability.

Interesting idea. Actually I also think it is necessary to introduce a provisioning phase in CoCo's design to establish a minimal RoT for Confidential Containers (RTCC) in the first boot of K8s. This phase can cover signing enclaves, signing boot image (for kata-cc), establishing the trust to service TD (introduced by TDX 1.5), caching collateral in node level, provisioning TEE-based vTPM instance and so on.

These are just two quick examples. I do think that this line of exploration - customer owned/influenced CoCo enclaves or guests - is important to the long term value of CoCo.

Fully agreed. A delegated party / role / service should be introduced to implement a layered architecture and assist the establishment of the trustworthiness of CoCo. Otherwise, Workload Provider as cloud tenant in practice needs to do too many things, and it is too difficulty to make it happen and secured.

So back to this topic, MRSIGNER identity can be used in the scenario where a delegated party / role / service / Workload Provider self as the provider of artifact is involved. By contrary, MRENCLAVE identity with local attestation protocol can be used where the provider of artifact is not trusted.

[Xynnn007]

One intermediate approach would be for CoCo to provide trusted builds of the enclaves. CSPs or other infrastructure/orchestration operators would deploy those without inserting themselves into the trust boundary.

Agreed. I think CoCo community should employ a in-toto supply chain or similar to provide the artifacts for the next official release.

I think it is a suitable goal for us to implement a verifiable build pipeline like in-toto in the next release.

The core idea for RVPS is to provide trusted digests to compare with the gathered digest of non-tenant components inside TEE. As mentioned in this slide, IMO, a verifiable build pipeline (means recording building metadata/provenance) will be good start for CoCo community to provide trusted digests.

With the signed metadata together with the digest of the artifact, RVPS can provide an engineering implementation to help compare it with the digest calculated using MRENCLAVE here.

CC @sameo @fitzthum @bpradipt

[dcmiddle]

@jiazhang0 I forgot one thing. You had suggested in confidential-containers/documentation#3 another option is building the agent and app into a single enclave. That way sealing with MRENCLAVE is possible.
This approach has a benefit of simplicity.

[dcmiddle]

@Xynnn007 do I understand correctly that RVPS would receive the digests, and that we need a separate issue/discussion to adopt in-toto practices? And also In-Toto doesn't cover how to get to repeatable builds, as much as just properly documenting the build process and actors? So probably we need to investigate repeatable builds for each trusted component, and then make sure we have an over-arching in-toto (or similar) build process with a stage that outputs measurements (to RVPS or other attestation services).

[jiazhang0]

@jiazhang0 I forgot one thing. You had suggested in confidential-containers/documentation#3 another option is building the agent and app into a single enclave. That way sealing with MRENCLAVE is possible. This approach has a benefit of simplicity.

I think of it in depth afterward but it is difficulty to implement in practice. See #3 (comment) for details. Please let me know whether you and @mythi have better solutions.

[monavij]

One problem with scenario confidential-containers/documentation#1 is that it will work only if you own the physical platform. If you rent a VM from a CSP there is no guarantee that it will land on the same HW machine everytime and so you can't rely on the sealing feature. That case is ONLY valid when you do physically own the platform. On the other hand since VM migration is not supported for SGX, maybe SGX enabled Tenant VMs won't be moved and will always be guaranteed to run on the same HW. But in general MRSIGNER based authentication is weak, so MRSIGNER based solution will be more robust. Also yo won't be able to distinguish between multiple apps signed by the same MRSIGNER.

[jiazhang0]

One problem with scenario confidential-containers/documentation#1 is that it will work only if you own the physical platform. If you rent a VM from a CSP there is no guarantee that it will land on the same HW machine everytime and so you can't rely on the sealing feature. That case is ONLY valid when you do physically own the platform. On the other hand since VM migration is not supported for SGX, maybe SGX enabled Tenant VMs won't be moved and will always be guaranteed to run on the same HW. But in general MRSIGNER based authentication is weak, so MRSIGNER based solution will be more robust. Also yo won't be able to distinguish between multiple apps signed by the same MRSIGNER.

Thanks for your hint about the vSGX scenario.

In practice, CSP will ensure the vSGX VM instance cannot be re-scheduled to another platform, even sacrificing VM migration feature support.

About the robustness of MRSIGNER based authentication, I also think it is weaker than MRENCLAVE based authentication. The risk of leaking sealing key with MRSIGNER policy happens when an exploitable vulnerability exists in any one of signed enclave binaries deployed on this platform, especially the ones with different purpose against enclave-cc. This risk is increased because all signed enclave binaries share the same attack surface due to using the MRSIGNER policy. This is specific in scenario 1. What are your comments here? @dcmiddle @mythi

[mythi]

If you rent a VM from a CSP there is no guarantee that it will land on the same HW machine everytime and so you can't rely on the sealing feature. That case is ONLY

But the sealed container image content is stateless. If the container lands on another node it needs to be pulled and prepared again.

[dcmiddle]

Regarding sealing, there are two practical factors. [A] the sealed FUSE key file is short-lived and [B] the deployment model for CoCo is in early evolution.

A. The key just needs to be written by the agent then read and the sealed file removed by the App. Certainly a privileged adversary could collect that sealed file and wait for a weak enclave in the future, but this is a limited risk.

B. I believe that after the first one or two releases deployment assumptions will evolve. We should bias our design towards simplicity and then refine as we learn. This particular portion of the design is focused on encrypted container images. We may find for example, that the world benefits much more from integrity enforced by confidential compute, and/or the ability to load secrets into that running high-integrity container.

[ariel-adam]

@jiazhang0 is this issue still relevant or can be closed?
If it's still relevant to what release do you think we should map it to (mid-November, end-December, mid-February etc...)?

[dcmiddle]

Updated the issue's project association to the next release. (redundantly mentioning it in the comments here since this is a new process).

[dcmiddle]

Local attestation implementation trending to v0.3.
Meanwhile we will continue to use the hard-coded key or we can consider sealing.
Solving the trusted build scenario (how the MRENCLAVE is known and trusted) is a related unresolved issue.

[hairongchen]

updates after community meeting: LA code submission into enclave-cc code base, targeting V0.4

[mythi]

closing as complete