Questions about "on-prem/private cloud" environments

[palenica]

Questions about "on-prem/private cloud" environments

We are exploring possibilities for enabling trusted Privacy Sandbox workloads to be run outside of public clouds with a dedicated TEE offering. We are interested in a better understanding of compute environments adtechs employ currently, and understanding how to architect Privacy Sandbox software so that it can be run in such environments.

In this issue, we'd like to focus on the adtech's ability to run a workload in their production environment. Most security and privacy challenges, while critical for us to address, are out of scope for this issue.

Consider a trusted server workload such as the Key-Value Server, or a server from the Bidding and Auction Services, for example the Bidding Server.

• Would you be able to bring up the "local" build of the KV server in your data center (compute environment)? If not, are there specific technical barriers that prevent you from doing so?

• In your production environment, do you have the ability to run a workload that is a Linux binary? Or a container?

• In your production environment, do you have the ability to run workloads on "bare metal" -- that is, not inside a VM?

• In your production environment, would the workload (say the KV server) be able to start a virtual machine (VM) using a VMM such as QEMU or Cloud Hypervisor? We ask, because eventually, we would like the "trusted" portion of the server to run inside a "Confidential VM" for confidentiality and security protection.

• In your production environment, do you use orchestration tools such as Kubernetes? We are interested in understanding how a trusted server design that depends on virtualization and confidential VMs can be integrated with your application management and orchestration solutions.

• Trusted Execution Environments can rely on hardware-rooted guarantees and CPU-specific protections like encryption of memory in use. Two such CPU platforms available today are AMD's SEV-SNP and Intel's TDX. We welcome feedback on your ability to procure and deploy these technologies in your production environment.

[thegreatfatzby]

I'll provide the best Xandr answers I can, I pinged our techops folks and will consult with them on more interesting pieces:

• We should be able to bring up local builds, this would seem sort of like any vendor deployment where we have to use keys. Right now I believe we have limited number of machines with the appropriate processor tech, but in theory can access them.
• Yes we run on linux, docker and non docker.
• We run quite a lot on BMs.
• I don't know, will talk to our folks.
• Yes we use K8 in particular.
• I'll need feedback from others on the ease of procurement.

[vminet]

Here are some answers for Criteo.

• We will be able to run local builds of the K/V.
• Yes to both. We run a large infrastructure with multiple k8s clusters (tens of thousands of servers).
• Yes. We do run many workloads on bare-metal servers.
• We don't yet have the ability to run confidential VMs but we are exploring it (we would prefer cloud-hypervisor but it doesn’t support all TEEs yet). We have had some experience with running SGX workloads in the past. We don't foresee any major issues.
• Yes, we are using k8s. We are looking at several confidential VMs/containers solutions (for example, CoCo). Do you have one in mind ?
• We are already procuring servers with TEEs such as SEV-SNP or TDX. Right now, I'm not sure we have enough to sustain everything needed for the PSB, but we are ramping up as part of our normal hardware refresh cycle.

In general, we don't foresee any major issues with hosting orchestrated confidential VMs in our environment. That's how we were already planning on doing things.

On the other hand, I would really stress the need for publishing and discussing the security requirements that you will mandate for non-public cloud as soon as possible. Considering the various things that have been throw here and there during meetings, we suspect that this will be the real challenge for most adtech. We would rather clear that uncertainty sooner than later.

[joshuaprismon]

Would you be able to bring up the "local" build of the KV server in your data center (compute environment)? If not, are there specific technical barriers that prevent you from doing so?

Yes. Any software we deploy would need to pass standard security/OSS scans.

In your production environment, do you have the ability to run a workload that is a Linux binary? Or a container?

Yes.

In your production environment, do you have the ability to run workloads on "bare metal" -- that is, not inside a VM?

Yes. though for a variety of reasons, we would prefer to use a containerization isolation strategy rather then a VM isolation pattern.

In your production environment, would the workload (say the KV server) be able to start a virtual machine (VM) using a VMM such as QEMU or Cloud Hypervisor? We ask, because eventually, we would like the "trusted" portion of the server to run inside a "Confidential VM" for confidentiality and security protection.

I want to understand the threat model. the data in the K/V server is unprotected by definition, so the primary difference is bulk egress or network monitoring. Confidential VMs are needed in a multi-tenant environment. Bu in a single-tenant environment, we can secure workloads on the server using standard cgroups and namespaces. Data being exfilled would be a problem on the host as well, due to network passthrough.

do you use orchestration tools such as Kubernetes in your production environment? We are interested in understanding how a trusted server design that depends on virtualization and confidential VMs can be integrated with your application management and orchestration solutions.

K8s is in use. Typically, something like ArgoCD, Spinnaker, or Helm would be used to manage the service's rollout.

Trusted Execution Environments can rely on hardware-rooted guarantees and CPU-specific protections like encryption of memory in use. Two such CPU platforms available today are AMD's SEV-SNP and Intel's TDX. We welcome feedback on your ability to procure and deploy these technologies in your production environment.

TDX is fairly new—it didn't start getting deployed in new hardware until 2022. With hardware approaching 5-+ year deprecation windows, this would mandate new servers for K/V, assuming that the server bill of materials were updated to saphir rapids-based servers once the chips became available.

[prismon]

@palenica - would love to get more feedback here.