s390x: SecureExecution enablement

Signed-off-by: Nikita Dubrovskii <[email protected]>
Co-authored-by: Benjamin Gilbert <[email protected]>

os/20211014-s390x-secure-execution.md

@@ -0,0 +1,236 @@
+# IBM Secure Execution for Linux
+---
+
+# Overview
+
+[IBM Secure Execution for Linux®](https://www.ibm.com/docs/en/linux-on-systems?topic=virtualization-introducing-secure-execution-linux) is a hardware-based security technology that is
+built into the IBM z15™ and LinuxONE III generation systems.
+The goal of Secure Execution is to protect data of workloads that run in a KVM guest from being inspected or modified by the hypervisor. In particular, no hardware administrator, no KVM code, and no KVM administrator can access the data in a guest that was started as an IBM Secure Execution guest.
+
+IBM Secure Execution provides the following benefits:
+ - Instead of relying on deterrence by using extensive audit tracks, IBM Secure Execution provides technology-enforced security rather than process or audit-based security.
+ - It is possible to securely deploy workloads in the cloud.
+ - As a cloud provider that uses IBM Secure Execution, you can attract sensitive workloads that, formerly, were restricted to the workload owner's system.
+ - As a secure workload owner, you know that your workload is run in a secure manner, even outside your data center.
+ - As a secure workload owner, you can choose where to run your workload, independently of the security level required.
+
+---
+
+# Theory of operation
+
+Secure Execution is designed to provide scalable isolation for individual workloads to help protect them from not
+only external attacks, but also insider threats. In traditional x86 ring architectures,
+the host can access the memory and data of guest applications freely, leading to the
+potential for malicious software to be proliferated throughout the entire system.
+Isolation between host and guest environments is necessary to help prevent system compromise.
+Users can enable Secure Execution feature on IBM zKVM Host and Guest VMs to isolate them from each other.
+
+That means:
+1. An encrypted Linux image, called `sdboot`, (kernel + initramfs + cmdline) is created using:
+ - universal host public key provided by IBM. This happens only during CoreOS Qemu image build.
+ - customer-specific host's public key(s). This happens only in runtime
+2. kernel, initramfs and cmdline couldn't be modified without regeneration of `sdboot` image
+3. zKVM host cannot access memory of secured guest VM
+4. Guest VMs cannot access memory of other secured guest VMs
+5. If anyone tries to access memory of secured guest VM, the attempt will get blocked at the HW level
+
+Since the encryption private keys are stored on the IBM Z hardware and firmware, the encrypted image can only be executed in a virtual machine on the host(s) it has been prepared for, and the image can’t be decrypted or tampered with outside of the designated host(s).
+
+## Specific attacks being addressed
+- protect guests against the system
+- protect guests against a Linux Admin (on the host)
+- protect the data services use within a secured guests from host admin and/or other guests
+
+## Specific attacks not being addressed
+- any attack over Internet on some service(s) running within Guest
+- usage of the fake ignition config for guest provisioning
+
+> **Note:**
+> `Hostkey` is generated by IBM and is stored somewhere in HW, the key is unique for each IBM Z mainframe.
+> Image signed with a key (or set of keys) could be used only on machine with the identical key.
+
+> **Note:**
+> Only production hostkey(s) could be verified (that a host key document is genuine) using [check_hostkeydoc](https://raw.githubusercontent.com/ibm-s390-tools/s390-tools/master/genprotimg/samples/check_hostkeydoc) tool. In a later version of `genprotimg` this function will
+be performed by the tooling itself.
+
+# High-level overview
+
+The goal of Secure Execution is to provide customer the highest level of isolation and security for each OCP node.
+To achive that new type of artifact for s390x would be available to download and use - Secure Execution Ready Qcow2 Image:
+ - Image comes with `sdboot` generated using universal key from IBM. Key itself is not known to public and used by RH/IBM during build of RHCOS
+ - Image couldn't be modified, by anyone except `ignition` during lifetime
+ - It won't possible to inspect and modify (using `guestfish` or other tools) contents of image's `/` and `/boot` filesystems
+ - It's only possible to run this image on zKVM host with enabled Secure Execution support
+ - It's not possible to disable Secure Execution once it was enabled
+
+To use Secure Execution customer must have hostkeys for host machine(s) and specify them in `ignition` config.
+
+## Prerequisites
+
+Secure Execution is supported only starting with z15 and only for zKVM guests.
+In order to use it zKVM host's kernel-cmdline should have `prot_virt=1 swiotlb=262144` options.
+
+KVM guests also require some additional options:
+- disk's driver should have `iommu=on` option:
+```xml
+<disk type='file' device='disk'>
+      <driver name='qemu' type='qcow2' iommu='on'/>
+      <target dev='vda' bus='virtio'/>
+</disk>
+```
+- network interfaces should have `iommu=on` option:
+```xml
+ <interface type='network'>
+      <model type='virtio'/>
+      <driver name='qemu' iommu='on'/>
+</interface>
+```
+- `memballoon` should be disabled:
+```xml
+<memballoon model='none'/>
+```
+
+Or same options for cmdline execution:
+```
+qemu-system-s390x \
+     -machine s390-ccw-virtio,accel=kvm \
+     -m 4096 -smp 2 \
+     -nographic \
+     -drive if=none,id=hda,file=/path/to/image.qcow2,auto-read-only=off,cache=unsafe -device virtio-blk,iommu_platform=on,drive=hda \
+     -netdev user,id=eth,hostfwd=tcp::2222-:22 -device virtio-net-ccw,iommu_platform=on,netdev=eth
+```
+
+# Detailed overview
+
+To achive goals listed above not only a special qcow2 image is generated, but also some additional steps during system's lifetime are required.
+
+##  Build
+ To build Secure Execution Ready image new command is added to `coreos-assembler`:
+ ```
+ cosa buildextend-secex --hostkey UNIVERSAL_KEY.crt
+ ```
+ Which is similar to `cosa buildextend-qemu` and also generates qcow2 image.
+
+The qcow2 image comes with `sdboot` installed into its own partition, and dm-verity setup for boot and rootfs filesystems, where hashes are part of kernel's cmdline.
+Here are partitions of the Secure Execution qcow2 image provided by RH/IBM:
+
+```
+NAME     PARTLABEL TYPE  MOUNTPOINT
+vda                disk  /
+vdb                disk
+|-vdb1   se        part
+|-vdb3   boot      part
+| `-boot           crypt /tmp/rootfs/boot
+|-vdb4   root      part
+| `-root           crypt /tmp/rootfs
+|-vdb5   boothash  part
+| `-boot           crypt /tmp/rootfs/boot
+`-vdb6   roothash  part
+  `-root           crypt /tmp/rootfs
+
+```
+
+- `se`   - This is a new partition `ext4`, where we store a `sdboot` encrypted image
+- `boot` - boot partition is the same to one on reqular qcow2 image
+- `root` - rootfs partition is the same to one on reqular qcow2 image
+- `boothash` - partition for settting up `dm-verity` on `boot`
+- `roothash` - partition for settting up `dm-verity` on `boot`
+
+`dm-verity` is used to ensure, that pristine qcow image wasn't modified after build.
+
+> **Note:**
+> Further partitioning of boot disk with those 3 partitons is not supported, but
+rootfs could be grown. That means, that when required user should enlarge `secex-qemu.qcow2` image before `firstboot`:
+```
+qemu-img resize /path/to/img.qcow2 +50G
+```
+
+## Firstboot
+
+During guest provisioning new hostkey(s) is/are fetched and used for generation of `sdboot`. User should specify them within ignition config with pre-defined name template: `/etc/se-hostkeys/ibm-z-hostkey-N` where `N` is a key number, several hostkeys could be used:
+
+```yaml
+storage:
+  files:
+    -
+      path: /etc/se-hostkeys/ibm-z-hostkey-1
+      overwrite: true
+      contents:
+        source: https://url.com/key1
+```
+
+System would also set a new random LUKS keys for both `boot` and `root` filesystems, so each node has its own LUKS keys.
+
+## Lifetime
+
+Secure Execution is enabled by 2 tools provided by IBM's s390-utils package:
+- `genprotimg` - tool to generate `sdboot`
+- `zipl` - tool to install new bootloader
+
+Despite `zipl` is well known and used by CoreOS, it couldn't be executed same way as before. When system runs in Secure Execution, `ostree` and `rdcore` automatically detect that and perform all required steps.
+That means, that user won't notice any difference, both mentioned components would automatically regenerate `sdboot` image and
+install it with `zipl` whenever required.
+
+# Implementation steps
+
+## Step 1: CoreOS with existing patches to `libostree` could be configured to perform in Secure Execution mode.
+
+Customer's public hostkey(s) should be placed into `/etc/se-hostkeys/` using name templates `ibm-z-hostkey-N`. Secure Execution is enabled by hands following https://www.ibm.com/docs/en/linux-on-systems?topic=linux-introduction.
+
+## Step 2: New `qemu-secex.qcow2` build artifact for CoreOS with enabled Secure Execution.
+
+`coreos-xyz-qemu-secex.qcow2` image with `dm-verity` setup for `/boot` and `rootfs` is available for download. This is done by using new `buildextend-secex` `coreos-assembler` command, which:
+ - adds new `/se` partition for `sdboot`
+ - adds `roothash` and `boothash` partitions for [dm-verity](https://docs.kernel.org/admin-guide/device-mapper/verity.html) setup
+ - calculates hashes for `/boot` and `/` and appends them to kernel's cmdline
+ - generates `/se/sdboot` encrypted by universal public hostkey (valid for all Z15 and newer machines)
+ - `zipl`es `/se/sdboot`, so `qemu-secex.qcow2` image could only be run with Secure Execution
+
+Customer should add his public hostkeys(s) into `ignition` config.
+
+## Step 3: During firstboot `/boot` and `rootfs` are encrypted with randomly generated LUKS keys; `integrity` option is enabled
+
+During firstboot `/boot` and `/` are luks encrypted with randomly generated luks keys. This ensures, that each CoreOS isntance has its own LUKS keys and also makes it not possible to read the contents of qcow2 image just by inspecting (`libguestfish`) it. Any modifications of qcow image will be detected in runtime by [dm-integrity](https://docs.kernel.org/admin-guide/device-mapper/dm-integrity.html).
+
+
+After provisioning, new disk is like:
+```
+NAME         PARTLABEL TYPE  MOUNTPOINT
+vdb                    disk
+|-vdb1       se        part
+|-vdb3       boot      part
+| `-boot_dif           crypt
+|   `-boot             crypt /boot
+`-vdb4       root      part
+  `-root_dif           crypt
+    `-root             crypt /sysroot
+
+```
+
+- `se`   - partition for newly generated`sdboot` encrypted image
+- `boot` - LUKS encrypted boot partition
+- `root` - LUKS encrypted root partition
+- `boot_dif` - `dm-integrity` for `/boot`
+- `root_dif` - `dm-integrity` for `/root`
+
+## Step 4: Ignition protection
+
+The goal is to prevent reading (to protect any secret stored in the Ignition config) and modification of the config.
+
+To achive this we do:
+- Generate at cosa compose time a pair of GPG keys:
+  - The private key is part of the `sdboot` image
+  - The public key is part of the build metadata
+- Disable Ignition logging to the console (e.g. serial and VGA)
+- Add a hook to emergency shell to remove private key and Ignition config before entering the shell
+- Disable local login
+
+User has to fetch the public GPG key for the `secex-qemu.qcow2` image and encrypt his Ignition config with  the key:
+  ```
+  gpg --recipient-file /path/to/ignition.gpg.pub --yes --output /path/to/config.ign.gpg --armor --encrypt /path/to/config.ign
+  ```
+And than instead of `serial=ignition` user uses `serial=ignition_crypted`:
+  ```
+  -drive if=none,id=ignition,format=raw,file=/path/to/config.ign.gpg,readonly=on
+  -device virtio-blk,serial=ignition_crypted,iommu_platform=on,drive=ignition
+  ```