Update samples and PreMain for Gramine v1.3 compatibility

cli/cmd/graminePrepare.go

@@ -64,6 +64,8 @@ func newGraminePrepareCmd() *cobra.Command {
 			return addToGramineManifest(fileName)
 		},
 		SilenceUsage: true,
+		Deprecated: "output generated by this command may not compatible with the latest version of Gramine. " +
+			"This command will be removed in a future release of MarbleRun.",
 	}
 
 	return cmd

docs/docs/building-services/gramine.md

@@ -1,34 +1,46 @@
 # Building a service: Gramine
-Running a Gramine app with MarbleRun requires some changes to its manifest. These are explained in the following. See also the [hello world example](https://github.com/edgelesssys/marblerun/tree/master/samples/gramine-hello).
+
+Running a Gramine app with MarbleRun requires some changes to its manifest. These are explained in the following. See also the [hello world example](https://github.com/edgelesssys/marblerun/tree/master/samples/gramine-hello) for a simple introduction, or the [nginx](https://github.com/edgelesssys/marblerun/tree/master/samples/gramine-nginx) and [Redis](https://github.com/edgelesssys/marblerun/tree/master/samples/gramine-redis) examples for more detailed applications.
 
 ## Requirements
+
 First, get Gramine up and running. Gramine is available [as a Debian package](https://github.com/gramineproject/gramine/releases). Alternatively you can follow either the [Building](https://gramine.readthedocs.io/en/latest/devel/building.html) or [Cloud Deployment](https://gramine.readthedocs.io/en/latest/cloud-deployment.html) guide to build and install Gramine from source.
 
 Before running your application, make sure you got the prerequisites for ECDSA remote attestation installed on your system. You can collectively install them with the following command:
+
 ```sh
 sudo apt install libsgx-quote-ex-dev
 ```
+
 ## Configuration
+
 ### Entrypoint and argv
+
 We provide the `premain-libos` executable with the [MarbleRun Releases](https://github.com/edgelesssys/marblerun/releases). It will contact the Coordinator, set up the environment, and run the actual application.
 
-Set the premain executable as the entry point of the Gramine project and place the actual entry point in argv0:
+Set the premain executable as [the entry point](https://gramine.readthedocs.io/en/v1.3/manifest-syntax.html#libos-entrypoint) of the Gramine application and place the actual entry point [in argv0](https://gramine.readthedocs.io/en/v1.3/manifest-syntax.html#command-line-arguments):
+
 ```toml
 libos.entrypoint = "file:premain-libos"
 
 # argv0 needs to contain the name of your executable
-loader.argv0_override = "hello"
+loader.argv = ["hello"]
 
 # add the premain to the list of trusted files
 sgx.trusted_files = [
     # ...
     "file:premain-libos"
 ]
 ```
+
 After the premain is done running, it will automatically spawn your application.
 
 ### Host environment variables
-The premain needs access to some host [environment variables for configuration](../workflows/add-service.md#step-3-start-your-service):
+
+By default, environment variables from the host won't be passed to the application.
+Gramine allows to [pass through whitelisted environment variables from the host](https://gramine.readthedocs.io/en/v1.3/manifest-syntax.html#environment-variables).
+The premain needs access to the following [environment variables for configuration](../workflows/add-service.md#step-3-start-your-service):
+
 ```toml
 loader.env.EDG_MARBLE_TYPE = { passthrough = true }
 loader.env.EDG_MARBLE_COORDINATOR_ADDR = { passthrough = true }
@@ -37,32 +49,49 @@ loader.env.EDG_MARBLE_DNS_NAMES = { passthrough = true }
 ```
 
 ### UUID file
+
 The Marble must be able to store its UUID:
+
 ```toml
-sgx.allowed_files.uuid = "file:uuid"
+sgx.allowed_files = [
+    # ...
+    "file:uuid"
+]
 ```
 
 ### Remote attestation
-The Marble will send an SGX quote to the Coordinator for remote attestation:
+
+The Marble will send an SGX quote to the Coordinator for remote attestation using [DCAP attestation](https://gramine.readthedocs.io/en/v1.3/manifest-syntax.html#attestation-and-quotes):
+
 ```toml
-sgx.remote_attestation = true
+sgx.remote_attestation = "dcap"
 ```
 
 ### Enclave size and threads
+
 The premain process is written in Go. The enclave needs to have enough resources for the Go runtime:
+
 ```toml
 sgx.enclave_size = "1024M"
 sgx.thread_num = 16
 ```
 
 If your application has high memory demands, you may need to increase the size even further.
+
 ### Secret files
+
 A Marble's secrets, e.g. a certificate and private key, can be provisioned as files. You can utilize Gramine's in-memory filesystem [`tmpfs`](https://gramine.readthedocs.io/en/latest/manifest-syntax.html#fs-mount-points), so the secrets will never show up on the host's file system :
+
 ```toml
-fs.mount.secrets.type = "tmpfs"
-fs.mount.secrets.path = "/secrets"
+fs.mounts = [
+    # ...
+    { type = "tmpfs", path = "/secrets" },
+    # ...
+]
 ```
+
 You can specify the files' content in the MarbleRun manifest:
+
 ```javascript
 ...
     "Parameters": {
@@ -74,21 +103,62 @@ You can specify the files' content in the MarbleRun manifest:
 ...
 ```
 
-Note that Gramine also allows to store files encrypted on the host's file system. The so called `protected files` require to initialize the protected files key by writing it hex-encoded to the virtual `protected_files_key` device:
+Gramine also allows to store files [encrypted on the host's file system](https://gramine.readthedocs.io/en/v1.3/manifest-syntax.html#encrypted-files).
 
+```toml
+fs.mounts = [
+  # ...
+  { type = "encrypted", path = "/secrets", uri = "file:/path/to/local/directory", key_name = "[KEY_NAME]" },
+  # ...
+]
 ```
-sgx.protected_files_key = "[16-byte hex value]"
-sgx.protected_files.[identifier] = "[URI]"
+
+Gramine provides access to a pseudo filesystem for [setting the encryption key](https://gramine.readthedocs.io/en/v1.3/attestation.html#low-level-dev-attestation-interface).
+MarbleRun can set up your enclave with keys at runtime by specifying them in the MarbleRun manifest:
+
+```javascript
+...
+    "Parameters": {
+        "Files": {
+            "/dev/attestation/[KEY_NAME]": "{{ raw .Secrets.encryptedFilesKey }}"
+        }
+    }
+...
 ```
-You can see how this key can be initialized with MarbleRun in the [nginx example](https://github.com/edgelesssys/marblerun/tree/master/samples/gramine-nginx).
+
+You can see how this is done in the [nginx example](https://github.com/edgelesssys/marblerun/tree/master/samples/gramine-nginx).
 
 ## Troubleshooting
+
 ### aesm_service returned error: 30
+
 If you receive the following error message on launch:
 
-```
+```sh
 aesm_service returned error: 30
 load_enclave() failed with error -1
 ```
 
 Make sure you installed the Intel AESM ECDSA plugins on your machine. You can do this by installing the `libsgx-quote-dev` package mentioned in the requirements above.
+
+If you are running your application in a container, you will need to mount the aesm socket. The socket is located at `/var/run/aesmd/`.
+
+If you are deploying your application on Kubernetes with the [Intel SGX device plugin](https://intel.github.io/intel-device-plugins-for-kubernetes/cmd/sgx_plugin/README.html) installed, the socket is automatically mounted by setting the `sgx.intel.com/quote-provider: aesmd` annotation for your deployment:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: gramine-marble
+  labels:
+    marblerun/marbletype: gramine-marble
+  annotations:
+    sgx.intel.com/quote-provider: aesmd
+spec:
+  container:
+    - name: gramine-marble
+      image: localhost/gramine-marble
+      resources:
+        limits:
+          sgx.intel.com/epc: 10Mi
+```

docs/styles/Vocab/edgeless/accept.txt

@@ -23,6 +23,7 @@ nonce
 Occlum
 [Pp]remain
 runtime
+Redis
 subcommand
 Tensorflow
 toleration

marble/premain/gramine.go

@@ -9,8 +9,8 @@ package premain
 import (
 	"crypto/sha256"
 	"errors"
-	"io/ioutil"
 	"os"
+	"strings"
 
 	"github.com/edgelesssys/marblerun/coordinator/rpc"
 	"google.golang.org/grpc/credentials"
@@ -24,13 +24,25 @@ func GramineActivate(req *rpc.ActivationReq, coordAddr string, tlsCredentials cr
 		return nil, err
 	}
 
-	// Write the protected files key if present. We must do this "manually" here because premain will write files
-	// in an unspecified order. However, the key must be written before any other protected file is written.
+	// Write encrypted file keys if present. We must do this "manually" here because premain will write files
+	// in an unspecified order. However, the keys must be written before any other encrypted file is written.
+	const atKeyBasePath = "/dev/attestation/keys/"
+	for keyPath, key := range params.Files {
+		if strings.HasPrefix(keyPath, atKeyBasePath) {
+			if err := os.WriteFile(keyPath, key, 0); err != nil {
+				return nil, err
+			}
+			delete(params.Files, keyPath)
+		}
+	}
+
+	// Gramine v1.2 and older use a different key pseudo file system, add this here if present
 	const pfKeyPath = "/dev/attestation/protected_files_key"
 	if key, ok := params.Files[pfKeyPath]; ok {
-		if err := ioutil.WriteFile(pfKeyPath, []byte(key), 0); err != nil {
+		if err := os.WriteFile(pfKeyPath, key, 0); err != nil {
 			return nil, err
 		}
+		delete(params.Files, pfKeyPath)
 	}
 
 	return params, nil

samples/gramine-hello/Makefile

@@ -1,3 +1,5 @@
+EDG_MARBLE_TYPE ?= hello
+
 all: sign
 .PHONY: clean all
 
@@ -24,4 +26,4 @@ premain-libos:
 	chmod u+x premain-libos
 
 run:
-	gramine-sgx hello
+	EDG_MARBLE_TYPE=$(EDG_MARBLE_TYPE) gramine-sgx hello

samples/gramine-hello/README.md

@@ -4,7 +4,7 @@ This example shows how to run a [Gramine](https://github.com/gramineproject/gram
 
 ## Requirements
 
-First, install Gramine on [release v1.0](https://github.com/gramineproject/gramine/releases/tag/v1.0). You will need hardware with Intel SGX support.
+First, install Gramine on [release v1.3](https://github.com/gramineproject/gramine/releases/tag/v1.3.1). You will need hardware with Intel SGX support.
 
 Then, before you can run the example, make sure you got the prerequisites for ECDSA remote attestation installed on your system. You can collectively install them with the following command:
 
@@ -39,8 +39,8 @@ Once the Coordinator instance is running, you can upload the manifest to the Coo
 curl -k --data-binary @manifest.json https://localhost:4433/manifest
 ```
 
-The type of the Marble is defined in the `manifest.json`. In this example, the manifest defines a single Marble, which is called "hello". To run the application, you need to set the `EDG_MARBLE_TYPE` environement variable to that name.
+The type of the Marble is defined in the `manifest.json`. In this example, the manifest defines a single Marble, which is called "hello". To run the application, you need to set the `EDG_MARBLE_TYPE` environment variable to that name.
 
 ```sh
-EDG_MARBLE_TYPE=hello make run
+EDG_MARBLE_TYPE=hello gramine-sgx hello
 ```

samples/gramine-hello/hello.manifest.template

@@ -5,7 +5,7 @@ loader.env.LD_LIBRARY_PATH = "/lib"
 libos.entrypoint = "premain-libos"
 
 # argv0 must be the path to the actual application
-loader.argv0_override = "hello"
+loader.argv = [ "hello" ]
 
 # Forward EDG environment variables, used by MarbleRun
 loader.env.EDG_MARBLE_TYPE = { passthrough = true }
@@ -44,7 +44,7 @@ sgx.allowed_files = [
 ]
 
 # enable DCAP
-sgx.remote_attestation = true
+sgx.remote_attestation = "dcap"
 
 # enclave must have enough memory and threads
 sgx.enclave_size = "1024M"

samples/gramine-nginx/Makefile

@@ -1,33 +1,18 @@
-# Makefile for Nginx:
-#
-# - make                  Build for plain Linux
-# - make DEBUG=1          Build for plain Linux, with Gramine debug output
-# - make SGX=1            Build for SGX
-# - make SGX=1 DEBUG=1    Build for SGX, with Gramine debug output
-#
-# Use `make clean` to remove Gramine-generated files.
-#
-# Use `make distclean` to further remove the Nginx tarball, source code,
-# and installation.
-
 THIS_DIR := $(dir $(lastword $(MAKEFILE_LIST)))
 
 ARCH_LIBDIR ?= /lib/$(shell $(CC) -dumpmachine)
-GRAMINEDIR ?= $(HOME)/gramine
 
 INSTALL_DIR ?= $(THIS_DIR)install
-NGINX_SRC ?= $(THIS_DIR)nginx-1.16.1
-NGINX_SHA256 ?= f11c2a6dd1d3515736f0324857957db2de98be862461b5a542a3ac6188dbe32b
-
-NGINX_MIRRORS ?= \
-    http://nginx.org/download \
-    https://packages.gramineproject.io/distfiles
+NGINX_SRC ?= $(THIS_DIR)nginx-1.22.0
+NGINX_SHA256 ?= b33d569a6f11a01433a57ce17e83935e953ad4dc77cdd4d40f896c88ac26eb53
+NGINX_URL ?= http://nginx.org/download
 
 LISTEN_HOST ?= 127.0.0.1
 LISTEN_PORT ?= 8002
 LISTEN_SSL_PORT ?= 8444
 
 SGX_SIGNER_KEY ?= enclave-key.pem
+SGX ?= 1
 
 ifeq ($(DEBUG),1)
 GRAMINE_LOG_LEVEL = debug
@@ -41,8 +26,6 @@ ifeq ($(SGX),1)
 all: nginx.manifest.sgx nginx.sig nginx.token
 endif
 
-include $(GRAMINEDIR)/Scripts/Makefile.configs
-
 # Note that Gramine doesn't support eventfd() and PR_SET_DUMPABLE, so we manually
 # overwrite these macros in the autogenerated configuration header of Nginx.
 $(INSTALL_DIR)/sbin/nginx: $(NGINX_SRC)/configure
@@ -59,8 +42,8 @@ $(NGINX_SRC)/configure: $(NGINX_SRC).tar.gz
 	tar --touch -xzf $<
 
 $(NGINX_SRC).tar.gz:
-	$(GRAMINEDIR)/CI-Examples/common_tools/download --output $@ --sha256 $(NGINX_SHA256) \
-		$(foreach mirror,$(NGINX_MIRRORS),--url $(mirror)/$(NGINX_SRC).tar.gz)
+	wget -O $@ $(NGINX_URL)/$(NGINX_SRC).tar.gz
+	echo "$(NGINX_SHA256) $@" | sha256sum -c --status
 
 nginx.manifest: nginx.manifest.template
 	gramine-manifest \
@@ -74,19 +57,21 @@ premain-libos:
 	wget https://github.com/edgelesssys/marblerun/releases/latest/download/premain-libos
 	chmod u+x premain-libos
 
-nginx.manifest.sgx: nginx.manifest $(INSTALL_DIR)/sbin/nginx \
+# Make on Ubuntu <= 20.04 doesn't support "Rules with Grouped Targets" (`&:`),
+# see the helloworld example for details on this workaround.
+nginx.manifest.sgx nginx.sig: sgx_sign
+	@:
+
+.INTERMEDIATE: sgx_sign
+sgx_sign: nginx.manifest $(INSTALL_DIR)/sbin/nginx \
 		$(INSTALL_DIR)/conf/nginx-gramine.conf \
 		$(TEST_DATA) \
 		$(INSTALL_DIR)/conf/server.crt \
 		premain-libos
-	@test -s $(SGX_SIGNER_KEY) || \
-	    { echo "SGX signer private key was not found, please specify SGX_SIGNER_KEY!"; exit 1; }
 	gramine-sgx-sign \
-		--key $(SGX_SIGNER_KEY) \
 		--manifest $< \
-		--output $@
-
-nginx.sig: nginx.manifest.sgx
+		--key $(SGX_SIGNER_KEY) \
+		--output $<.sgx
 
 nginx.token: nginx.sig
 	gramine-sgx-get-token --output $@ --sig $<
@@ -99,6 +84,7 @@ $(INSTALL_DIR)/conf/nginx-gramine.conf: nginx-gramine.conf.template $(INSTALL_DI
 	sed -e 's|$$(LISTEN_PORT)|'"$(LISTEN_PORT)"'|g' \
 		-e 's|$$(LISTEN_SSL_PORT)|'"$(LISTEN_SSL_PORT)"'|g' \
 		-e 's|$$(LISTEN_HOST)|'"$(LISTEN_HOST)"'|g' \
+		-e 's|$$(INSTALL_DIR)|'"$(INSTALL_DIR)"'|g' \
 	$< > $@
 
 # HTTP docs: Generating random HTML files in $(INSTALL_DIR)/html/random

samples/gramine-nginx/README.md

@@ -2,7 +2,7 @@
 
 This example is a slightly modified variant of the [Gramine nginx example](https://github.com/gramineproject/gramine/tree/master/CI-Examples/nginx). These changes are required to run it with MarbleRun.
 
-*Prerequisite*: Gramine is installed on [release v1.0](https://github.com/gramineproject/gramine/releases/tag/v1.0) and the original nginx example is working. You will need hardware with Intel SGX support, and the Coordinator must not run in simulation mode.
+*Prerequisite*: Gramine is installed on [release v1.3](https://github.com/gramineproject/gramine/releases/tag/v1.3.1) and the original nginx example is working. You will need hardware with Intel SGX support, and the Coordinator must not run in simulation mode.
 
 To marbleize the example we edited [nginx.manifest.template](nginx.manifest.template). See comments starting with `MARBLERUN` for explanations of the required changes.
 
@@ -29,7 +29,7 @@ Once the Coordinator instance is running, you can upload the manifest to the Coo
 curl -k --data-binary @manifest.json https://localhost:4433/manifest
 ```
 
-The type of the Marble is defined in the `manifest.json`. In this example, the manifest defines a single Marble, which is called "frontend". To run the application, you need to set the `EDG_MARBLE_TYPE` environement variable to that name.
+The type of the Marble is defined in the `manifest.json`. In this example, the manifest defines a single Marble, which is called "frontend". To run the application, you need to set the `EDG_MARBLE_TYPE` environment variable to that name.
 
 ```sh
 EDG_MARBLE_TYPE=frontend gramine-sgx nginx
@@ -38,5 +38,5 @@ EDG_MARBLE_TYPE=frontend gramine-sgx nginx
 From a new terminal, check if nginx is running properly:
 
 ```sh
-curl -k https://localhost:8443
+curl -k https://localhost:8444
 ```