Added snapshot documentation + pipeline example code

Signed-off-by: David Son <[email protected]>

.buildkite/pipeline.yml

@@ -102,6 +102,15 @@ steps:
       queue: "${BUILDKITE_AGENT_META_DATA_QUEUE:-default}"
       distro: "${BUILDKITE_AGENT_META_DATA_DISTRO}"
       hostname: "${BUILDKITE_AGENT_META_DATA_HOSTNAME}"
+
+  - label: ':book: examples'
+    commands:
+      - "sudo -E PATH=$PATH FC_TEST_DATA_PATH=${FC_TEST_DATA_PATH} make -C examples/cmd/snapshotting run"
+      - "sudo -E PATH=$PATH FC_TEST_DATA_PATH=${FC_TEST_DATA_PATH} make -C examples/cmd/snapshotting clean"
+    agents:
+      queue: "${BUILDKITE_AGENT_META_DATA_QUEUE:-default}"
+      distro: "${BUILDKITE_AGENT_META_DATA_DISTRO}"
+      hostname: "${BUILDKITE_AGENT_META_DATA_HOSTNAME}"
 
   - label: ':hammer: tests'
     commands:

docs/snapshotting.md

@@ -0,0 +1,58 @@
+# Snapshotting
+
+Snapshotting is currently supported in the Firecracker Go SDK using Firecracker v1.0.0's API.
+
+Due to [known issues and limitations](https://github.com/firecracker-microvm/firecracker/blob/firecracker-v1.0/docs/snapshotting/snapshot-support.md#known-issues-and-limitations), it is currently not recommended to use snapshots in production.
+
+Snapshots created in this version only save the following:
+- guest memory
+- emulated hardware state (both KVM & Firecracker eumlated hardware)
+
+Each of the above are saved in its own separate file. Anything else is up to the user to restore (tap devices, drives, etc.).
+
+In particular, drives must be in the same location as they were when loading the snapshot. Otherwise, the API call will fail. Changing said drive file can lead to some unexpected behaviors, so it is recommended to make minimal changes to the drive.
+
+Snapshots can only be loaded upon device startup. Upon loading the snapshot, the emulated hardware state is restored, and normal VM activites can resume right where they left off.
+
+Read more in-depth documentation on Firecracker's snapshotting tool [here](https://github.com/firecracker-microvm/firecracker/blob/firecracker-v1.0/docs/snapshotting/snapshot-support.md).
+
+## Using Snapshots via Firecracker Go SDK
+
+Snapshots can be created via a machine object's `CreateSnapshot()` function. The call will make the snapshot files at the specified paths, with the memory saved to `memPath`, and the machine state saved to `snapPath`. The VM must be paused beforehand.
+
+```
+import (
+	sdk "github.com/firecracker-microvm/firecracker-go-sdk"
+)
+
+...
+
+ctx := context.Background()
+cfg := sdk.Config{
+
+    ...
+
+}
+
+m, _ := sdk.NewMachine(ctx, cfg)
+m.Start(ctx)
+m.PauseVM(ctx)
+m.CreateSnapshot(ctx, memPath, snapPath)
+```
+
+The snapshot can be loaded at any later time at startup of a machine via the machine's `Start()` function, using `WithSnapshot()` as an option. The VM must then be resumed before attempting to use it.
+
+```
+ctx := context.Background()
+cfg := sdk.Config{
+
+    ...
+
+}
+m, _ := sdk.NewMachine(ctx, cfg)
+
+m.Start(ctx, sdk.WithSnapshot(memPath, snapPath))
+m.ResumeVM(ctx)
+```
+
+Check out [examples/cmd/snapshotting](../examples/cmd/snapshotting) for a quick example that can be run on your machine.

examples/cmd/snapshotting/Makefile

@@ -0,0 +1,73 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You may
+# not use this file except in compliance with the License. A copy of the
+# License is located at
+#
+# 	http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is distributed
+# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+# express or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+
+RELEASE_URL=https://github.com/firecracker-microvm/firecracker/releases
+VER=v1.0.0
+
+ARCH=$(shell uname -m)
+GID = $(shell id -g)
+
+PWD=$(shell pwd)
+FC_TEST_DATA_PATH?=$(PWD)
+
+all: plugins image vmlinux firecracker
+
+plugins: bin/tc-redirect-tap bin/ptp bin/host-local | bin
+
+bin:
+	mkdir -p bin
+
+bin/tc-redirect-tap: bin
+	GO111MODULE=off GOBIN=$(PWD)/bin \
+	go get github.com/awslabs/tc-redirect-tap/cmd/tc-redirect-tap
+
+bin/ptp: bin
+	GO111MODULE=off GOBIN=$(PWD)/bin \
+	go get github.com/containernetworking/plugins/plugins/main/ptp
+
+bin/host-local: bin
+	GO111MODULE=off GOBIN=$(PWD)/bin \
+	go get github.com/containernetworking/plugins/plugins/ipam/host-local
+
+image:
+ifeq ($(GID), 0)
+	- cp ${FC_TEST_DATA_PATH}/root-drive-with-ssh.img root-drive-with-ssh.img
+	- cp ${FC_TEST_DATA_PATH}/root-drive-ssh-key root-drive-ssh-key
+	$(MAKE) root-drive-with-ssh.img root-drive-ssh-key
+else
+	$(error unable to place ssh key without root permissions)
+endif
+
+vmlinux:
+	curl --location -o vmlinux https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/${ARCH}/kernels/vmlinux.bin
+
+firecracker:
+	curl -L ${RELEASE_URL}/download/${VER}/firecracker-${VER}-${ARCH}.tgz | tar -xz
+	mv release-${VER}-${ARCH}/firecracker-${VER}-${ARCH} firecracker
+	rm -rf release-${VER}-${ARCH}
+
+root-drive-with-ssh.img root-drive-ssh-key:
+	- mkdir temp
+	- git clone https://github.com/firecracker-microvm/firecracker temp
+	temp/tools/devtool build_rootfs
+	cp temp/build/rootfs/bionic.rootfs.ext4 root-drive-with-ssh.img
+	cp temp/build/rootfs/ssh/id_rsa root-drive-ssh-key
+	rm -rf temp
+
+run: all
+	go run example_demo.go
+
+clean:
+	rm -rf bin firecracker root-drive-ssh-key root-drive-with-ssh.img vmlinux
+
+.PHONY: all clean image plugins run

examples/cmd/snapshotting/README.md

@@ -0,0 +1,81 @@
+# Snapshotting demo
+
+This example shows snapshotting in action by sending a marker to a VM via a running process (in this case `sleep 422`), snapshotting the VM, closing it, loading and starting a new machine via the same snapshot, and checking for the marker.
+
+This test requires both KVM and root access.
+
+## Running the test
+
+Run this test by first running
+
+```
+sudo -E env PATH=$PATH make all
+```
+
+followed by
+
+```
+sudo -E env PATH=$PATH go run example_demo.go
+```
+
+Alternatively, to do both of the above,
+```
+sudo -E env PATH=$PATH make run
+```
+
+Note the user PATH variable is different from the root user's PATH variable, hence the need for `-E env PATH=$PATH`.
+
+Upon running, the VM logs will be printed to the console, as well as the IP of the VM. It will then show that it is sending the marker (in our case, `sleep 422`).
+
+Afterwards, the snapshot is created and the machine is terminated. The snapshot files are saved in the snapshotssh folder created in the directory.
+
+Then, a new machine is created, booted with the snapshot that was just taken, and the IP of the VM will once again be printed to the console (which should be the same as the last machine). The output of searching for the marker (in our case `ps -aux | grep "sleep 422"`) is then printed to the console and the user can confirm that the snapshot loaded properly.
+
+To run this test more dynamically, you can pause the execution of the program after starting the machine (i.e. after the call to m.Start() and the IP is shown on the screen).
+
+```
+err = m.Start()
+
+...
+
+vmIP := m.Cfg.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IPAddr.IP.String()
+fmt.Printf("IP of VM: %v\n", vmIP)
+fmt.Scanln() // block, allows you to ssh from another shell
+
+...
+
+err = m.Start()
+
+...
+
+fmt.Println("Snapshot loaded")
+fmt.Printf("IP of VM: %v\n", ipToRestore)
+fmt.Scanln() // block, allows you to ssh from another shell
+```
+
+```
+sudo ssh -i root-drive-ssh-key root@[ip]
+```
+
+Pressing enter resumes execution of the program.
+
+You can remove dependencies via a simple `make clean`.
+
+```
+sudo make clean
+```
+
+## Issues
+
+You may encounter an issue where the image does not build properly. This is often indicated via the following near the end of terminal output:
+
+```
+umount: /firecracker/build/rootfs/mnt: not mounted.
+```
+
+This is due to an issue in Firecracker's devtool command used to dynamically create an image. Fixing this is often as simple as rerunning the command.
+
+```
+sudo rm -rf root-drive-with-ssh.img root-drive-ssh-key
+sudo make image
+```