MZOCI - Make Zarf OCI (Take #2)

.github/workflows/scan-labels.yml

@@ -9,4 +9,4 @@ jobs:
     steps:
       - uses: yogevbd/[email protected]
         with:
-          BANNED_LABELS: "needs-docs,needs-tests,needs-adr,needs-git-sign-off"
+          BANNED_LABELS: "needs-docs,needs-tests,needs-adr,needs-git-sign-off,needs-walkthrough"

adr/0014-oci-publish.md

@@ -0,0 +1,60 @@
+# 14. Zarf Packages as OCI Artifacts
+
+Date: 2023-03-10
+
+## Status
+
+Accepted
+
+## Context
+
+Zarf packages are currently only available if built locally or through manual file transfers. This is not a scalable way to distribute packages. We wanted to find a way to distribute and publish packages in a way that is easily consumable for the majority of users. When considering the goal of being able to share packages, security and trust are very important considerations. We wanted our publishing solution and architecture changes to keep in mind signing of packages and the ability to verify the integrity of packages.
+
+We know we are successful when:
+
+1. (Priority) Users can use Zarf to natively publish a Zarf package to an OCI compliant registry
+2. (Secondary goal) Package creators can sign Zarf packages to enable package deployers can trust a packages supply chain security
+
+## Decision
+
+We decided that changing the structure of Zarf packages to be an OCI artifact would be the best way to distribute and publish packages as registries are already an integral part of the container ecosystem.
+
+## Implementation
+
+A handful of changes were introduced to the structure of Zarf packages.
+
+```text
+zarf-package-adr-arm64.tar.zst
+├── checksums.txt
+├── components
+│   └── [...].tar
+├── images
+│   ├── index.json
+│   ├── oci-layout
+│   └── blobs
+│       └── sha256
+│           └── ... # OCI image layers
+├── sboms.tar
+└── zarf.yaml
+```
+
+- Each component folder is now a tarball instead of a directory
+  - This enables us to treat each component as a layer within the package artifact
+- Images are now stored in a flattened state instead of an images.tar file
+  - This enables us to keep each image layer as a layer within the package artifact (allowing for server side de-duping)
+- SBOM files are now stored in a tarball instead of a directory
+  - This enables us to treat the SBOM artifacts as a single layer within the package artifact
+
+With this new structure in place, we can now publish Zarf packages as OCI artifacts. Under the hood this implements the `oras` Go library using Docker's authentication system. For interacting with these packages, the `oci://` package path prefix has been added (ex. `zarf package publish oci://...`).
+
+For an example of this in action, please see the [walkthrough](./docs/../../docs/13-walkthroughs/6-publish-and-deploy.md).
+
+## Consequences
+
+Backwards compatibility was an important considering when making these changes. We had to implement logic to make sure a new version of the Zarf binary could still operate with older versions of Zarf packages.
+
+At the moment we are testing the backwards compatibility by virtue of maintaining the `./src/test/e2e/27_cosign_deploy_test.go` where we are deploying an old Zarf package via `sget`.
+
+One thing we may want to look at more in the future is how we can get more intricate tests around the backwards compatibility.
+
+The reason why testing backwards compatibility is difficult is because this isn't a `zarf.yaml` schema change (like we had recently with the 'Scripts to Actions' PR) but an compiled package architecture change. This means that we will either need to maintain an 'old' Zarf package that will follow future `zarf.yaml` schema changes OR we maintain a modified Zarf binary that creates the old package structure.

docs/13-walkthroughs/6-publish-and-deploy.md

@@ -0,0 +1,219 @@
+# Using OCI to Store & Deploy Zarf Packages
+
+## Introduction
+
+In this walkthrough, we are going to run through how to publish a Zarf package to an [OCI](https://github.com/opencontainers/image-spec) compliant registry, allowing end users to pull and deploy packages without needing to build locally, or transfer the package to their environment.
+
+## Prerequisites
+
+For following along locally, please ensure the following prerequisites are met:
+
+1. Zarf binary installed on your `$PATH`: ([Install Instructions](../3-getting-started.md#installing-zarf))
+2. Access to a [Registry supporting the OCI Distribution Spec](https://oras.land/implementors/#registries-supporting-oci-artifacts), this walkthrough will be using Docker Hub
+
+## Setup
+
+This walkthrough will require a registry to be configured (see [prerequisites](#prerequisites) for more information).  The below sets up some variables for us to use when logging into the registry:
+
+```bash
+# Setup some variables for the registry we will be using
+$ REGISTRY=docker.io
+$ set +o history
+$ REGISTRY_USERNAME=<username> # <-- replace with your username
+$ REPOSITORY_URL=$REGISTRY/$REGISTRY_USERNAME
+$ REGISTRY_SECRET=<secret> # <-- replace with your password or auth token
+$ set -o history
+```
+
+With those set, you can tell Zarf to login to your registry with the following:
+
+```bash
+$ echo $REGISTRY_SECRET | zarf tools registry login $REGISTRY --username $REGISTRY_USERNAME --password-stdin
+
+2023/03/07 23:03:16 logged in via /home/zarf/.docker/config.json
+```
+
+:::note
+
+If you do not have the Docker CLI installed, you may need to create a Docker compliant auth config file manually:
+
+```bash
+$ mkdir -p ~/.docker
+$ AUTH=$(echo -n "$REGISTRY_USERNAME:$REGISTRY_SECRET" | base64)
+# Note: If using Docker Hub, the registry URL is `https://index.docker.io/v1/` for the auth config
+$ cat <<EOF > ~/.docker/config.json
+{
+  "auths": {
+    "$REGISTRY": {
+      "auth": "$AUTH"
+    }
+  }
+}
+EOF
+```
+
+:::
+
+## Publish Package
+
+First, create a valid Zarf package definition (`zarf.yaml`), with the `metadata.version` key set.
+
+```yaml
+# Make a new directory to work in
+$ mkdir -p zarf-publish-walkthrough && cd zarf-publish-walkthrough
+
+# For this walkthrough we will use the `helm-oci-chart` example package
+# located here: https://github.com/defenseunicorns/zarf/blob/main/examples/helm-oci-chart/zarf.yaml
+$ cat <<EOF > zarf.yaml
+kind: ZarfPackageConfig
+metadata:
+  name: helm-oci-chart
+  description: Deploy podinfo using a Helm OCI chart
+  # Note: In order to publish, the package must have a version
+  version: 0.0.1
+
+components:
+  - name: helm-oci-chart
+    required: true
+    charts:
+      - name: podinfo
+        version: 6.3.3
+        namespace: helm-oci-demo
+        url: oci://ghcr.io/stefanprodan/charts/podinfo
+    images:
+      - "ghcr.io/stefanprodan/podinfo:6.3.3"
+EOF
+```
+
+Create the package locally:
+
+[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_create.md)
+
+```bash
+# Create the package (interactively)
+$ zarf package create .
+# Make these choices at the prompts:
+# Create this Zarf package? Yes
+# Please provide a value for "Maximum Package Size" 0
+```
+
+Then publish the package to the registry:
+
+:::note
+
+Your package tarball may be named differently based on your machine's architecture.  For example, if you are running on an AMD64 machine, the tarball will be named `zarf-package-helm-oci-chart-amd64-0.0.1.tar.zst`.
+
+:::
+
+[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_publish.md)
+
+```bash
+$ zarf package publish zarf-package-helm-oci-chart-arm64-0.0.1.tar.zst oci://$REPOSITORY_URL
+
+...
+
+  •  Publishing package to $REPOSITORY_URL/helm-oci-chart:0.0.1-arm64
+  ✔  Prepared 14 layers
+  ✔  515aceaacb8d images/index.json
+  ✔  4615b4f0c1ed zarf.yaml
+  ✔  1300d6545c84 sboms.tar
+  ✔  b66dbb27a733 images/oci-layout
+  ✔  46564f0eff85 images/blobs/sha256/46564f0...06008f762391a7bb7d58f339ee
+  ✔  4f4fb700ef54 images/blobs/sha256/4f4fb70...b5577484a6d75e68dc38e8acc1
+  ✔  6ff8f4799d50 images/blobs/sha256/6ff8f47...4bc00ec8b988d28cef78ea9a5b
+  ✔  74eae207aa23 images/blobs/sha256/74eae20...fcb007d3da7b842313f80d2c33
+  ✔  a9eaa45ef418 images/blobs/sha256/a9eaa45...6789c52a87ba5a9e6483f2b74f
+  ✔  8c5b695f4724 images/blobs/sha256/8c5b695...014f94c8d4ea62772c477c1e03
+  ✔  ab67ffd6e92e images/blobs/sha256/ab67ffd...f8c9d93c0e719f6350e99d3aea
+  ✔  b95c82728c36 images/blobs/sha256/b95c827...042a9c5d84426c1674044916d4
+  ✔  e2b45cdcd8bf images/blobs/sha256/e2b45cd...000f1bc1695014e38821dc675c
+  ✔  79be488a834e components/helm-oci-chart.tar
+  ✔  aed84ba183e7 [application/vnd.oci.artifact.manifest.v1+json]
+  ✔  Published $REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 [application/vnd.oci.artifact.manifest.v1+json]
+
+  •  To inspect/deploy/pull:
+  •  zarf package inspect oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64
+  •  zarf package deploy oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64
+  •  zarf package pull oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64
+```
+
+:::note
+
+The name and reference of this OCI artifact is derived from the package metadata, e.g.: `helm-oci-chart:0.0.1-arm64`
+
+To modify, edit `zarf.yaml` and re-run `zarf package create .`
+
+:::
+
+## Inspect Package
+
+[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_inspect.md)
+
+Inspecting a Zarf package stored in an OCI registry is the same as inspecting a local package and has the same flags:
+
+```yaml
+$ zarf package inspect oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64
+---
+kind: ZarfPackageConfig
+metadata:
+  name: helm-oci-chart
+  description: Deploy podinfo using a Helm OCI chart
+  version: 0.0.1
+  architecture: arm64
+build:
+  terminal: minimind.local
+  user: whoami
+  architecture: arm64
+  timestamp: Tue, 07 Mar 2023 14:27:25 -0600
+  version: v0.25.0-rc1-41-g07d61ba7
+  migrations:
+    - scripts-to-actions
+components:
+  - name: helm-oci-chart
+    required: true
+    charts:
+      - name: podinfo
+        url: oci://ghcr.io/stefanprodan/charts/podinfo
+        version: 6.3.3
+        namespace: helm-oci-demo
+    images:
+      - ghcr.io/stefanprodan/podinfo:6.3.3
+```
+
+## Deploy Package
+
+[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md)
+
+Deploying a package stored in an OCI registry is nearly the same experience as deploying a local package:
+
+```bash
+# Due to the length of the console output from this command,
+# it has been omitted from this walkthrough
+$ zarf package deploy oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64
+# Make these choices at the prompts:
+# Deploy this Zarf package? Yes
+
+$ zarf packages list
+
+    Package        | Components
+    helm-oci-chart | [helm-oci-chart]
+    init           | [zarf-injector zarf-seed-registry zarf-registry zarf-agent git-server]
+```
+
+## Pull Package
+
+[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_pull.md)
+
+Packages can be saved to the local disk in order to deploy a package multiple times without needing to fetch it every time.
+
+```bash
+# go home so we don't clobber our currently local built package
+$ cd ~
+$ mkdir -p zarf-packages && cd zarf-packages
+
+$ zarf package pull oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64
+
+# use vim if you want to inspect the tarball's contents without decompressing it
+$ vim zarf-package-helm-oci-chart-arm64-0.0.1.tar.zst
+# don't forget to escape w/ `:q`
+```

docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package.md

@@ -28,5 +28,7 @@ Zarf package commands for creating, deploying, and inspecting packages
 * [zarf package deploy](zarf_package_deploy.md)	 - Use to deploy a Zarf package from a local file or URL (runs offline)
 * [zarf package inspect](zarf_package_inspect.md)	 - Lists the payload of a Zarf package (runs offline)
 * [zarf package list](zarf_package_list.md)	 - List out all of the packages that have been deployed to the cluster
+* [zarf package publish](zarf_package_publish.md)	 - Publish a Zarf package to a remote registry
+* [zarf package pull](zarf_package_pull.md)	 - Pull a Zarf package from a remote registry and save to the local file system
 * [zarf package remove](zarf_package_remove.md)	 - Use to remove a Zarf package that has been deployed already
 

docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md

@@ -14,12 +14,13 @@ zarf package deploy [PACKAGE] [flags]
 ## Options
 
 ```
-      --components string    Comma-separated list of components to install.  Adding this flag will skip the init prompts for which components to install
-      --confirm              Confirm package deployment without prompting
-  -h, --help                 help for deploy
-      --set stringToString   Specify deployment variables to set on the command line (KEY=value) (default [])
-      --sget string          Path to public sget key file for remote packages signed via cosign
-      --shasum string        Shasum of the package to deploy. Required if deploying a remote package and "--insecure" is not provided
+      --components string     Comma-separated list of components to install.  Adding this flag will skip the init prompts for which components to install
+      --confirm               Confirm package deployment without prompting
+  -h, --help                  help for deploy
+      --oci-concurrency int   Number of concurrent layer operations to perform when interacting with a remote package. (default 3)
+      --set stringToString    Specify deployment variables to set on the command line (KEY=value) (default [])
+      --sget string           Path to public sget key file for remote packages signed via cosign
+      --shasum string         Shasum of the package to deploy. Required if deploying a remote package and "--insecure" is not provided
 ```
 
 ## Options inherited from parent commands

docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_publish.md

@@ -0,0 +1,38 @@
+# zarf package publish
+<!-- Auto-generated by docs/gen-cli-docs.sh -->
+
+Publish a Zarf package to a remote registry
+
+```
+zarf package publish [PACKAGE] [REPOSITORY] [flags]
+```
+
+## Examples
+
+```
+  zarf package publish my-package.tar oci://my-registry.com/my-namespace
+```
+
+## Options
+
+```
+  -h, --help                  help for publish
+      --oci-concurrency int   Number of concurrent layer operations to perform when interacting with a remote package. (default 3)
+```
+
+## Options inherited from parent commands
+
+```
+  -a, --architecture string   Architecture for OCI images
+      --insecure              Allow access to insecure registries and disable other recommended security enforcements. This flag should only be used if you have a specific reason and accept the reduced security posture.
+  -l, --log-level string      Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info")
+      --no-log-file           Disable log file creation
+      --no-progress           Disable fancy UI progress bars, spinners, logos, etc
+      --tmpdir string         Specify the temporary directory to use for intermediate files
+      --zarf-cache string     Specify the location of the Zarf cache directory (default "~/.zarf-cache")
+```
+
+## SEE ALSO
+
+* [zarf package](zarf_package.md)	 - Zarf package commands for creating, deploying, and inspecting packages
+

docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_pull.md

@@ -0,0 +1,38 @@
+# zarf package pull
+<!-- Auto-generated by docs/gen-cli-docs.sh -->
+
+Pull a Zarf package from a remote registry and save to the local file system
+
+```
+zarf package pull [REFERENCE] [flags]
+```
+
+## Examples
+
+```
+  zarf package pull oci://my-registry.com/my-namespace/my-package:0.0.1-arm64
+```
+
+## Options
+
+```
+  -h, --help                  help for pull
+      --oci-concurrency int   Number of concurrent layer operations to perform when interacting with a remote package. (default 3)
+```
+
+## Options inherited from parent commands
+
+```
+  -a, --architecture string   Architecture for OCI images
+      --insecure              Allow access to insecure registries and disable other recommended security enforcements. This flag should only be used if you have a specific reason and accept the reduced security posture.
+  -l, --log-level string      Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info")
+      --no-log-file           Disable log file creation
+      --no-progress           Disable fancy UI progress bars, spinners, logos, etc
+      --tmpdir string         Specify the temporary directory to use for intermediate files
+      --zarf-cache string     Specify the location of the Zarf cache directory (default "~/.zarf-cache")
+```
+
+## SEE ALSO
+
+* [zarf package](zarf_package.md)	 - Zarf package commands for creating, deploying, and inspecting packages
+