diff --git a/.github/workflows/scan-labels.yml b/.github/workflows/scan-labels.yml index 00915aa1f2..a28854d62f 100644 --- a/.github/workflows/scan-labels.yml +++ b/.github/workflows/scan-labels.yml @@ -9,4 +9,4 @@ jobs: steps: - uses: yogevbd/enforce-label-action@2.2.2 with: - BANNED_LABELS: "needs-docs,needs-tests,needs-adr,needs-git-sign-off" + BANNED_LABELS: "needs-docs,needs-tests,needs-adr,needs-git-sign-off,needs-walkthrough" diff --git a/adr/0014-oci-publish.md b/adr/0014-oci-publish.md new file mode 100644 index 0000000000..23e84dcad3 --- /dev/null +++ b/adr/0014-oci-publish.md @@ -0,0 +1,60 @@ +# 14. Zarf Packages as OCI Artifacts + +Date: 2023-03-10 + +## Status + +Accepted + +## Context + +Zarf packages are currently only available if built locally or through manual file transfers. This is not a scalable way to distribute packages. We wanted to find a way to distribute and publish packages in a way that is easily consumable for the majority of users. When considering the goal of being able to share packages, security and trust are very important considerations. We wanted our publishing solution and architecture changes to keep in mind signing of packages and the ability to verify the integrity of packages. + +We know we are successful when: + +1. (Priority) Users can use Zarf to natively publish a Zarf package to an OCI compliant registry +2. (Secondary goal) Package creators can sign Zarf packages to enable package deployers can trust a packages supply chain security + +## Decision + +We decided that changing the structure of Zarf packages to be an OCI artifact would be the best way to distribute and publish packages as registries are already an integral part of the container ecosystem. + +## Implementation + +A handful of changes were introduced to the structure of Zarf packages. + +```text +zarf-package-adr-arm64.tar.zst +├── checksums.txt +├── components +│ └── [...].tar +├── images +│ ├── index.json +│ ├── oci-layout +│ └── blobs +│ └── sha256 +│ └── ... # OCI image layers +├── sboms.tar +└── zarf.yaml +``` + +- Each component folder is now a tarball instead of a directory + - This enables us to treat each component as a layer within the package artifact +- Images are now stored in a flattened state instead of an images.tar file + - This enables us to keep each image layer as a layer within the package artifact (allowing for server side de-duping) +- SBOM files are now stored in a tarball instead of a directory + - This enables us to treat the SBOM artifacts as a single layer within the package artifact + +With this new structure in place, we can now publish Zarf packages as OCI artifacts. Under the hood this implements the `oras` Go library using Docker's authentication system. For interacting with these packages, the `oci://` package path prefix has been added (ex. `zarf package publish oci://...`). + +For an example of this in action, please see the [walkthrough](./docs/../../docs/13-walkthroughs/6-publish-and-deploy.md). + +## Consequences + +Backwards compatibility was an important considering when making these changes. We had to implement logic to make sure a new version of the Zarf binary could still operate with older versions of Zarf packages. + +At the moment we are testing the backwards compatibility by virtue of maintaining the `./src/test/e2e/27_cosign_deploy_test.go` where we are deploying an old Zarf package via `sget`. + +One thing we may want to look at more in the future is how we can get more intricate tests around the backwards compatibility. + +The reason why testing backwards compatibility is difficult is because this isn't a `zarf.yaml` schema change (like we had recently with the 'Scripts to Actions' PR) but an compiled package architecture change. This means that we will either need to maintain an 'old' Zarf package that will follow future `zarf.yaml` schema changes OR we maintain a modified Zarf binary that creates the old package structure. diff --git a/docs/13-walkthroughs/6-publish-and-deploy.md b/docs/13-walkthroughs/6-publish-and-deploy.md new file mode 100644 index 0000000000..539d181a07 --- /dev/null +++ b/docs/13-walkthroughs/6-publish-and-deploy.md @@ -0,0 +1,219 @@ +# Using OCI to Store & Deploy Zarf Packages + +## Introduction + +In this walkthrough, we are going to run through how to publish a Zarf package to an [OCI](https://github.com/opencontainers/image-spec) compliant registry, allowing end users to pull and deploy packages without needing to build locally, or transfer the package to their environment. + +## Prerequisites + +For following along locally, please ensure the following prerequisites are met: + +1. Zarf binary installed on your `$PATH`: ([Install Instructions](../3-getting-started.md#installing-zarf)) +2. Access to a [Registry supporting the OCI Distribution Spec](https://oras.land/implementors/#registries-supporting-oci-artifacts), this walkthrough will be using Docker Hub + +## Setup + +This walkthrough will require a registry to be configured (see [prerequisites](#prerequisites) for more information). The below sets up some variables for us to use when logging into the registry: + +```bash +# Setup some variables for the registry we will be using +$ REGISTRY=docker.io +$ set +o history +$ REGISTRY_USERNAME= # <-- replace with your username +$ REPOSITORY_URL=$REGISTRY/$REGISTRY_USERNAME +$ REGISTRY_SECRET= # <-- replace with your password or auth token +$ set -o history +``` + +With those set, you can tell Zarf to login to your registry with the following: + +```bash +$ echo $REGISTRY_SECRET | zarf tools registry login $REGISTRY --username $REGISTRY_USERNAME --password-stdin + +2023/03/07 23:03:16 logged in via /home/zarf/.docker/config.json +``` + +:::note + +If you do not have the Docker CLI installed, you may need to create a Docker compliant auth config file manually: + +```bash +$ mkdir -p ~/.docker +$ AUTH=$(echo -n "$REGISTRY_USERNAME:$REGISTRY_SECRET" | base64) +# Note: If using Docker Hub, the registry URL is `https://index.docker.io/v1/` for the auth config +$ cat < ~/.docker/config.json +{ + "auths": { + "$REGISTRY": { + "auth": "$AUTH" + } + } +} +EOF +``` + +::: + +## Publish Package + +First, create a valid Zarf package definition (`zarf.yaml`), with the `metadata.version` key set. + +```yaml +# Make a new directory to work in +$ mkdir -p zarf-publish-walkthrough && cd zarf-publish-walkthrough + +# For this walkthrough we will use the `helm-oci-chart` example package +# located here: https://github.com/defenseunicorns/zarf/blob/main/examples/helm-oci-chart/zarf.yaml +$ cat < zarf.yaml +kind: ZarfPackageConfig +metadata: + name: helm-oci-chart + description: Deploy podinfo using a Helm OCI chart + # Note: In order to publish, the package must have a version + version: 0.0.1 + +components: + - name: helm-oci-chart + required: true + charts: + - name: podinfo + version: 6.3.3 + namespace: helm-oci-demo + url: oci://ghcr.io/stefanprodan/charts/podinfo + images: + - "ghcr.io/stefanprodan/podinfo:6.3.3" +EOF +``` + +Create the package locally: + +[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_create.md) + +```bash +# Create the package (interactively) +$ zarf package create . +# Make these choices at the prompts: +# Create this Zarf package? Yes +# Please provide a value for "Maximum Package Size" 0 +``` + +Then publish the package to the registry: + +:::note + +Your package tarball may be named differently based on your machine's architecture. For example, if you are running on an AMD64 machine, the tarball will be named `zarf-package-helm-oci-chart-amd64-0.0.1.tar.zst`. + +::: + +[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_publish.md) + +```bash +$ zarf package publish zarf-package-helm-oci-chart-arm64-0.0.1.tar.zst oci://$REPOSITORY_URL + +... + + • Publishing package to $REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 + ✔ Prepared 14 layers + ✔ 515aceaacb8d images/index.json + ✔ 4615b4f0c1ed zarf.yaml + ✔ 1300d6545c84 sboms.tar + ✔ b66dbb27a733 images/oci-layout + ✔ 46564f0eff85 images/blobs/sha256/46564f0...06008f762391a7bb7d58f339ee + ✔ 4f4fb700ef54 images/blobs/sha256/4f4fb70...b5577484a6d75e68dc38e8acc1 + ✔ 6ff8f4799d50 images/blobs/sha256/6ff8f47...4bc00ec8b988d28cef78ea9a5b + ✔ 74eae207aa23 images/blobs/sha256/74eae20...fcb007d3da7b842313f80d2c33 + ✔ a9eaa45ef418 images/blobs/sha256/a9eaa45...6789c52a87ba5a9e6483f2b74f + ✔ 8c5b695f4724 images/blobs/sha256/8c5b695...014f94c8d4ea62772c477c1e03 + ✔ ab67ffd6e92e images/blobs/sha256/ab67ffd...f8c9d93c0e719f6350e99d3aea + ✔ b95c82728c36 images/blobs/sha256/b95c827...042a9c5d84426c1674044916d4 + ✔ e2b45cdcd8bf images/blobs/sha256/e2b45cd...000f1bc1695014e38821dc675c + ✔ 79be488a834e components/helm-oci-chart.tar + ✔ aed84ba183e7 [application/vnd.oci.artifact.manifest.v1+json] + ✔ Published $REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 [application/vnd.oci.artifact.manifest.v1+json] + + • To inspect/deploy/pull: + • zarf package inspect oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 + • zarf package deploy oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 + • zarf package pull oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 +``` + +:::note + +The name and reference of this OCI artifact is derived from the package metadata, e.g.: `helm-oci-chart:0.0.1-arm64` + +To modify, edit `zarf.yaml` and re-run `zarf package create .` + +::: + +## Inspect Package + +[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_inspect.md) + +Inspecting a Zarf package stored in an OCI registry is the same as inspecting a local package and has the same flags: + +```yaml +$ zarf package inspect oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 +--- +kind: ZarfPackageConfig +metadata: + name: helm-oci-chart + description: Deploy podinfo using a Helm OCI chart + version: 0.0.1 + architecture: arm64 +build: + terminal: minimind.local + user: whoami + architecture: arm64 + timestamp: Tue, 07 Mar 2023 14:27:25 -0600 + version: v0.25.0-rc1-41-g07d61ba7 + migrations: + - scripts-to-actions +components: + - name: helm-oci-chart + required: true + charts: + - name: podinfo + url: oci://ghcr.io/stefanprodan/charts/podinfo + version: 6.3.3 + namespace: helm-oci-demo + images: + - ghcr.io/stefanprodan/podinfo:6.3.3 +``` + +## Deploy Package + +[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md) + +Deploying a package stored in an OCI registry is nearly the same experience as deploying a local package: + +```bash +# Due to the length of the console output from this command, +# it has been omitted from this walkthrough +$ zarf package deploy oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 +# Make these choices at the prompts: +# Deploy this Zarf package? Yes + +$ zarf packages list + + Package | Components + helm-oci-chart | [helm-oci-chart] + init | [zarf-injector zarf-seed-registry zarf-registry zarf-agent git-server] +``` + +## Pull Package + +[CLI Reference](../4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_pull.md) + +Packages can be saved to the local disk in order to deploy a package multiple times without needing to fetch it every time. + +```bash +# go home so we don't clobber our currently local built package +$ cd ~ +$ mkdir -p zarf-packages && cd zarf-packages + +$ zarf package pull oci://$REPOSITORY_URL/helm-oci-chart:0.0.1-arm64 + +# use vim if you want to inspect the tarball's contents without decompressing it +$ vim zarf-package-helm-oci-chart-arm64-0.0.1.tar.zst +# don't forget to escape w/ `:q` +``` diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package.md index d367d81e3c..e65ed08760 100644 --- a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package.md +++ b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package.md @@ -28,5 +28,7 @@ Zarf package commands for creating, deploying, and inspecting packages * [zarf package deploy](zarf_package_deploy.md) - Use to deploy a Zarf package from a local file or URL (runs offline) * [zarf package inspect](zarf_package_inspect.md) - Lists the payload of a Zarf package (runs offline) * [zarf package list](zarf_package_list.md) - List out all of the packages that have been deployed to the cluster +* [zarf package publish](zarf_package_publish.md) - Publish a Zarf package to a remote registry +* [zarf package pull](zarf_package_pull.md) - Pull a Zarf package from a remote registry and save to the local file system * [zarf package remove](zarf_package_remove.md) - Use to remove a Zarf package that has been deployed already diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md index c42afea493..124ff8d67c 100644 --- a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md +++ b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md @@ -14,12 +14,13 @@ zarf package deploy [PACKAGE] [flags] ## Options ``` - --components string Comma-separated list of components to install. Adding this flag will skip the init prompts for which components to install - --confirm Confirm package deployment without prompting - -h, --help help for deploy - --set stringToString Specify deployment variables to set on the command line (KEY=value) (default []) - --sget string Path to public sget key file for remote packages signed via cosign - --shasum string Shasum of the package to deploy. Required if deploying a remote package and "--insecure" is not provided + --components string Comma-separated list of components to install. Adding this flag will skip the init prompts for which components to install + --confirm Confirm package deployment without prompting + -h, --help help for deploy + --oci-concurrency int Number of concurrent layer operations to perform when interacting with a remote package. (default 3) + --set stringToString Specify deployment variables to set on the command line (KEY=value) (default []) + --sget string Path to public sget key file for remote packages signed via cosign + --shasum string Shasum of the package to deploy. Required if deploying a remote package and "--insecure" is not provided ``` ## Options inherited from parent commands diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_publish.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_publish.md new file mode 100644 index 0000000000..563b3b4423 --- /dev/null +++ b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_publish.md @@ -0,0 +1,38 @@ +# zarf package publish + + +Publish a Zarf package to a remote registry + +``` +zarf package publish [PACKAGE] [REPOSITORY] [flags] +``` + +## Examples + +``` + zarf package publish my-package.tar oci://my-registry.com/my-namespace +``` + +## Options + +``` + -h, --help help for publish + --oci-concurrency int Number of concurrent layer operations to perform when interacting with a remote package. (default 3) +``` + +## Options inherited from parent commands + +``` + -a, --architecture string Architecture for OCI images + --insecure Allow access to insecure registries and disable other recommended security enforcements. This flag should only be used if you have a specific reason and accept the reduced security posture. + -l, --log-level string Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info") + --no-log-file Disable log file creation + --no-progress Disable fancy UI progress bars, spinners, logos, etc + --tmpdir string Specify the temporary directory to use for intermediate files + --zarf-cache string Specify the location of the Zarf cache directory (default "~/.zarf-cache") +``` + +## SEE ALSO + +* [zarf package](zarf_package.md) - Zarf package commands for creating, deploying, and inspecting packages + diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_pull.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_pull.md new file mode 100644 index 0000000000..de340a86d6 --- /dev/null +++ b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_pull.md @@ -0,0 +1,38 @@ +# zarf package pull + + +Pull a Zarf package from a remote registry and save to the local file system + +``` +zarf package pull [REFERENCE] [flags] +``` + +## Examples + +``` + zarf package pull oci://my-registry.com/my-namespace/my-package:0.0.1-arm64 +``` + +## Options + +``` + -h, --help help for pull + --oci-concurrency int Number of concurrent layer operations to perform when interacting with a remote package. (default 3) +``` + +## Options inherited from parent commands + +``` + -a, --architecture string Architecture for OCI images + --insecure Allow access to insecure registries and disable other recommended security enforcements. This flag should only be used if you have a specific reason and accept the reduced security posture. + -l, --log-level string Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info") + --no-log-file Disable log file creation + --no-progress Disable fancy UI progress bars, spinners, logos, etc + --tmpdir string Specify the temporary directory to use for intermediate files + --zarf-cache string Specify the location of the Zarf cache directory (default "~/.zarf-cache") +``` + +## SEE ALSO + +* [zarf package](zarf_package.md) - Zarf package commands for creating, deploying, and inspecting packages + diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_tools_archiver_decompress.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_tools_archiver_decompress.md index f6b4385aed..62d1f900c6 100644 --- a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_tools_archiver_decompress.md +++ b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_tools_archiver_decompress.md @@ -10,7 +10,8 @@ zarf tools archiver decompress {ARCHIVE} {DESTINATION} [flags] ## Options ``` - -h, --help help for decompress + --decompress-all Decompress all layers in the archive + -h, --help help for decompress ``` ## Options inherited from parent commands diff --git a/examples/helm-oci-chart/zarf.yaml b/examples/helm-oci-chart/zarf.yaml index d0b33927a8..487a88c4c1 100644 --- a/examples/helm-oci-chart/zarf.yaml +++ b/examples/helm-oci-chart/zarf.yaml @@ -2,6 +2,7 @@ kind: ZarfPackageConfig metadata: name: helm-oci-chart description: Deploy podinfo using a Helm OCI chart + version: 0.0.1 components: - name: helm-oci-chart diff --git a/go.mod b/go.mod index a92adfde59..cd9777b2b6 100644 --- a/go.mod +++ b/go.mod @@ -3,6 +3,8 @@ module github.com/defenseunicorns/zarf go 1.19 replace ( + // TODO (@WSTARR) use the OG syft once they merge in https://github.com/anchore/syft/pull/1668 + github.com/anchore/syft => github.com/defenseunicorns/syft v0.75.0-DU // TODO (@WSTARR) remove this temporary replacement of oras-go 1.2.2 with defenseunicorns version due to upgraded docker lib oras.land/oras-go v1.2.2 => github.com/defenseunicorns/oras-go v1.2.3 // TODO (@JMCCOY) not updating due to bug in kyaml, https://github.com/kubernetes-sigs/kustomize/issues/4896 @@ -16,6 +18,7 @@ require ( github.com/anchore/syft v0.75.0 github.com/derailed/k9s v0.27.3 github.com/distribution/distribution v2.8.1+incompatible + github.com/docker/cli v20.10.22+incompatible github.com/fatih/color v1.15.0 github.com/fluxcd/helm-controller/api v0.31.1 github.com/fluxcd/source-controller/api v0.36.0 @@ -26,6 +29,7 @@ require ( github.com/google/go-containerregistry v0.13.0 github.com/mholt/archiver/v3 v3.5.1 github.com/moby/moby v23.0.1+incompatible + github.com/opencontainers/image-spec v1.1.0-rc2 github.com/otiai10/copy v1.9.0 github.com/pkg/errors v0.9.1 github.com/pterm/pterm v0.12.55 @@ -41,6 +45,7 @@ require ( k8s.io/component-base v0.26.2 k8s.io/klog/v2 v2.90.1 k8s.io/kubectl v0.26.2 + oras.land/oras-go/v2 v2.0.0 sigs.k8s.io/kustomize/api v0.12.1 sigs.k8s.io/kustomize/kyaml v0.13.9 sigs.k8s.io/yaml v1.3.0 @@ -144,7 +149,6 @@ require ( github.com/derailed/tcell/v2 v2.3.1-rc.3 // indirect github.com/derailed/tview v0.8.1 // indirect github.com/dimchansky/utfbom v1.1.1 // indirect - github.com/docker/cli v20.10.22+incompatible // indirect github.com/docker/distribution v2.8.1+incompatible // indirect github.com/docker/docker v23.0.1+incompatible // indirect github.com/docker/docker-credential-helpers v0.7.0 // indirect @@ -280,7 +284,6 @@ require ( github.com/oklog/ulid v1.3.1 // indirect github.com/olekukonko/tablewriter v0.0.5 // indirect github.com/opencontainers/go-digest v1.0.0 // indirect - github.com/opencontainers/image-spec v1.1.0-rc2 // indirect github.com/opentracing/opentracing-go v1.2.0 // indirect github.com/pelletier/go-toml v1.9.5 // indirect github.com/pelletier/go-toml/v2 v2.0.6 // indirect diff --git a/go.sum b/go.sum index b0c7453674..0a6f8ffe69 100644 --- a/go.sum +++ b/go.sum @@ -264,8 +264,6 @@ github.com/anchore/packageurl-go v0.1.1-0.20230104203445-02e0a6721501 h1:AV7qjwM github.com/anchore/packageurl-go v0.1.1-0.20230104203445-02e0a6721501/go.mod h1:Blo6OgJNiYF41ufcgHKkbCKF2MDOMlrqhXv/ij6ocR4= github.com/anchore/stereoscope v0.0.0-20230301191755-abfb374a1122 h1:Oe2PE8zNbJH4nGZoCIC/VZBgpr62BInLnUqIMZICUOk= github.com/anchore/stereoscope v0.0.0-20230301191755-abfb374a1122/go.mod h1:IihP/SUVHP94PBwIP2bepOB/c0MVadcII7lxo13Ijzs= -github.com/anchore/syft v0.75.0 h1:DF6/TDMRC7L2ypWufQHezlE6XCfVHLyQHnjXmSZfNKA= -github.com/anchore/syft v0.75.0/go.mod h1:TljwLtC66GzBIiJmGhAMctgV9wjVp4g71aTJs4LkEyc= github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8= github.com/andybalholm/brotli v1.0.1/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y= github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY= @@ -493,6 +491,8 @@ github.com/daviddengcn/go-colortext v1.0.0 h1:ANqDyC0ys6qCSvuEK7l3g5RaehL/Xck9EX github.com/daviddengcn/go-colortext v1.0.0/go.mod h1:zDqEI5NVUop5QPpVJUxE9UO10hRnmkD5G4Pmri9+m4c= github.com/defenseunicorns/oras-go v1.2.3 h1:PAxO1Ows6gFVRaf7qYReONWbnjYJxfIh7avifw6QruE= github.com/defenseunicorns/oras-go v1.2.3/go.mod h1:Tsn7yEnnTlAgyJztm02IalpzdKWXaFbpiLVYF0F5gKU= +github.com/defenseunicorns/syft v0.75.0-DU h1:8/DCPRQbQxHHydJX7wsrB7W/1rX0Dgyh8O9FHTB54DU= +github.com/defenseunicorns/syft v0.75.0-DU/go.mod h1:TljwLtC66GzBIiJmGhAMctgV9wjVp4g71aTJs4LkEyc= github.com/denisenkom/go-mssqldb v0.9.0/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU= github.com/depcheck-test/depcheck-test v0.0.0-20220607135614-199033aaa936 h1:foGzavPWwtoyBvjWyKJYDYsyzy+23iBV7NKTwdk+LRY= github.com/derailed/k9s v0.27.3 h1:2PMIq3SWpQsnVMXr5HuP//ruarZkXFtrOfe8Ml5WOK0= @@ -2724,6 +2724,8 @@ modernc.org/token v1.0.0 h1:a0jaWiNMDhDUtqOj09wvjWWAqd3q7WpBulmL9H2egsk= modernc.org/token v1.0.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM= modernc.org/z v1.5.1 h1:RTNHdsrOpeoSeOF4FbzTo8gBYByaJ5xT7NgZ9ZqRiJM= modernc.org/z v1.5.1/go.mod h1:eWFB510QWW5Th9YGZT81s+LwvaAs3Q2yr4sP0rmLkv8= +oras.land/oras-go/v2 v2.0.0 h1:+LRAz92WF7AvYQsQjPEAIw3Xb2zPPhuydjpi4pIHmc0= +oras.land/oras-go/v2 v2.0.0/go.mod h1:iVExH1NxrccIxjsiq17L91WCZ4KIw6jVQyCLsZsu1gc= pack.ag/amqp v0.11.2/go.mod h1:4/cbmt4EJXSKlG6LCfWHoqmN0uFdy5i/+YFz+fTfhV4= rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4= diff --git a/purgatory/dos-games/compare.html b/purgatory/dos-games/compare.html new file mode 100644 index 0000000000..eef4de0851 --- /dev/null +++ b/purgatory/dos-games/compare.html @@ -0,0 +1,894 @@ + + + + + + + Zarf SBOM Comparison + + + + + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Zarf SBOM Comparison +
+
+ +
+ +
+
+
+

+ + + + +
+
+

Old File

+ +
+
+

New File

+ +
+
+

 

+ +
+
+
+

 

+ +
+
+ + + + + diff --git a/purgatory/dos-games/defenseunicorns_zarf-game_multi-tile-dark.json b/purgatory/dos-games/defenseunicorns_zarf-game_multi-tile-dark.json new file mode 100644 index 0000000000..8481f7d170 --- /dev/null +++ b/purgatory/dos-games/defenseunicorns_zarf-game_multi-tile-dark.json @@ -0,0 +1,44 @@ +{ + "artifacts": [], + "artifactRelationships": [], + "source": { + "id": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "type": "image", + "target": { + "userInput": "", + "imageID": "sha256:be9619e8e2570e0012bd9e71cb3dfcef65ce33b443dcca525de20b4cabad04cd", + "manifestDigest": "", + "mediaType": "application/vnd.docker.distribution.manifest.v2+json", + "tags": [ + "defenseunicorns/zarf-game:multi-tile-dark" + ], + "imageSize": 26910742, + "layers": [ + { + "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", + "digest": "sha256:48e874f265bb97e23b3820cbbcf3ced8d0263a241cf8b4580a5f980b4ae8d812", + "size": 26166294 + }, + { + "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", + "digest": "sha256:05055c8f398d69d2de1c3f5621a99de576f577aab77a0a0b782458c0f1599984", + "size": 744448 + } + ], + "manifest": null, + "config": "eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsImNvbmZpZyI6eyJFbnYiOlsiUEFUSD0vdXNyL2xvY2FsL3NiaW46L3Vzci9sb2NhbC9iaW46L3Vzci9zYmluOi91c3IvYmluOi9zYmluOi9iaW4iXSwiRW50cnlwb2ludCI6WyIvYmluYXJ5L2RhcmtodHRwZCIsIi9zaXRlIiwiLS1wb3J0IiwiODAwMCJdLCJXb3JraW5nRGlyIjoiL3NpdGUiLCJPbkJ1aWxkIjpudWxsfSwiY3JlYXRlZCI6IjIwMjItMDEtMzBUMTI6NTA6MDAuOTYyMDA5ODg2WiIsImhpc3RvcnkiOlt7ImNyZWF0ZWQiOiIyMDIyLTAxLTMwVDEyOjUwOjAwLjg5MzgwODA1M1oiLCJjcmVhdGVkX2J5IjoiQ09QWSAvc2l0ZSAvc2l0ZSAjIGJ1aWxka2l0IiwiY29tbWVudCI6ImJ1aWxka2l0LmRvY2tlcmZpbGUudjAifSx7ImNyZWF0ZWQiOiIyMDIyLTAxLTMwVDEyOjUwOjAwLjk2MjAwOTg4NloiLCJjcmVhdGVkX2J5IjoiQ09QWSAvYmluYXJ5IC9iaW5hcnkgIyBidWlsZGtpdCIsImNvbW1lbnQiOiJidWlsZGtpdC5kb2NrZXJmaWxlLnYwIn0seyJjcmVhdGVkIjoiMjAyMi0wMS0zMFQxMjo1MDowMC45NjIwMDk4ODZaIiwiY3JlYXRlZF9ieSI6IldPUktESVIgL3NpdGUiLCJjb21tZW50IjoiYnVpbGRraXQuZG9ja2VyZmlsZS52MCIsImVtcHR5X2xheWVyIjp0cnVlfSx7ImNyZWF0ZWQiOiIyMDIyLTAxLTMwVDEyOjUwOjAwLjk2MjAwOTg4NloiLCJjcmVhdGVkX2J5IjoiRU5UUllQT0lOVCBbXCIvYmluYXJ5L2RhcmtodHRwZFwiIFwiL3NpdGVcIiBcIi0tcG9ydFwiIFwiODAwMFwiXSIsImNvbW1lbnQiOiJidWlsZGtpdC5kb2NrZXJmaWxlLnYwIiwiZW1wdHlfbGF5ZXIiOnRydWV9XSwib3MiOiJsaW51eCIsInJvb3RmcyI6eyJ0eXBlIjoibGF5ZXJzIiwiZGlmZl9pZHMiOlsic2hhMjU2OjQ4ZTg3NGYyNjViYjk3ZTIzYjM4MjBjYmJjZjNjZWQ4ZDAyNjNhMjQxY2Y4YjQ1ODBhNWY5ODBiNGFlOGQ4MTIiLCJzaGEyNTY6MDUwNTVjOGYzOThkNjlkMmRlMWMzZjU2MjFhOTlkZTU3NmY1NzdhYWI3N2EwYTBiNzgyNDU4YzBmMTU5OTk4NCJdfX0=", + "repoDigests": [], + "architecture": "", + "os": "" + } + }, + "distro": {}, + "descriptor": { + "name": "zarf", + "version": "" + }, + "schema": { + "version": "7.0.1", + "url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-7.0.1.json" + } +} diff --git a/purgatory/dos-games/sbom-viewer-defenseunicorns_zarf-game_multi-tile-dark.html b/purgatory/dos-games/sbom-viewer-defenseunicorns_zarf-game_multi-tile-dark.html new file mode 100644 index 0000000000..f45219ad54 --- /dev/null +++ b/purgatory/dos-games/sbom-viewer-defenseunicorns_zarf-game_multi-tile-dark.html @@ -0,0 +1,848 @@ + + + + + + + Zarf SBOM Viewer + + + + + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Zarf SBOM Viewer +
+
+ +
+ +
+
+
+ + + + + +
+

+
+ +
+ + + + + diff --git a/src/cmd/internal.go b/src/cmd/internal.go index 5dfc49e65f..6eb66ec905 100644 --- a/src/cmd/internal.go +++ b/src/cmd/internal.go @@ -54,7 +54,7 @@ var generateCLIDocs = &cobra.Command{ if err := doc.GenMarkdownTree(rootCmd, "./docs/4-user-guide/1-the-zarf-cli/100-cli-commands"); err != nil { message.Fatalf("Unable to generate the CLI documentation: %s", err.Error()) } else { - message.SuccessF(lang.CmdInternalGenerateCliDocsSuccess) + message.Successf(lang.CmdInternalGenerateCliDocsSuccess) } }, } diff --git a/src/cmd/package.go b/src/cmd/package.go index cf806d552b..bb242c52de 100644 --- a/src/cmd/package.go +++ b/src/cmd/package.go @@ -15,6 +15,7 @@ import ( "github.com/defenseunicorns/zarf/src/pkg/message" "github.com/defenseunicorns/zarf/src/types" "github.com/pterm/pterm" + "oras.land/oras-go/v2/registry" "github.com/AlecAivazis/survey/v2" "github.com/defenseunicorns/zarf/src/config" @@ -192,6 +193,56 @@ var packageRemoveCmd = &cobra.Command{ }, } +var packagePublishCmd = &cobra.Command{ + Use: "publish [PACKAGE] [REPOSITORY]", + Short: "Publish a Zarf package to a remote registry", + Example: " zarf package publish my-package.tar oci://my-registry.com/my-namespace", + Args: cobra.ExactArgs(2), + Run: func(cmd *cobra.Command, args []string) { + pkgConfig.PublishOpts.PackagePath = choosePackage(args) + + if !utils.IsOCIURL(args[1]) { + message.Fatalf(nil, "Registry must be prefixed with 'oci://'") + } + parts := strings.Split(strings.TrimPrefix(args[1], "oci://"), "/") + pkgConfig.PublishOpts.Reference = registry.Reference{ + Registry: parts[0], + Repository: strings.Join(parts[1:], "/"), + } + + // Configure the packager + pkgClient := packager.NewOrDie(&pkgConfig) + defer pkgClient.ClearTempPaths() + + // Publish the package + if err := pkgClient.Publish(); err != nil { + message.Fatalf(err, "Failed to publish package: %s", err.Error()) + } + }, +} + +var packagePullCmd = &cobra.Command{ + Use: "pull [REFERENCE]", + Short: "Pull a Zarf package from a remote registry and save to the local file system", + Example: " zarf package pull oci://my-registry.com/my-namespace/my-package:0.0.1-arm64", + Args: cobra.ExactArgs(1), + Run: func(cmd *cobra.Command, args []string) { + if !utils.IsOCIURL(args[0]) { + message.Fatalf(nil, "Registry must be prefixed with 'oci://'") + } + pkgConfig.DeployOpts.PackagePath = choosePackage(args) + + // Configure the packager + pkgClient := packager.NewOrDie(&pkgConfig) + defer pkgClient.ClearTempPaths() + + // Pull the package + if err := pkgClient.Pull(); err != nil { + message.Fatalf(err, "Failed to pull package: %s", err.Error()) + } + }, +} + func choosePackage(args []string) string { if len(args) > 0 { return args[0] @@ -226,11 +277,15 @@ func init() { packageCmd.AddCommand(packageInspectCmd) packageCmd.AddCommand(packageRemoveCmd) packageCmd.AddCommand(packageListCmd) + packageCmd.AddCommand(packagePublishCmd) + packageCmd.AddCommand(packagePullCmd) bindCreateFlags() bindDeployFlags() bindInspectFlags() bindRemoveFlags() + bindPublishFlags() + bindPullFlags() } func bindCreateFlags() { @@ -264,11 +319,13 @@ func bindDeployFlags() { v.SetDefault(V_PKG_DEPLOY_COMPONENTS, "") v.SetDefault(V_PKG_DEPLOY_SHASUM, "") v.SetDefault(V_PKG_DEPLOY_SGET, "") + v.SetDefault(V_PKG_PUBLISH_OCI_CONCURRENCY, 3) deployFlags.StringToStringVar(&pkgConfig.DeployOpts.SetVariables, "set", v.GetStringMapString(V_PKG_DEPLOY_SET), lang.CmdPackageDeployFlagSet) deployFlags.StringVar(&pkgConfig.DeployOpts.Components, "components", v.GetString(V_PKG_DEPLOY_COMPONENTS), lang.CmdPackageDeployFlagComponents) deployFlags.StringVar(&pkgConfig.DeployOpts.Shasum, "shasum", v.GetString(V_PKG_DEPLOY_SHASUM), lang.CmdPackageDeployFlagShasum) deployFlags.StringVar(&pkgConfig.DeployOpts.SGetKeyPath, "sget", v.GetString(V_PKG_DEPLOY_SGET), lang.CmdPackageDeployFlagSget) + deployFlags.IntVar(&pkgConfig.PublishOpts.CopyOptions.Concurrency, "oci-concurrency", v.GetInt(V_PKG_PUBLISH_OCI_CONCURRENCY), lang.CmdPackagePublishFlagConcurrency) } func bindInspectFlags() { @@ -283,3 +340,13 @@ func bindRemoveFlags() { removeFlags.StringVar(&pkgConfig.DeployOpts.Components, "components", v.GetString(V_PKG_DEPLOY_COMPONENTS), lang.CmdPackageRemoveFlagComponents) _ = packageRemoveCmd.MarkFlagRequired("confirm") } + +func bindPublishFlags() { + publishFlags := packagePublishCmd.Flags() + publishFlags.IntVar(&pkgConfig.PublishOpts.CopyOptions.Concurrency, "oci-concurrency", v.GetInt(V_PKG_PUBLISH_OCI_CONCURRENCY), lang.CmdPackagePublishFlagConcurrency) +} + +func bindPullFlags() { + pullFlags := packagePullCmd.Flags() + pullFlags.IntVar(&pkgConfig.PublishOpts.CopyOptions.Concurrency, "oci-concurrency", v.GetInt(V_PKG_PUBLISH_OCI_CONCURRENCY), lang.CmdPackagePublishFlagConcurrency) +} diff --git a/src/cmd/tools/archiver.go b/src/cmd/tools/archiver.go index 5072246301..e9c2c3a2f6 100644 --- a/src/cmd/tools/archiver.go +++ b/src/cmd/tools/archiver.go @@ -5,6 +5,10 @@ package tools import ( + "os" + "path/filepath" + "strings" + "github.com/defenseunicorns/zarf/src/config/lang" "github.com/defenseunicorns/zarf/src/pkg/message" "github.com/mholt/archiver/v3" @@ -31,6 +35,8 @@ var archiverCompressCmd = &cobra.Command{ }, } +var decompressLayers bool + var archiverDecompressCmd = &cobra.Command{ Use: "decompress {ARCHIVE} {DESTINATION}", Aliases: []string{"d"}, @@ -42,6 +48,38 @@ var archiverDecompressCmd = &cobra.Command{ if err != nil { message.Fatal(err, lang.CmdToolsArchiverDecompressErr) } + + // Decompress component layers in the destination path + if decompressLayers { + // Decompress the components + layersDir := filepath.Join(destinationPath, "components") + + files, err := os.ReadDir(layersDir) + if err != nil { + message.Fatalf(err, "failed to read the layers of components") + } + for _, file := range files { + if strings.HasSuffix(file.Name(), ".tar") { + if err := archiver.Unarchive(filepath.Join(layersDir, file.Name()), layersDir); err != nil { + message.Fatalf(err, "failed to decompress the component layer") + } else { + // Without unarchive error, delete original tar.zst in component folder + // This will leave the tar.zst if their is a failure for post mortem check + _ = os.Remove(filepath.Join(layersDir, file.Name())) + } + } + } + + // Decompress the SBOMs + sbomsTar := filepath.Join(destinationPath, "sboms.tar") + _, err = os.Stat(sbomsTar) + if err == nil { + if err := archiver.Unarchive(sbomsTar, filepath.Join(destinationPath, "sboms")); err != nil { + message.Fatalf(err, "failed to decompress the sboms layer") + } + _ = os.Remove(sbomsTar) + } + } }, } @@ -50,4 +88,5 @@ func init() { archiverCmd.AddCommand(archiverCompressCmd) archiverCmd.AddCommand(archiverDecompressCmd) + archiverDecompressCmd.Flags().BoolVar(&decompressLayers, "decompress-all", false, "Decompress all layers in the archive") } diff --git a/src/cmd/tools/zarf.go b/src/cmd/tools/zarf.go index c1a8c6e9bd..7a189ad725 100644 --- a/src/cmd/tools/zarf.go +++ b/src/cmd/tools/zarf.go @@ -68,7 +68,7 @@ func init() { if err := os.RemoveAll(config.GetAbsCachePath()); err != nil { message.Fatalf(err, lang.CmdToolsClearCacheErr, config.GetAbsCachePath()) } - message.SuccessF(lang.CmdToolsClearCacheSuccess, config.GetAbsCachePath()) + message.Successf(lang.CmdToolsClearCacheSuccess, config.GetAbsCachePath()) }, } @@ -88,7 +88,7 @@ func init() { if err := os.WriteFile("tls.key", pki.Key, 0600); err != nil { message.Fatalf(err, lang.ErrWritingFile, "tls.key", err.Error()) } - message.SuccessF(lang.CmdToolsGenPkiSuccess, args[0]) + message.Successf(lang.CmdToolsGenPkiSuccess, args[0]) }, } diff --git a/src/cmd/viper.go b/src/cmd/viper.go index c8b0221c74..20455f82e7 100644 --- a/src/cmd/viper.go +++ b/src/cmd/viper.go @@ -56,6 +56,9 @@ const ( V_PKG_DEPLOY_COMPONENTS = "package.deploy.components" V_PKG_DEPLOY_SHASUM = "package.deploy.shasum" V_PKG_DEPLOY_SGET = "package.deploy.sget" + + // Package publish config keys + V_PKG_PUBLISH_OCI_CONCURRENCY = "package.publish.oci_concurrency" ) func initViper() { diff --git a/src/config/lang/english.go b/src/config/lang/english.go index c1c0942c17..7555c052f1 100644 --- a/src/config/lang/english.go +++ b/src/config/lang/english.go @@ -208,6 +208,8 @@ const ( CmdPackageRemoveFlagConfirm = "REQUIRED. Confirm the removal action to prevent accidental deletions" CmdPackageRemoveFlagComponents = "Comma-separated list of components to uninstall" + CmdPackagePublishFlagConcurrency = "Number of concurrent layer operations to perform when interacting with a remote package." + // zarf prepare CmdPrepareShort = "Tools to help prepare assets for packaging" diff --git a/src/internal/packager/images/common.go b/src/internal/packager/images/common.go index 4cc16ecf9e..d5f3ffc7ae 100644 --- a/src/internal/packager/images/common.go +++ b/src/internal/packager/images/common.go @@ -4,11 +4,21 @@ // Package images provides functions for building and pushing images. package images -import "github.com/defenseunicorns/zarf/src/types" +import ( + "fmt" + "os" + + "github.com/defenseunicorns/zarf/src/config" + "github.com/defenseunicorns/zarf/src/types" + "github.com/google/go-containerregistry/pkg/crane" + v1 "github.com/google/go-containerregistry/pkg/v1" + "github.com/google/go-containerregistry/pkg/v1/layout" + ocispec "github.com/opencontainers/image-spec/specs-go/v1" +) // ImgConfig is the main struct for managing container images. type ImgConfig struct { - TarballPath string + ImagesPath string ImgList []string @@ -18,3 +28,43 @@ type ImgConfig struct { Insecure bool } + +// GetLegacyImgTarballPath returns the ImagesPath as if it were a path to a tarball instead of a directory. +func (i *ImgConfig) GetLegacyImgTarballPath() string { + return fmt.Sprintf("%s.tar", i.ImagesPath) +} + +// LoadImageFromPackage returns a v1.Image from the image tag specified, or an error if the image cannot be found. +func (i ImgConfig) LoadImageFromPackage(imgTag string) (v1.Image, error) { + // If the package still has a images.tar that contains all of the images, use crane to load the specific tag we want + if _, statErr := os.Stat(i.GetLegacyImgTarballPath()); statErr == nil { + return crane.LoadTag(i.GetLegacyImgTarballPath(), imgTag, config.GetCraneOptions(i.Insecure)...) + } + + // Load the image from the OCI formatted images directory + return LoadImage(i.ImagesPath, imgTag) +} + +// LoadImage returns a v1.Image with the image tag specified from a location provided, or an error if the image cannot be found. +func LoadImage(imgPath, imgTag string) (v1.Image, error) { + // Use the manifest within the index.json to load the specific image we want + layoutPath := layout.Path(imgPath) + imgIdx, err := layoutPath.ImageIndex() + if err != nil { + return nil, err + } + idxManifest, err := imgIdx.IndexManifest() + if err != nil { + return nil, err + } + + // Search through all the manifests within this package until we find the annotation that matches our tag + for _, manifest := range idxManifest.Manifests { + if manifest.Annotations[ocispec.AnnotationBaseImageName] == imgTag { + // This is the image we are looking for, load it and then return + return layoutPath.Image(manifest.Digest) + } + } + + return nil, fmt.Errorf("unable to find image (%s) at the path (%s)", imgTag, imgPath) +} diff --git a/src/internal/packager/images/pull.go b/src/internal/packager/images/pull.go index fdd81f8194..28ee34b61f 100644 --- a/src/internal/packager/images/pull.go +++ b/src/internal/packager/images/pull.go @@ -6,11 +6,13 @@ package images import ( "context" - "errors" + "encoding/json" "fmt" "io" + "os" "path/filepath" "strings" + "sync" "github.com/defenseunicorns/zarf/src/config" "github.com/defenseunicorns/zarf/src/pkg/message" @@ -21,18 +23,19 @@ import ( v1 "github.com/google/go-containerregistry/pkg/v1" "github.com/google/go-containerregistry/pkg/v1/cache" "github.com/google/go-containerregistry/pkg/v1/daemon" - "github.com/google/go-containerregistry/pkg/v1/tarball" "github.com/moby/moby/client" + ocispec "github.com/opencontainers/image-spec/specs-go/v1" "github.com/pterm/pterm" ) // PullAll pulls all of the images in the provided tag map. func (i *ImgConfig) PullAll() error { var ( - longer string - imgCount = len(i.ImgList) - imageMap = map[string]v1.Image{} - tagToImage = map[name.Tag]v1.Image{} + longer string + imgCount = len(i.ImgList) + imageMap = map[string]v1.Image{} + tagToImage = map[name.Tag]v1.Image{} + digestToTag = make(map[string]string) ) // Give some additional user feedback on larger image sets @@ -60,50 +63,70 @@ func (i *ImgConfig) PullAll() error { imageMap[src] = img } + // Create the ImagePath directory + err := os.Mkdir(i.ImagesPath, 0755) + if err != nil { + return fmt.Errorf("failed to create image path %s: %w", i.ImagesPath, err) + } + + totalBytes := int64(0) for src, img := range imageMap { tag, err := name.NewTag(src, name.WeakValidation) if err != nil { return fmt.Errorf("failed to create tag for image %s: %w", src, err) } tagToImage[tag] = img + // Get the byte size for this image + layers, err := img.Layers() + if err != nil { + return fmt.Errorf("unable to get layers for image %s: %w", src, err) + } + for _, layer := range layers { + size, err := layer.Size() + if err != nil { + return fmt.Errorf("unable to get size of layer: %w", err) + } + totalBytes += size + } } spinner.Updatef("Preparing image sources and cache for image pulling") + spinner.Success() - var ( - progress = make(chan v1.Update, 200) - progressBar *message.ProgressBar - title string - ) + // Create a thread to update a progress bar as we save the image files to disk + doneSaving := make(chan int) + var wg sync.WaitGroup + wg.Add(1) + go utils.RenderProgressBarForLocalDirWrite(i.ImagesPath, totalBytes, &wg, doneSaving, fmt.Sprintf("Pulling %d images", imgCount)) + + for tag, img := range tagToImage { - go func() { - _ = tarball.MultiWriteToFile(i.TarballPath, tagToImage, tarball.WithProgress(progress)) - }() - - for update := range progress { - switch { - case update.Error != nil && errors.Is(update.Error, io.EOF): - progressBar.Successf("Pulling %d images (%s)", len(imageMap), utils.ByteFormat(float64(update.Total), 2)) - return nil - case update.Error != nil && strings.HasPrefix(update.Error.Error(), "archive/tar: missed writing "): - // Handle potential image cache corruption with a more helpful error. See L#54 in libexec/src/archive/tar/writer.go - message.Warnf("Potential image cache corruption: %s of %v bytes - try clearing cache with \"zarf tools clear-cache\"", update.Error.Error(), update.Total) - return fmt.Errorf("failed to write image tarball: %w", update.Error) - case update.Error != nil: - return fmt.Errorf("failed to write image tarball: %w", update.Error) - default: - title = fmt.Sprintf("Pulling %d images (%s of %s)", len(imageMap), - utils.ByteFormat(float64(update.Complete), 2), - utils.ByteFormat(float64(update.Total), 2), - ) - if progressBar == nil { - spinner.Success() - progressBar = message.NewProgressBar(update.Total, title) + // Save the image + err := crane.SaveOCI(img, i.ImagesPath) + if err != nil { + // Check if the cache has been invalidated, and warn the user if so + if strings.HasPrefix(err.Error(), "error writing layer: expected blob size") { + message.Warnf("Potential image cache corruption: %s - try clearing cache with \"zarf tools clear-cache\"", err.Error()) } - progressBar.Update(update.Complete, title) + return fmt.Errorf("error when trying to save the img (%s): %w", tag.Name(), err) + } + + // Get the image digest so we can set an annotation in the image.json later + imgDigest, err := img.Digest() + if err != nil { + return err } + digestToTag[imgDigest.String()] = tag.String() } - return nil + if err := addImageNameAnnotation(i.ImagesPath, digestToTag); err != nil { + return fmt.Errorf("unable to format OCI layout: %w", err) + } + + // Send a signal to the progress bar that we're done and ait for the thread to finish + doneSaving <- 1 + wg.Wait() + + return err } // PullImage returns a v1.Image either by loading a local tarball or the wider internet. @@ -168,3 +191,63 @@ func (i *ImgConfig) PullImage(src string, spinner *message.Spinner) (img v1.Imag return img, nil } + +// IndexJSON represents the index.json file in an OCI layout. +type IndexJSON struct { + SchemaVersion int `json:"schemaVersion"` + Manifests []struct { + MediaType string `json:"mediaType"` + Size int `json:"size"` + Digest string `json:"digest"` + Annotations map[string]string `json:"annotations"` + } `json:"manifests"` +} + +// addImageNameAnnotation adds an annotation to the index.json file so that the deploying code can figure out what the image tag <-> digest shasum will be. +func addImageNameAnnotation(ociPath string, digestToTag map[string]string) error { + indexPath := filepath.Join(ociPath, "index.json") + + // Add an 'org.opencontainers.image.base.name' annotation so we can figure out what the image tag/digest shasum will be during deploy time + indexJSON, err := os.Open(indexPath) + if err != nil { + message.Errorf(err, "Unable to open %s/index.json", ociPath) + return err + } + + // Read the file contents and turn it into a usable struct that we can manipulate + var index IndexJSON + byteValue, err := io.ReadAll(indexJSON) + if err != nil { + return fmt.Errorf("unable to read the contents of the file (%s) so we can add an annotation: %w", indexPath, err) + } + indexJSON.Close() + if err = json.Unmarshal(byteValue, &index); err != nil { + return fmt.Errorf("unable to process the conents of the file (%s): %w", indexPath, err) + } + for idx, manifest := range index.Manifests { + if manifest.Annotations == nil { + manifest.Annotations = make(map[string]string) + } + manifest.Annotations[ocispec.AnnotationBaseImageName] = digestToTag[manifest.Digest] + index.Manifests[idx] = manifest + } + + // Remove any file that might already exist + _ = os.Remove(indexPath) + + // Create the index.json file and save the data to it + indexJSON, err = os.Create(indexPath) + if err != nil { + return err + } + indexJSONBytes, err := json.Marshal(index) + if err != nil { + return err + } + _, err = indexJSON.Write(indexJSONBytes) + if err != nil { + return err + } + + return indexJSON.Close() +} diff --git a/src/internal/packager/images/push.go b/src/internal/packager/images/push.go index 214b4afaab..982c6a4e61 100644 --- a/src/internal/packager/images/push.go +++ b/src/internal/packager/images/push.go @@ -56,9 +56,11 @@ func (i *ImgConfig) PushToZarfRegistry() error { pushOptions = append(pushOptions, config.GetCraneAuthOption(i.RegInfo.PushUsername, i.RegInfo.PushPassword)) message.Debugf("crane pushOptions = %#v", pushOptions) + for _, src := range i.ImgList { spinner.Updatef("Updating image %s", src) - img, err := crane.LoadTag(i.TarballPath, src, config.GetCraneOptions(i.Insecure)...) + + img, err := i.LoadImageFromPackage(src) if err != nil { return err } @@ -70,7 +72,7 @@ func (i *ImgConfig) PushToZarfRegistry() error { return err } - message.Debugf("crane.Push() %s:%s -> %s)", i.TarballPath, src, offlineNameCRC) + message.Debugf("crane.Push() %s:%s -> %s)", i.ImagesPath, src, offlineNameCRC) if err = crane.Push(img, offlineNameCRC, pushOptions...); err != nil { return err @@ -84,7 +86,7 @@ func (i *ImgConfig) PushToZarfRegistry() error { return err } - message.Debugf("crane.Push() %s:%s -> %s)", i.TarballPath, src, offlineName) + message.Debugf("crane.Push() %s:%s -> %s)", i.ImagesPath, src, offlineName) if err = crane.Push(img, offlineName, pushOptions...); err != nil { return err diff --git a/src/internal/packager/sbom/catalog.go b/src/internal/packager/sbom/catalog.go old mode 100644 new mode 100755 index e2d3798f82..eea72d7e42 --- a/src/internal/packager/sbom/catalog.go +++ b/src/internal/packager/sbom/catalog.go @@ -20,20 +20,23 @@ import ( "github.com/anchore/syft/syft/sbom" "github.com/anchore/syft/syft/source" "github.com/defenseunicorns/zarf/src/config" + "github.com/defenseunicorns/zarf/src/internal/packager/images" "github.com/defenseunicorns/zarf/src/pkg/message" "github.com/defenseunicorns/zarf/src/pkg/utils" "github.com/defenseunicorns/zarf/src/types" "github.com/google/go-containerregistry/pkg/name" - "github.com/google/go-containerregistry/pkg/v1/tarball" + v1 "github.com/google/go-containerregistry/pkg/v1" + "github.com/mholt/archiver/v3" ) // Builder is the main struct used to build SBOM artifacts. type Builder struct { - spinner *message.Spinner - cachePath string - imagesPath string - sbomPath string - jsonList []byte + spinner *message.Spinner + cachePath string + imagesPath string + tmpSBOMPath string + sbomTarPath string + jsonList []byte } //go:embed viewer/* @@ -43,40 +46,51 @@ var transformRegex = regexp.MustCompile(`(?m)[^a-zA-Z0-9\.\-]`) var componentPrefix = "zarf-component-" // Catalog catalogs the given components and images to create an SBOM. -func Catalog(componentSBOMs map[string]*types.ComponentSBOM, imgList []string, imagesPath, sbomPath string) { +// func Catalog(componentSBOMs map[string]*types.ComponentSBOM, imgList []string, imagesPath, sbomPath string) error { +func Catalog(componentSBOMs map[string]*types.ComponentSBOM, imgList []string, tmpPaths types.TempPaths) error { imageCount := len(imgList) componentCount := len(componentSBOMs) builder := Builder{ - spinner: message.NewProgressSpinner("Creating SBOMs for %d images and %d components with files.", imageCount, componentCount), - cachePath: config.GetAbsCachePath(), - imagesPath: imagesPath, - sbomPath: sbomPath, + spinner: message.NewProgressSpinner("Creating SBOMs for %d images and %d components with files.", imageCount, componentCount), + cachePath: config.GetAbsCachePath(), + imagesPath: tmpPaths.Images, + sbomTarPath: tmpPaths.SbomTar, + tmpSBOMPath: tmpPaths.Sboms, } defer builder.spinner.Stop() // Ensure the sbom directory exists - _ = utils.CreateDirectory(builder.sbomPath, 0700) + _ = utils.CreateDirectory(builder.tmpSBOMPath, 0700) // Generate a list of images and files for the sbom viewer - if json, err := builder.generateJSONList(componentSBOMs, imgList); err != nil { - builder.spinner.Fatalf(err, "Unable to generate the SBOM image list") - } else { - builder.jsonList = json + json, err := builder.generateJSONList(componentSBOMs, imgList) + if err != nil { + builder.spinner.Errorf(err, "Unable to generate the SBOM image list") + return err } - - currImage := 1 + builder.jsonList = json // Generate SBOM for each image + currImage := 1 for _, tag := range imgList { builder.spinner.Updatef("Creating image SBOMs (%d of %d): %s", currImage, imageCount, tag) - jsonData, err := builder.createImageSBOM(tag) + // Get the image that we are creating an SBOM for + img, err := images.LoadImage(tmpPaths.Images, tag) + if err != nil { + builder.spinner.Errorf(err, "Unable to load the image to generate an SBOM") + return err + } + + jsonData, err := builder.createImageSBOM(img, tag) if err != nil { - builder.spinner.Fatalf(err, "Unable to create SBOM for image %s", tag) + builder.spinner.Errorf(err, "Unable to create SBOM for image %s", tag) + return err } if err = builder.createSBOMViewerAsset(tag, jsonData); err != nil { - builder.spinner.Fatalf(err, "Unable to create SBOM viewer for image %s", tag) + builder.spinner.Errorf(err, "Unable to create SBOM viewer for image %s", tag) + return err } currImage++ @@ -95,37 +109,52 @@ func Catalog(componentSBOMs map[string]*types.ComponentSBOM, imgList []string, i jsonData, err := builder.createFileSBOM(*componentSBOMs[component], component) if err != nil { - builder.spinner.Fatalf(err, "Unable to create SBOM for component %s", component) + builder.spinner.Errorf(err, "Unable to create SBOM for component %s", component) + return err } if err = builder.createSBOMViewerAsset(fmt.Sprintf("%s%s", componentPrefix, component), jsonData); err != nil { - builder.spinner.Fatalf(err, "Unable to create SBOM viewer for component %s", component) + builder.spinner.Errorf(err, "Unable to create SBOM viewer for component %s", component) + return err } - currImage++ + currComponent++ } // Include the compare tool if there are any image SBOMs OR component SBOMs if len(componentSBOMs) > 0 || len(imgList) > 0 { if err := builder.createSBOMCompareAsset(); err != nil { - builder.spinner.Fatalf(err, "Unable to create SBOM compare tool") + builder.spinner.Errorf(err, "Unable to create SBOM compare tool") + return err } } + allSBOMFiles, err := filepath.Glob(filepath.Join(builder.tmpSBOMPath, "*")) + if err != nil { + builder.spinner.Errorf(err, "Unable to get a list of all SBOM files") + return err + } + + if err = archiver.Archive(allSBOMFiles, builder.sbomTarPath); err != nil { + builder.spinner.Errorf(err, "Unable to create the sbom archive") + return err + } + + if err = os.RemoveAll(builder.tmpSBOMPath); err != nil { + builder.spinner.Errorf(err, "Unable to remove the temporary SBOM directory") + return err + } + builder.spinner.Success() + + return nil } // createImageSBOM uses syft to generate SBOM for an image, // some code/structure migrated from https://github.com/testifysec/go-witness/blob/v0.1.12/attestation/syft/syft.go. -func (b *Builder) createImageSBOM(src string) ([]byte, error) { +func (b *Builder) createImageSBOM(img v1.Image, tagStr string) ([]byte, error) { // Get the image reference. - tag, err := name.NewTag(src, name.WeakValidation) - if err != nil { - return nil, err - } - - // Load the image tarball. - tarballImg, err := tarball.ImageFromPath(b.imagesPath, &tag) + tag, err := name.NewTag(tagStr, name.WeakValidation) if err != nil { return nil, err } @@ -138,7 +167,7 @@ func (b *Builder) createImageSBOM(src string) ([]byte, error) { return nil, err } - syftImage := image.NewImage(tarballImg, imageCachePath, image.WithTags(tag.String())) + syftImage := image.NewImage(img, imageCachePath, image.WithTags(tag.String())) if err := syftImage.Read(); err != nil { return nil, err } @@ -258,6 +287,6 @@ func (b *Builder) getNormalizedFileName(identifier string) string { } func (b *Builder) createSBOMFile(filename string) (*os.File, error) { - path := filepath.Join(b.sbomPath, b.getNormalizedFileName(filename)) + path := filepath.Join(b.tmpSBOMPath, b.getNormalizedFileName(filename)) return os.Create(path) } diff --git a/src/internal/packager/sbom/tools.go b/src/internal/packager/sbom/tools.go index 8ffe7710b0..e8c929576b 100644 --- a/src/internal/packager/sbom/tools.go +++ b/src/internal/packager/sbom/tools.go @@ -18,7 +18,8 @@ import ( // ViewSBOMFiles opens a browser to view the SBOM files and pauses for user input. func ViewSBOMFiles(tmp types.TempPaths) { - sbomViewFiles, _ := filepath.Glob(filepath.Join(tmp.Sboms, "sbom-viewer-*")) + sbomFilePath := filepath.Join(tmp.Base, "sboms") + sbomViewFiles, _ := filepath.Glob(filepath.Join(sbomFilePath, "sbom-viewer-*")) if len(sbomViewFiles) > 0 { link := sbomViewFiles[0] @@ -49,5 +50,9 @@ func OutputSBOMFiles(tmp types.TempPaths, outputDir string, packageName string) return err } + if err := utils.CreateDirectory(packagePath, 0700); err != nil { + return err + } + return utils.CreatePathAndCopy(tmp.Sboms, packagePath) } diff --git a/src/pkg/message/message.go b/src/pkg/message/message.go index 60032e7104..5b732cb089 100644 --- a/src/pkg/message/message.go +++ b/src/pkg/message/message.go @@ -164,8 +164,8 @@ func Infof(format string, a ...any) { } } -// SuccessF prints a success message. -func SuccessF(format string, a ...any) { +// Successf prints a success message. +func Successf(format string, a ...any) { message := paragraph(format, a...) pterm.Success.Println(message) } diff --git a/src/pkg/message/progress.go b/src/pkg/message/progress.go index bf5be316b4..482ccf4899 100644 --- a/src/pkg/message/progress.go +++ b/src/pkg/message/progress.go @@ -41,14 +41,26 @@ func (p *ProgressBar) Update(complete int64, text string) { } p.progress.UpdateTitle(" " + text) chunk := int(complete) - p.progress.Current - p.progress.Add(chunk) + p.Add(chunk) +} + +// Add updates the ProgressBar with completed progress. +func (p *ProgressBar) Add(n int) { + if p.progress != nil { + if p.progress.Current+n >= p.progress.Total { + // @RAZZLE TODO: This is a hack to prevent the progress bar from going over 100% and causing TUI ugliness. + overflow := p.progress.Current + n - p.progress.Total + p.progress.Total += overflow + 1 + } + p.progress.Add(n) + } } // Write updates the ProgressBar with the number of bytes in a buffer as the completed progress. func (p *ProgressBar) Write(data []byte) (int, error) { n := len(data) if p.progress != nil { - p.progress.Add(n) + p.Add(n) } return n, nil } diff --git a/src/pkg/message/spinner.go b/src/pkg/message/spinner.go index 2df82c00f1..1ce46694a7 100644 --- a/src/pkg/message/spinner.go +++ b/src/pkg/message/spinner.go @@ -14,6 +14,8 @@ import ( var activeSpinner *Spinner +var sequence = []string{` ⠋ `, ` ⠙ `, ` ⠹ `, ` ⠸ `, ` ⠼ `, ` ⠴ `, ` ⠦ `, ` ⠧ `, ` ⠇ `, ` ⠏ `} + // Spinner is a wrapper around pterm.SpinnerPrinter. type Spinner struct { spinner *pterm.SpinnerPrinter @@ -37,7 +39,7 @@ func NewProgressSpinner(format string, a ...any) *Spinner { spinner, _ = pterm.DefaultSpinner. WithRemoveWhenDone(false). // Src: https://github.com/gernest/wow/blob/master/spin/spinners.go#L335 - WithSequence(` ⠋ `, ` ⠙ `, ` ⠹ `, ` ⠸ `, ` ⠼ `, ` ⠴ `, ` ⠦ `, ` ⠧ `, ` ⠇ `, ` ⠏ `). + WithSequence(sequence...). Start(text) } diff --git a/src/pkg/packager/common.go b/src/pkg/packager/common.go index f2e95300aa..005635a926 100644 --- a/src/pkg/packager/common.go +++ b/src/pkg/packager/common.go @@ -165,9 +165,10 @@ func createPaths() (paths types.TempPaths, err error) { Base: basePath, InjectBinary: filepath.Join(basePath, "zarf-injector"), - SeedImage: filepath.Join(basePath, "seed-image.tar"), - Images: filepath.Join(basePath, "images.tar"), + SeedImage: filepath.Join(basePath, "seed-image"), + Images: filepath.Join(basePath, "images"), Components: filepath.Join(basePath, "components"), + SbomTar: filepath.Join(basePath, "sboms.tar"), Sboms: filepath.Join(basePath, "sboms"), ZarfYaml: filepath.Join(basePath, config.ZarfYAML), } @@ -184,13 +185,14 @@ func getRequestedComponentList(requestedComponents string) []string { } func (p *Packager) loadZarfPkg() error { - spinner := message.NewProgressSpinner("Loading Zarf Package %s", p.cfg.DeployOpts.PackagePath) - defer spinner.Stop() if err := p.handlePackagePath(); err != nil { return fmt.Errorf("unable to handle the provided package path: %w", err) } + spinner := message.NewProgressSpinner("Loading Zarf Package %s", p.cfg.DeployOpts.PackagePath) + defer spinner.Stop() + // Make sure the user gave us a package we can work with if utils.InvalidPath(p.cfg.DeployOpts.PackagePath) { return fmt.Errorf("unable to find the package at %s", p.cfg.DeployOpts.PackagePath) @@ -201,20 +203,48 @@ func (p *Packager) loadZarfPkg() error { return fmt.Errorf("unable to process partial package: %w", err) } - // Extract the archive - spinner.Updatef("Extracting the package, this may take a few moments") - if err := archiver.Unarchive(p.cfg.DeployOpts.PackagePath, p.tmp.Base); err != nil { - return fmt.Errorf("unable to extract the package: %w", err) + // If the package was pulled from OCI, there is no need to extract it since it is unpacked already + if p.cfg.DeployOpts.PackagePath != p.tmp.Base { + // Extract the archive + spinner.Updatef("Extracting the package, this may take a few moments") + if err := archiver.Unarchive(p.cfg.DeployOpts.PackagePath, p.tmp.Base); err != nil { + return fmt.Errorf("unable to extract the package: %w", err) + } } // Load the config from the extracted archive zarf.yaml spinner.Updatef("Loading the zarf package config") - configPath := filepath.Join(p.tmp.Base, config.ZarfYAML) + configPath := p.tmp.ZarfYaml if err := p.readYaml(configPath, true); err != nil { return fmt.Errorf("unable to read the zarf.yaml in %s: %w", p.tmp.Base, err) } - // If SBOM files exist, temporarily place them in the deploy directory + // Get a list of paths for the components of the package + components, err := os.ReadDir(p.tmp.Components) + if err != nil { + return fmt.Errorf("unable to get a list of components... %w", err) + } + for _, component := range components { + // If the components are tarballs, extract them! + componentPath := filepath.Join(p.tmp.Components, component.Name()) + if !component.IsDir() && strings.HasSuffix(component.Name(), ".tar") { + if err := archiver.Unarchive(componentPath, p.tmp.Components); err != nil { + return fmt.Errorf("unable to extract the component: %w", err) + } + + // After extracting the component, remove the compressed tarball to release disk space + _ = os.Remove(filepath.Join(p.tmp.Components, component.Name())) + } + } + + // If a SBOM tar file exist, temporarily place them in the deploy directory + _, tarErr := os.Stat(p.tmp.SbomTar) + if tarErr == nil { + if err = archiver.Unarchive(p.tmp.SbomTar, p.tmp.Sboms); err != nil { + return fmt.Errorf("unable to extract the sbom data from the component: %w", err) + } + } + p.cfg.SBOMViewFiles, _ = filepath.Glob(filepath.Join(p.tmp.Sboms, "sbom-viewer-*")) if err := sbom.OutputSBOMFiles(p.tmp, config.ZarfSBOMDir, ""); err != nil { // Don't stop the deployment, let the user decide if they want to continue the deployment diff --git a/src/pkg/packager/create.go b/src/pkg/packager/create.go old mode 100644 new mode 100755 index 8a133a12ee..988af79f30 --- a/src/pkg/packager/create.go +++ b/src/pkg/packager/create.go @@ -157,9 +157,9 @@ func (p *Packager) Create(baseDir string) error { doPull := func() error { imgConfig := images.ImgConfig{ - TarballPath: p.tmp.Images, - ImgList: imgList, - Insecure: config.CommonOptions.Insecure, + ImagesPath: p.tmp.Images, + ImgList: imgList, + Insecure: config.CommonOptions.Insecure, } return imgConfig.PullAll() @@ -174,7 +174,26 @@ func (p *Packager) Create(baseDir string) error { if p.cfg.CreateOpts.SkipSBOM { message.Debug("Skipping image SBOM processing per --skip-sbom flag") } else { - sbom.Catalog(componentSBOMs, imgList, p.tmp.Images, p.tmp.Sboms) + if err := sbom.Catalog(componentSBOMs, imgList, p.tmp); err != nil { + return fmt.Errorf("unable to create an SBOM catalog for the package: %w", err) + } + } + + // Process the component directories into compressed tarballs + // NOTE: This is purposefully being done after the SBOM cataloging + for _, component := range p.cfg.Pkg.Components { + // Make the component a tar archive + componentPaths, _ := p.createOrGetComponentPaths(component) + componentName := fmt.Sprintf("%s.%s", component.Name, "tar") + componentTarPath := filepath.Join(p.tmp.Components, componentName) + if err := archiver.Archive([]string{componentPaths.Base}, componentTarPath); err != nil { + return fmt.Errorf("unable to create package: %w", err) + } + + // Remove the deflated component directory + if err := os.RemoveAll(filepath.Join(p.tmp.Components, component.Name)); err != nil { + return fmt.Errorf("unable to remove the component directory (%s): %w", componentPaths.Base, err) + } } // In case the directory was changed, reset to prevent breaking relative target paths. @@ -182,17 +201,17 @@ func (p *Packager) Create(baseDir string) error { _ = os.Chdir(originalDir) } - // Use the output path if the user specified it. - packageName := filepath.Join(p.cfg.CreateOpts.OutputDirectory, p.GetPackageName()) - - // Try to remove the package if it already exists. - _ = os.RemoveAll(packageName) - // Save the transformed config. if err := p.writeYaml(); err != nil { return fmt.Errorf("unable to write zarf.yaml: %w", err) } + // Use the output path if the user specified it. + packageName := filepath.Join(p.cfg.CreateOpts.OutputDirectory, p.GetPackageName()) + + // Try to remove the package if it already exists. + _ = os.Remove(packageName) + // Make the archive archiveSrc := []string{p.tmp.Base + string(os.PathSeparator)} if err := archiver.Archive(archiveSrc, packageName); err != nil { @@ -242,15 +261,21 @@ func (p *Packager) Create(baseDir string) error { } // Output the SBOM files into a directory if specified. - if p.cfg.CreateOpts.SBOMOutputDir != "" { - if err := sbom.OutputSBOMFiles(p.tmp, p.cfg.CreateOpts.SBOMOutputDir, p.cfg.Pkg.Metadata.Name); err != nil { + if p.cfg.CreateOpts.SBOMOutputDir != "" || p.cfg.CreateOpts.ViewSBOM { + if err = archiver.Unarchive(p.tmp.SbomTar, p.tmp.Sboms); err != nil { return err } - } - // Open a browser to view the SBOM if specified. - if p.cfg.CreateOpts.ViewSBOM { - sbom.ViewSBOMFiles(p.tmp) + if p.cfg.CreateOpts.SBOMOutputDir != "" { + if err := sbom.OutputSBOMFiles(p.tmp, p.cfg.CreateOpts.SBOMOutputDir, p.cfg.Pkg.Metadata.Name); err != nil { + return err + } + } + + // Open a browser to view the SBOM if specified. + if p.cfg.CreateOpts.ViewSBOM { + sbom.ViewSBOMFiles(p.tmp) + } } return nil diff --git a/src/pkg/packager/deploy.go b/src/pkg/packager/deploy.go index e058a3151d..f172cea30c 100644 --- a/src/pkg/packager/deploy.go +++ b/src/pkg/packager/deploy.go @@ -71,7 +71,7 @@ func (p *Packager) Deploy() error { } // Notify all the things about the successful deployment - message.SuccessF("Zarf deployment complete") + message.Successf("Zarf deployment complete") p.printTablesForDeployment(deployedComponents) // Save deployed package information to k8s @@ -387,11 +387,11 @@ func (p *Packager) pushImagesToRegistry(componentImages []string, noImgChecksum } imgConfig := images.ImgConfig{ - TarballPath: p.tmp.Images, - ImgList: componentImages, - NoChecksum: noImgChecksum, - RegInfo: p.cfg.State.RegistryInfo, - Insecure: config.CommonOptions.Insecure, + ImagesPath: p.tmp.Images, + ImgList: componentImages, + NoChecksum: noImgChecksum, + RegInfo: p.cfg.State.RegistryInfo, + Insecure: config.CommonOptions.Insecure, } return utils.Retry(func() error { diff --git a/src/pkg/packager/inspect.go b/src/pkg/packager/inspect.go index 186d969643..fd2298440d 100644 --- a/src/pkg/packager/inspect.go +++ b/src/pkg/packager/inspect.go @@ -6,17 +6,68 @@ package packager import ( "fmt" + "path/filepath" + "strings" "github.com/defenseunicorns/zarf/src/internal/packager/sbom" + "github.com/defenseunicorns/zarf/src/pkg/message" "github.com/defenseunicorns/zarf/src/pkg/utils" + "github.com/mholt/archiver/v3" + ocispec "github.com/opencontainers/image-spec/specs-go/v1" "github.com/pterm/pterm" + "oras.land/oras-go/v2/registry" ) // Inspect list the contents of a package. func (p *Packager) Inspect(includeSBOM bool, outputSBOM string) error { + if utils.IsOCIURL(p.cfg.DeployOpts.PackagePath) { + spinner := message.NewProgressSpinner("Loading Zarf Package %s", p.cfg.DeployOpts.PackagePath) + ref, err := registry.ParseReference(strings.TrimPrefix(p.cfg.DeployOpts.PackagePath, "oci://")) + if err != nil { + return err + } + + dst, err := utils.NewOrasRemote(ref) + if err != nil { + return err + } + + // get the manifest + spinner.Updatef("Fetching the manifest for %s", p.cfg.DeployOpts.PackagePath) + layers, err := getLayers(dst) + if err != nil { + return err + } + spinner.Updatef("Loading Zarf Package %s", p.cfg.DeployOpts.PackagePath) + zarfYamlDesc := utils.Find(layers, func(d ocispec.Descriptor) bool { + return d.Annotations["org.opencontainers.image.title"] == "zarf.yaml" + }) + err = pullLayer(dst, zarfYamlDesc, p.tmp.ZarfYaml) + if err != nil { + return err + } - if err := p.loadZarfPkg(); err != nil { - return fmt.Errorf("unable to load the package: %w", err) + if includeSBOM || outputSBOM != "" { + sbmomsTarDesc := utils.Find(layers, func(d ocispec.Descriptor) bool { + return d.Annotations["org.opencontainers.image.title"] == "sboms.tar" + }) + err = pullLayer(dst, sbmomsTarDesc, p.tmp.SbomTar) + if err != nil { + return err + } + if err := archiver.Unarchive(p.tmp.SbomTar, filepath.Join(p.tmp.Base, "sboms")); err != nil { + return err + } + } + err = utils.ReadYaml(p.tmp.ZarfYaml, &p.cfg.Pkg) + if err != nil { + return err + } + spinner.Successf("Loaded Zarf Package %s", p.cfg.DeployOpts.PackagePath) + } else { + if err := p.loadZarfPkg(); err != nil { + return fmt.Errorf("unable to load the package: %w", err) + } } pterm.Println() diff --git a/src/pkg/packager/interactive.go b/src/pkg/packager/interactive.go index 0dd43929f5..6acba9bdec 100644 --- a/src/pkg/packager/interactive.go +++ b/src/pkg/packager/interactive.go @@ -35,7 +35,7 @@ func (p *Packager) confirmAction(userMessage string, sbomViewFiles []string) (co // Display prompt if not auto-confirmed if config.CommonOptions.Confirm { - message.SuccessF("%s Zarf package confirmed", userMessage) + message.Successf("%s Zarf package confirmed", userMessage) return config.CommonOptions.Confirm } diff --git a/src/pkg/packager/network.go b/src/pkg/packager/network.go index ae7de2486c..9076357257 100644 --- a/src/pkg/packager/network.go +++ b/src/pkg/packager/network.go @@ -4,6 +4,8 @@ import ( "context" "crypto/sha256" "encoding/hex" + "encoding/json" + "errors" "fmt" "io" "net/http" @@ -11,10 +13,17 @@ import ( "os" "path/filepath" "strings" + "sync" "github.com/defenseunicorns/zarf/src/config" "github.com/defenseunicorns/zarf/src/pkg/message" "github.com/defenseunicorns/zarf/src/pkg/utils" + ocispec "github.com/opencontainers/image-spec/specs-go/v1" + "oras.land/oras-go/v2" + "oras.land/oras-go/v2/content" + "oras.land/oras-go/v2/content/file" + "oras.land/oras-go/v2/registry" + "oras.land/oras-go/v2/registry/remote/errcode" ) // handlePackagePath If provided package is a URL download it to a temp directory. @@ -30,11 +39,19 @@ func (p *Packager) handlePackagePath() error { return nil } + // Handle case where deploying remote package stored in an OCI registry + if utils.IsOCIURL(opts.PackagePath) { + return p.handleOciPackage() + } + // Handle case where deploying remote package validated via sget if strings.HasPrefix(opts.PackagePath, "sget://") { return p.handleSgetPackage() } + spinner := message.NewProgressSpinner("Loading Zarf Package %s", opts.PackagePath) + defer spinner.Stop() + if !config.CommonOptions.Insecure && opts.Shasum == "" { return fmt.Errorf("remote package provided without a shasum, use --insecure to ignore") } @@ -76,6 +93,7 @@ func (p *Packager) handlePackagePath() error { opts.PackagePath = localPath + spinner.Success() return nil } @@ -84,6 +102,9 @@ func (p *Packager) handleSgetPackage() error { opts := p.cfg.DeployOpts + spinner := message.NewProgressSpinner("Loading Zarf Package %s", opts.PackagePath) + defer spinner.Stop() + // Create the local file for the package localPath := filepath.Join(p.tmp.Base, "remote.tar.zst") destinationFile, err := os.Create(localPath) @@ -109,5 +130,176 @@ func (p *Packager) handleSgetPackage() error { p.cfg.DeployOpts.PackagePath = localPath + spinner.Success() return nil } + +func (p *Packager) handleOciPackage() error { + message.Debug("packager.handleOciPackage()") + ref, err := registry.ParseReference(strings.TrimPrefix(p.cfg.DeployOpts.PackagePath, "oci://")) + if err != nil { + return fmt.Errorf("failed to parse OCI reference: %w", err) + } + + outDir := p.tmp.Base + message.Debugf("Pulling %s", ref.String()) + message.Infof("Pulling Zarf package from %s", ref) + + src, err := utils.NewOrasRemote(ref) + if err != nil { + return err + } + + estimatedBytes, err := getOCIPackageSize(src, ref) + if err != nil { + return err + } + + dst, err := file.New(outDir) + if err != nil { + return err + } + defer dst.Close() + + // Create a thread to update a progress bar as we save the package to disk + doneSaving := make(chan int) + var wg sync.WaitGroup + wg.Add(1) + src.ProgressBar = nil // NOTE: Disable this inbuilt progress bar so we don't double render a spinner + go utils.RenderProgressBarForLocalDirWrite(outDir, estimatedBytes, &wg, doneSaving, "Pulling Zarf package data") + + copyOpts := oras.DefaultCopyOptions + copyOpts.Concurrency = p.cfg.PublishOpts.CopyOptions.Concurrency + copyOpts.OnCopySkipped = func(ctx context.Context, desc ocispec.Descriptor) error { + title := desc.Annotations[ocispec.AnnotationTitle] + var format string + if title != "" { + format = fmt.Sprintf("%s %s", desc.Digest.Encoded()[:12], utils.First30last30(title)) + } else { + format = fmt.Sprintf("%s [%s]", desc.Digest.Encoded()[:12], desc.MediaType) + } + message.Successf(format) + return nil + } + copyOpts.PostCopy = copyOpts.OnCopySkipped + + _, err = oras.Copy(src.Context, src.Repository, ref.Reference, dst, ref.Reference, copyOpts) + if err != nil { + return err + } + + // Send a signal to the progress bar that we're done and wait for it to finish + doneSaving <- 1 + wg.Wait() + + message.Debugf("Pulled %s", ref.String()) + message.Successf("Pulled %s", ref.String()) + + p.cfg.DeployOpts.PackagePath = outDir + return nil +} + +// isManifestUnsupported returns true if the error is an unsupported artifact manifest error. +// +// This function was copied verbatim from https://github.com/oras-project/oras/blob/main/cmd/oras/push.go +func isManifestUnsupported(err error) bool { + var errResp *errcode.ErrorResponse + if !errors.As(err, &errResp) || errResp.StatusCode != http.StatusBadRequest { + return false + } + + var errCode errcode.Error + if !errors.As(errResp, &errCode) { + return false + } + + // As of November 2022, ECR is known to return UNSUPPORTED error when + // pulling an OCI artifact manifest. + switch errCode.Code { + case errcode.ErrorCodeManifestInvalid, errcode.ErrorCodeUnsupported: + return true + } + return false +} + +func getOCIPackageSize(src *utils.OrasRemote, ref registry.Reference) (int64, error) { + var total int64 + // get the manifest descriptor + // ref.Reference can be a tag or a digest + descriptor, err := src.Resolve(src.Context, ref.Reference) + if err != nil { + return 0, err + } + + // get the manifest itself + pulled, err := content.FetchAll(src.Context, src, descriptor) + if err != nil { + return 0, err + } + manifest := ocispec.Manifest{} + artifact := ocispec.Artifact{} + var layers []ocispec.Descriptor + // if the manifest is an artifact, unmarshal it as an artifact + // otherwise, unmarshal it as a manifest + if descriptor.MediaType == ocispec.MediaTypeArtifactManifest { + if err = json.Unmarshal(pulled, &artifact); err != nil { + return 0, err + } + layers = artifact.Blobs + } else { + if err = json.Unmarshal(pulled, &manifest); err != nil { + return 0, err + } + layers = manifest.Layers + } + + for _, layer := range layers { + total += layer.Size + } + + return total, nil +} + +// getLayers returns the manifest layers of a Zarf OCI package +func getLayers(dst *utils.OrasRemote) ([]ocispec.Descriptor, error) { + // get the manifest descriptor + // ref.Reference can be a tag or a digest + descriptor, err := dst.Resolve(dst.Context, dst.Reference.Reference) + if err != nil { + return nil, err + } + + // get the manifest itself + pulled, err := content.FetchAll(dst.Context, dst, descriptor) + if err != nil { + return nil, err + } + manifest := ocispec.Manifest{} + artifact := ocispec.Artifact{} + var layers []ocispec.Descriptor + // if the manifest is an artifact, unmarshal it as an artifact + // otherwise, unmarshal it as a manifest + if descriptor.MediaType == ocispec.MediaTypeArtifactManifest { + if err = json.Unmarshal(pulled, &artifact); err != nil { + return nil, err + } + layers = artifact.Blobs + } else { + if err = json.Unmarshal(pulled, &manifest); err != nil { + return nil, err + } + layers = manifest.Layers + } + + return layers, nil +} + +// pullLayer fetches a single layer from a Zarf OCI package +func pullLayer(dst *utils.OrasRemote, desc ocispec.Descriptor, out string) error { + bytes, err := content.FetchAll(dst.Context, dst, desc) + if err != nil { + return err + } + err = utils.WriteFile(out, bytes) + return err +} diff --git a/src/pkg/packager/publish.go b/src/pkg/packager/publish.go new file mode 100644 index 0000000000..548c01f819 --- /dev/null +++ b/src/pkg/packager/publish.go @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: 2021-Present The Zarf Authors + +// Package packager contains functions for interacting with, managing and deploying Zarf packages. +package packager + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + + "github.com/defenseunicorns/zarf/src/config" + "github.com/defenseunicorns/zarf/src/pkg/message" + "github.com/defenseunicorns/zarf/src/pkg/utils" + "github.com/mholt/archiver/v3" + ocispec "github.com/opencontainers/image-spec/specs-go/v1" + + "oras.land/oras-go/v2" + "oras.land/oras-go/v2/content" + "oras.land/oras-go/v2/content/file" + "oras.land/oras-go/v2/registry" +) + +// ZarfLayerMediaType is the media type for Zarf layers. +const ( + ZarfLayerMediaTypeTarZstd = "application/vnd.zarf.layer.v1.tar+zstd" + ZarfLayerMediaTypeTarGzip = "application/vnd.zarf.layer.v1.tar+gzip" + ZarfLayerMediaTypeYaml = "application/vnd.zarf.layer.v1.yaml" + ZarfLayerMediaTypeJSON = "application/vnd.zarf.layer.v1.json" + ZarfLayerMediaTypeTxt = "application/vnd.zarf.layer.v1.txt" + ZarfLayerMediaTypeUnknown = "application/vnd.zarf.layer.v1.unknown" +) + +// parseZarfLayerMediaType returns the Zarf layer media type for the given filename. +func parseZarfLayerMediaType(filename string) string { + // since we are controlling the filenames, we can just use the extension + switch filepath.Ext(filename) { + case ".zst": + return ZarfLayerMediaTypeTarZstd + case ".gz": + return ZarfLayerMediaTypeTarGzip + case ".yaml": + return ZarfLayerMediaTypeYaml + case ".json": + return ZarfLayerMediaTypeJSON + case ".txt": + return ZarfLayerMediaTypeTxt + default: + return ZarfLayerMediaTypeUnknown + } +} + +// Publish publishes the package to a registry +// +// This is a wrapper around the oras library +// and much of the code was adapted from the oras CLI - https://github.com/oras-project/oras/blob/main/cmd/oras/push.go +// +// Authentication is handled via the Docker config file created w/ `zarf tools registry login` +func (p *Packager) Publish() error { + p.cfg.DeployOpts.PackagePath = p.cfg.PublishOpts.PackagePath + if err := p.loadZarfPkg(); err != nil { + return fmt.Errorf("unable to load the package: %w", err) + } + + paths := []string{ + p.tmp.ZarfYaml, + p.tmp.SbomTar, + filepath.Join(p.tmp.Images, "index.json"), + filepath.Join(p.tmp.Images, "oci-layout"), + } + // if checksums.txt file exists, include it + if !utils.InvalidPath(filepath.Join(p.tmp.Base, "checksums.txt")) { + paths = append(paths, filepath.Join(p.tmp.Base, "checksums.txt")) + } + + if p.cfg.Pkg.Kind == "ZarfInitConfig" { + seedImagePaths := []string{ + filepath.Join(p.tmp.SeedImage, "index.json"), + filepath.Join(p.tmp.SeedImage, "oci-layout"), + } + seedImageLayers, err := filepath.Glob(filepath.Join(p.tmp.SeedImage, "blobs", "sha256", "*")) + if err != nil { + return err + } + seedImagePaths = append(seedImagePaths, seedImageLayers...) + paths = append(paths, seedImagePaths...) + } + componentDirs, err := filepath.Glob(filepath.Join(p.tmp.Components, "*")) + if err != nil { + return err + } + componentTarballs := []string{} + + // repackage the component directories into tarballs + for _, componentDir := range componentDirs { + dst := filepath.Join(p.tmp.Components, filepath.Base(componentDir)+".tar") + err = archiver.Archive([]string{componentDir}, dst) + if err != nil { + return err + } + componentTarballs = append(componentTarballs, dst) + _ = os.RemoveAll(componentDir) + } + paths = append(paths, componentTarballs...) + imagesLayers, err := filepath.Glob(filepath.Join(p.tmp.Images, "blobs", "sha256", "*")) + if err != nil { + return err + } + paths = append(paths, imagesLayers...) + ref, err := p.ref("") + if err != nil { + return err + } + message.HeaderInfof("📦 PACKAGE PUBLISH %s:%s", p.cfg.Pkg.Metadata.Name, ref.Reference) + err = p.publish(ref, paths) + if err != nil { + return fmt.Errorf("unable to publish package %s: %w", ref, err) + } + return nil +} + +func (p *Packager) publish(ref registry.Reference, paths []string) error { + message.Infof("Publishing package to %s", ref) + spinner := message.NewProgressSpinner("") + defer spinner.Stop() + + // destination remote + dst, err := utils.NewOrasRemote(ref) + if err != nil { + return err + } + ctx := dst.Context + + // source file store + src, err := file.New(p.tmp.Base) + if err != nil { + return err + } + defer src.Close() + + var descs []ocispec.Descriptor + + for idx, path := range paths { + name, err := filepath.Rel(p.tmp.Base, path) + if err != nil { + return err + } + spinner.Updatef("Preparing layer %d/%d: %s", idx+1, len(paths), name) + + mediaType := parseZarfLayerMediaType(name) + + desc, err := src.Add(ctx, name, mediaType, path) + if err != nil { + return err + } + descs = append(descs, desc) + } + spinner.Successf("Prepared %d layers", len(descs)) + + copyOpts := oras.DefaultCopyOptions + copyOpts.Concurrency = p.cfg.PublishOpts.CopyOptions.Concurrency + copyOpts.OnCopySkipped = utils.PrintLayerExists + copyOpts.PostCopy = utils.PrintLayerExists + + var root ocispec.Descriptor + + // try to push an ArtifactManifest first + // not every registry supports ArtifactManifests, so fallback to an ImageManifest if the push fails + // see https://oras.land/implementors/#registries-supporting-oci-artifacts + root, err = p.publishArtifact(dst, src, descs, copyOpts) + if err != nil { + // reset the progress bar between attempts + dst.ProgressBar.Stop() + + // log the error, the expected error is a 400 manifest invalid + message.Debug("ArtifactManifest push failed with the following error, falling back to an ImageManifest push:", err) + + // if the error returned from the push is not an expected error, then return the error + if !isManifestUnsupported(err) { + return err + } + // fallback to an ImageManifest push + root, err = p.publishImage(dst, src, descs, copyOpts) + if err != nil { + return err + } + } + dst.ProgressBar.Successf("Published %s [%s]", ref, root.MediaType) + fmt.Println() + flags := "" + if config.CommonOptions.Insecure { + flags = "--insecure" + } + message.Info("To inspect/deploy/pull:") + message.Infof("zarf package inspect oci://%s %s", ref, flags) + message.Infof("zarf package deploy oci://%s %s", ref, flags) + message.Infof("zarf package pull oci://%s %s", ref, flags) + return nil +} + +func (p *Packager) publishArtifact(dst *utils.OrasRemote, src *file.Store, descs []ocispec.Descriptor, copyOpts oras.CopyOptions) (root ocispec.Descriptor, err error) { + var total int64 + for _, desc := range descs { + total += desc.Size + } + packOpts := p.cfg.PublishOpts.PackOptions + + // first attempt to do a ArtifactManifest push + root, err = pack(dst.Context, ocispec.MediaTypeArtifactManifest, descs, src, packOpts) + if err != nil { + return root, err + } + total += root.Size + + dst.ProgressBar = message.NewProgressBar(total, fmt.Sprintf("Publishing %s:%s", dst.Reference.Repository, dst.Reference.Reference)) + defer dst.ProgressBar.Stop() + + // attempt to push the artifact manifest + _, err = oras.Copy(dst.Context, src, root.Digest.String(), dst, dst.Reference.Reference, copyOpts) + return root, err +} + +func (p *Packager) publishImage(dst *utils.OrasRemote, src *file.Store, descs []ocispec.Descriptor, copyOpts oras.CopyOptions) (root ocispec.Descriptor, err error) { + var total int64 + for _, desc := range descs { + total += desc.Size + } + // assumes referrers API is not supported since OCI artifact + // media type is not supported + dst.SetReferrersCapability(false) + + // fallback to an ImageManifest push + manifestConfigDesc, manifestConfigContent, err := p.generateManifestConfigFile() + if err != nil { + return root, err + } + // push the manifest config + // since this config is so tiny, and the content is not used again + // it is not logged to the progress, but will error if it fails + err = dst.Push(dst.Context, manifestConfigDesc, bytes.NewReader(manifestConfigContent)) + if err != nil { + return root, err + } + packOpts := p.cfg.PublishOpts.PackOptions + packOpts.ConfigDescriptor = &manifestConfigDesc + packOpts.PackImageManifest = true + root, err = pack(dst.Context, ocispec.MediaTypeImageManifest, descs, src, packOpts) + if err != nil { + return root, err + } + total += root.Size + manifestConfigDesc.Size + + dst.ProgressBar = message.NewProgressBar(total, fmt.Sprintf("Publishing %s:%s", dst.Reference.Repository, dst.Reference.Reference)) + defer dst.ProgressBar.Stop() + // attempt to push the image manifest + _, err = oras.Copy(dst.Context, src, root.Digest.String(), dst, dst.Reference.Reference, copyOpts) + if err != nil { + return root, err + } + + return root, nil +} + +func (p *Packager) generateManifestConfigFile() (ocispec.Descriptor, []byte, error) { + // Unless specified, an empty manifest config will be used: `{}` + // which causes an error on Google Artifact Registry + // to negate this, we create a simple manifest config with some build metadata + // the contents of this file are not used by Zarf + type OCIConfigPartial struct { + Architecture string `json:"architecture"` + OCIVersion string `json:"ociVersion"` + Annotations map[string]string `json:"annotations,omitempty"` + } + + annotations := map[string]string{ + "org.opencontainers.image.title": p.cfg.Pkg.Metadata.Name, + "org.opencontainers.image.description": p.cfg.Pkg.Metadata.Description, + } + + manifestConfig := OCIConfigPartial{ + Architecture: p.cfg.Pkg.Build.Architecture, + OCIVersion: "1.0.1", + Annotations: annotations, + } + manifestConfigBytes, err := json.Marshal(manifestConfig) + if err != nil { + return ocispec.Descriptor{}, nil, err + } + manifestConfigDesc := content.NewDescriptorFromBytes("application/vnd.unknown.config.v1+json", manifestConfigBytes) + + return manifestConfigDesc, manifestConfigBytes, nil +} + +// pack creates an artifact/image manifest from the provided descriptors and pushes it to the store +func pack(ctx context.Context, artifactType string, descs []ocispec.Descriptor, src *file.Store, packOpts oras.PackOptions) (ocispec.Descriptor, error) { + root, err := oras.Pack(ctx, src, artifactType, descs, packOpts) + if err != nil { + return ocispec.Descriptor{}, err + } + if err = src.Tag(ctx, root, root.Digest.String()); err != nil { + return ocispec.Descriptor{}, err + } + + return root, nil +} + +// ref returns a registry.Reference using metadata from the package's build config and the PublishOpts +// +// if skeleton is not empty, the architecture will be replaced with the skeleton string (e.g. "skeleton") +func (p *Packager) ref(skeleton string) (registry.Reference, error) { + ver := p.cfg.Pkg.Metadata.Version + if len(ver) == 0 { + return registry.Reference{}, errors.New("version is required for publishing") + } + arch := p.cfg.Pkg.Build.Architecture + // changes package ref from "name:version-arch" to "name:version-skeleton" + if len(skeleton) > 0 { + arch = skeleton + } + ref := registry.Reference{ + Registry: p.cfg.PublishOpts.Reference.Registry, + Repository: fmt.Sprintf("%s/%s", p.cfg.PublishOpts.Reference.Repository, p.cfg.Pkg.Metadata.Name), + Reference: fmt.Sprintf("%s-%s", ver, arch), + } + if len(p.cfg.PublishOpts.Reference.Repository) == 0 { + ref.Repository = p.cfg.Pkg.Metadata.Name + } + err := ref.Validate() + if err != nil { + return registry.Reference{}, err + } + return ref, nil +} diff --git a/src/pkg/packager/pull.go b/src/pkg/packager/pull.go new file mode 100644 index 0000000000..6ed8efcba0 --- /dev/null +++ b/src/pkg/packager/pull.go @@ -0,0 +1,33 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: 2021-Present The Zarf Authors + +// Package packager contains functions for interacting with, managing and deploying Zarf packages. +package packager + +import ( + "fmt" + "path/filepath" + + "github.com/mholt/archiver/v3" +) + +// Pull pulls a Zarf package and saves it as a compressed tarball. +func (p *Packager) Pull() error { + err := p.loadZarfPkg() + if err != nil { + return err + } + + // Get all the layers from within the temp directory + allTheLayers, err := filepath.Glob(filepath.Join(p.tmp.Base, "*")) + if err != nil { + return err + } + + name := fmt.Sprintf("zarf-package-%s-%s-%s.tar.zst", p.cfg.Pkg.Metadata.Name, p.cfg.Pkg.Build.Architecture, p.cfg.Pkg.Metadata.Version) + err = archiver.Archive(allTheLayers, name) + if err != nil { + return err + } + return nil +} diff --git a/src/pkg/utils/bytes.go b/src/pkg/utils/bytes.go index bf894fbc7d..f09e35c487 100644 --- a/src/pkg/utils/bytes.go +++ b/src/pkg/utils/bytes.go @@ -6,8 +6,13 @@ package utils import ( + "fmt" "math" "strconv" + "sync" + "time" + + "github.com/defenseunicorns/zarf/src/pkg/message" ) // RoundUp rounds a float64 to the given number of decimal places. @@ -50,3 +55,36 @@ func ByteFormat(inputNum float64, precision int) string { return strconv.FormatFloat(returnVal, 'f', precision, 64) + unit } + +// RenderProgressBarForLocalDirWrite creates a progress bar that continuously tracks the progress of writing files to a local directory and all of its subdirectories. +// NOTE: This function runs infinitely until the completeChan is triggered, this function should be run in a goroutine while a different thread/process is writing to the directory. +func RenderProgressBarForLocalDirWrite(filepath string, expectedTotal int64, wg *sync.WaitGroup, completeChan chan int, updateText string) { + + // Create a progress bar + title := fmt.Sprintf("Pulling Zarf package data (%s of %s)", ByteFormat(float64(0), 2), ByteFormat(float64(expectedTotal), 2)) + progressBar := message.NewProgressBar(expectedTotal, title) + + for { + select { + case <-completeChan: + // Send success message + progressBar.Successf("%s (%s)", updateText, ByteFormat(float64(expectedTotal), 2)) + wg.Done() + return + + default: + // Read the directory size + currentBytes, dirErr := GetDirSize(filepath) + if dirErr != nil { + message.Warnf("unable to get the updated progress of the image pull: %s", dirErr.Error()) + time.Sleep(200 * time.Millisecond) + continue + } + + // Update the progress bar with the current size + title := fmt.Sprintf("%s (%s of %s)", updateText, ByteFormat(float64(expectedTotal), 2), ByteFormat(float64(currentBytes), 2)) + progressBar.Update(currentBytes, title) + time.Sleep(200 * time.Millisecond) + } + } +} diff --git a/src/pkg/utils/io.go b/src/pkg/utils/io.go old mode 100644 new mode 100755 index f9400b52d6..c85e086811 --- a/src/pkg/utils/io.go +++ b/src/pkg/utils/io.go @@ -13,7 +13,6 @@ import ( "net/http" "os" "os/exec" - "path" "path/filepath" "regexp" "strings" @@ -152,7 +151,7 @@ func RecursiveFileList(dir string, pattern *regexp.Regexp) (files []string, err // CreateFilePath creates the parent directory for the given file path. func CreateFilePath(destination string) error { - parentDest := path.Dir(destination) + parentDest := filepath.Dir(destination) return CreateDirectory(parentDest, 0700) } @@ -235,3 +234,21 @@ func IsTextFile(path string) (bool, error) { return hasText || hasJSON || hasXML, nil } + +// GetDirSize walks through all files and directories in the provided path and returns the total size in bytes. +func GetDirSize(path string) (int64, error) { + dirSize := int64(0) + + // Walk through all files in the path + err := filepath.Walk(path, func(_ string, info os.FileInfo, err error) error { + if err != nil { + return err + } + if !info.IsDir() { + dirSize += info.Size() + } + return nil + }) + + return dirSize, err +} diff --git a/src/pkg/utils/json.go b/src/pkg/utils/json.go new file mode 100644 index 0000000000..17987c2838 --- /dev/null +++ b/src/pkg/utils/json.go @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: 2021-Present The Zarf Authors + +// Package utils provides generic helper functions. +package utils + +import ( + "encoding/json" + "os" +) + +// WriteJSON writes a given json struct to a json file on disk. +func WriteJSON(path string, v any) error { + // Remove any file that might already exist + _ = os.Remove(path) + + data, err := json.Marshal(v) + if err != nil { + return err + } + + // Create the index.json file and save the data to it + return os.WriteFile(path, data, 0644) +} diff --git a/src/pkg/utils/network.go b/src/pkg/utils/network.go index c66f030673..04ed94cb56 100644 --- a/src/pkg/utils/network.go +++ b/src/pkg/utils/network.go @@ -29,6 +29,12 @@ func IsURL(source string) bool { return err == nil && parsedURL.Scheme != "" && parsedURL.Host != "" } +// IsOCIURL returns true if the given URL is an OCI URL. +func IsOCIURL(source string) bool { + parsedURL, err := url.Parse(source) + return err == nil && parsedURL.Scheme == "oci" +} + // DoHostnamesMatch returns a boolean indicating if the hostname of two different URLs are the same. func DoHostnamesMatch(url1 string, url2 string) (bool, error) { parsedURL1, err := url.Parse(url1) diff --git a/src/pkg/utils/oras.go b/src/pkg/utils/oras.go new file mode 100644 index 0000000000..f122ca4eb0 --- /dev/null +++ b/src/pkg/utils/oras.go @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: 2021-Present The Zarf Authors + +// Package utils provides generic helper functions. +package utils + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + + zarfconfig "github.com/defenseunicorns/zarf/src/config" + "github.com/defenseunicorns/zarf/src/pkg/message" + "github.com/docker/cli/cli/config" + "github.com/docker/cli/cli/config/configfile" + ocispec "github.com/opencontainers/image-spec/specs-go/v1" + "oras.land/oras-go/v2/registry" + "oras.land/oras-go/v2/registry/remote" + "oras.land/oras-go/v2/registry/remote/auth" + "oras.land/oras-go/v2/registry/remote/retry" +) + +// OrasRemote is a wrapper around the Oras remote repository that includes a progress bar for interactive feedback. +type OrasRemote struct { + *remote.Repository + context.Context + *message.ProgressBar +} + +// withScopes returns a context with the given scopes. +// +// This is needed for pushing to Docker Hub. +func withScopes(ref registry.Reference) context.Context { + // For pushing to Docker Hub, we need to set the scope to the repository with pull+push actions, otherwise a 401 is returned + scopes := []string{ + fmt.Sprintf("repository:%s:pull,push", ref.Repository), + } + return auth.WithScopes(context.Background(), scopes...) +} + +// withAuthClient returns an auth client for the given reference. +// +// The credentials are pulled using Docker's default credential store. +func (o *OrasRemote) withAuthClient(ref registry.Reference) (*auth.Client, error) { + cfg, err := config.Load(config.Dir()) + if err != nil { + return &auth.Client{}, err + } + if !cfg.ContainsAuth() { + return &auth.Client{}, errors.New("no docker config file found, run 'zarf tools registry login --help'") + } + + configs := []*configfile.ConfigFile{cfg} + + var key = ref.Registry + if key == "registry-1.docker.io" { + // Docker stores its credentials under the following key, otherwise credentials use the registry URL + key = "https://index.docker.io/v1/" + } + + authConf, err := configs[0].GetCredentialsStore(key).Get(key) + if err != nil { + return &auth.Client{}, fmt.Errorf("unable to get credentials for %s: %w", key, err) + } + + cred := auth.Credential{ + Username: authConf.Username, + Password: authConf.Password, + AccessToken: authConf.RegistryToken, + RefreshToken: authConf.IdentityToken, + } + + transport := http.DefaultTransport.(*http.Transport).Clone() + transport.TLSClientConfig.InsecureSkipVerify = zarfconfig.CommonOptions.Insecure + + client := &auth.Client{ + Credential: auth.StaticCredential(ref.Registry, cred), + Cache: auth.NewCache(), + Client: &http.Client{ + Transport: retry.NewTransport(transport), + }, + } + client.SetUserAgent("zarf/" + zarfconfig.CLIVersion) + + client.Client.Transport = NewTransport(client.Client.Transport, o) + + return client, nil +} + +// Transport is an http.RoundTripper that keeps track of the in-flight +// request and add hooks to report HTTP tracing events. +type Transport struct { + http.RoundTripper + orasRemote *OrasRemote +} + +// NewTransport returns a custom transport that tracks an http.RoundTripper and an OrasRemote reference. +func NewTransport(base http.RoundTripper, o *OrasRemote) *Transport { + return &Transport{base, o} +} + +type readCloser struct { + io.Reader + io.Closer +} + +// RoundTrip calls base roundtrip while keeping track of the current request. +// This is currently only used to track the progress of publishes, not pulls. +func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error) { + if req.Method != http.MethodHead && req.Body != nil && t.orasRemote.ProgressBar != nil { + tee := io.TeeReader(req.Body, t.orasRemote.ProgressBar) + teeCloser := readCloser{tee, req.Body} + req.Body = teeCloser + } + + resp, err = t.RoundTripper.RoundTrip(req) + + if req.Method == http.MethodHead && resp != nil && req.Body == nil && t.orasRemote.ProgressBar != nil && resp.ContentLength > 0 { + t.orasRemote.ProgressBar.Add(int(resp.ContentLength)) + } + + return resp, err +} + +// NewOrasRemote returns an oras remote repository client and context for the given reference. +func NewOrasRemote(ref registry.Reference) (*OrasRemote, error) { + o := &OrasRemote{} + o.Context = withScopes(ref) + // patch docker.io to registry-1.docker.io + // this allows end users to use docker.io as an alias for registry-1.docker.io + if ref.Registry == "docker.io" { + ref.Registry = "registry-1.docker.io" + } + repo, err := remote.NewRepository(ref.String()) + if err != nil { + return &OrasRemote{}, err + } + repo.PlainHTTP = zarfconfig.CommonOptions.Insecure + authClient, err := o.withAuthClient(ref) + if err != nil { + return &OrasRemote{}, err + } + repo.Client = authClient + o.Repository = repo + return o, nil +} + +// PrintLayerExists prints a success message to the console when a layer has been successfully published to a registry. +func PrintLayerExists(_ context.Context, desc ocispec.Descriptor) error { + title := desc.Annotations[ocispec.AnnotationTitle] + var format string + if title != "" { + format = fmt.Sprintf("%s %s", desc.Digest.Encoded()[:12], First30last30(title)) + } else { + format = fmt.Sprintf("%s [%s]", desc.Digest.Encoded()[:12], desc.MediaType) + } + message.Successf(format) + return nil +} diff --git a/src/pkg/utils/random.go b/src/pkg/utils/random.go index 0608a8b649..f41e48c3e9 100644 --- a/src/pkg/utils/random.go +++ b/src/pkg/utils/random.go @@ -28,3 +28,12 @@ func RandomString(length int) string { return string(bytes) } + +// First30last30 returns the source string that has been trimmed to 30 characters at the beginning and end. +func First30last30(s string) string { + if len(s) > 60 { + return s[0:27] + "..." + s[len(s)-26:] + } + + return s +} diff --git a/src/test/e2e/04_create_templating_test.go b/src/test/e2e/04_create_templating_test.go index 41e5b7ea19..e49907e2f3 100644 --- a/src/test/e2e/04_create_templating_test.go +++ b/src/test/e2e/04_create_templating_test.go @@ -36,7 +36,7 @@ func TestCreateTemplating(t *testing.T) { stdOut, stdErr, err := e2e.execZarfCommand("package", "create", "examples/package-variables", "--set", "CONFIG_MAP=simple-configmap.yaml", "--set", "ACTION=template", "--confirm", "--zarf-cache", cachePath) require.NoError(t, err, stdOut, stdErr) - stdOut, stdErr, err = e2e.execZarfCommand("t", "archiver", "decompress", pkgName, decompressPath) + stdOut, stdErr, err = e2e.execZarfCommand("t", "archiver", "decompress", pkgName, decompressPath, "--decompress-all", "-l=trace") require.NoError(t, err, stdOut, stdErr) // Check that the configmap exists and is readable diff --git a/src/test/e2e/07_create_git_test.go b/src/test/e2e/07_create_git_test.go index d6de7ef720..5ea19c4a16 100644 --- a/src/test/e2e/07_create_git_test.go +++ b/src/test/e2e/07_create_git_test.go @@ -22,7 +22,7 @@ func TestCreateGit(t *testing.T) { // Extract the test package. path := fmt.Sprintf("build/zarf-package-git-data-%s-v1.0.0.tar.zst", e2e.arch) - stdOut, stdErr, err := e2e.execZarfCommand("tools", "archiver", "decompress", path, extractDir) + stdOut, stdErr, err := e2e.execZarfCommand("tools", "archiver", "decompress", path, extractDir, "--decompress-all") require.NoError(t, err, stdOut, stdErr) defer e2e.cleanFiles(extractDir) diff --git a/src/test/e2e/25_helm_test.go b/src/test/e2e/25_helm_test.go index 938d8552a8..904bb3b62a 100644 --- a/src/test/e2e/25_helm_test.go +++ b/src/test/e2e/25_helm_test.go @@ -91,7 +91,7 @@ func testHelmEscaping(t *testing.T) { func testHelmOCIChart(t *testing.T) { t.Log("E2E: Helm OCI chart") - path := fmt.Sprintf("build/zarf-package-helm-oci-chart-%s.tar.zst", e2e.arch) + path := fmt.Sprintf("build/zarf-package-helm-oci-chart-%s-0.0.1.tar.zst", e2e.arch) // Deploy the package. stdOut, stdErr, err := e2e.execZarfCommand("package", "deploy", path, "--confirm") diff --git a/src/test/e2e/50_oci_package_test.go b/src/test/e2e/50_oci_package_test.go new file mode 100644 index 0000000000..432f23daea --- /dev/null +++ b/src/test/e2e/50_oci_package_test.go @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: 2021-Present The Zarf Authors + +// Package test provides e2e tests for Zarf. +package test + +import ( + "fmt" + "path/filepath" + "testing" + "time" + + "github.com/defenseunicorns/zarf/src/internal/cluster" + "github.com/defenseunicorns/zarf/src/pkg/utils" + "github.com/defenseunicorns/zarf/src/types" + "github.com/stretchr/testify/require" + "github.com/stretchr/testify/suite" + "oras.land/oras-go/v2/registry" +) + +type RegistryClientTestSuite struct { + suite.Suite + Remote *utils.OrasRemote + Reference registry.Reference + PackagesDir string + ZarfState types.ZarfState + RegistryTunnel *cluster.Tunnel +} + +var badRef = registry.Reference{ + Registry: "", + Repository: "zarf-test", + Reference: "bad-tag", +} + +func (suite *RegistryClientTestSuite) SetupSuite() { + t := suite.T() + e2e.setupWithCluster(t) + defer e2e.teardown(t) + + // Get reference to the current cluster + c, err := cluster.NewClusterWithWait(1 * time.Minute) + require.NoError(t, err, "unable to connect to the cluster") + + // Get the Zarf state from the cluster + state, err := c.LoadZarfState() + require.NoError(t, err, "unable to load the current Zarf state") + suite.ZarfState = state + + // Create a tunnel to the registry running in the cluster + suite.RegistryTunnel, err = cluster.NewZarfTunnel() + require.NoError(t, err, "unable to create a tunnel to the registry") + err = suite.RegistryTunnel.Connect("registry", false) + require.NoError(t, err, "unable to connect to the registry") + suite.Reference.Registry = suite.RegistryTunnel.Endpoint() + badRef.Registry = suite.RegistryTunnel.Endpoint() + + suite.PackagesDir = "build" + + _, stdErr, err := e2e.execZarfCommand("tools", "registry", "login", "--username", suite.ZarfState.RegistryInfo.PushUsername, "-p", suite.ZarfState.RegistryInfo.PushPassword, suite.Reference.Registry) + require.NoError(t, err) + require.Contains(t, stdErr, "logged in", "failed to login to the registry") +} + +func (suite *RegistryClientTestSuite) TearDownSuite() { + t := suite.T() + defer e2e.teardown(t) + + suite.RegistryTunnel.Close() + + stdOut, stdErr, err := e2e.execZarfCommand("package", "remove", "helm-oci-chart", "--confirm") + require.NoError(t, err, stdOut, stdErr) +} + +func (suite *RegistryClientTestSuite) Test_0_Publish() { + t := suite.T() + t.Log("E2E: Package Publish oci://") + + // Publish package. + example := filepath.Join(suite.PackagesDir, fmt.Sprintf("zarf-package-helm-oci-chart-%s-0.0.1.tar.zst", e2e.arch)) + ref := suite.Reference.String() + stdOut, stdErr, err := e2e.execZarfCommand("package", "publish", example, "oci://"+ref, "--insecure") + require.NoError(t, err, stdOut, stdErr) + require.Contains(t, stdErr, "Published "+ref) + + // Publish w/ package missing `metadata.version` field. + example = filepath.Join(suite.PackagesDir, fmt.Sprintf("zarf-package-dos-games-%s.tar.zst", e2e.arch)) + _, stdErr, err = e2e.execZarfCommand("package", "publish", example, "oci://"+ref, "--insecure") + require.Error(t, err, stdErr) +} + +func (suite *RegistryClientTestSuite) Test_1_Pull() { + t := suite.T() + t.Log("E2E: Package Pull oci://") + + out := fmt.Sprintf("zarf-package-helm-oci-chart-%s-0.0.1.tar.zst", e2e.arch) + + // Build the fully qualified reference. + suite.Reference.Repository = "helm-oci-chart" + suite.Reference.Reference = fmt.Sprintf("0.0.1-%s", e2e.arch) + ref := suite.Reference.String() + + // Pull the package via OCI. + stdOut, stdErr, err := e2e.execZarfCommand("package", "pull", "oci://"+ref, "--insecure") + require.NoError(t, err, stdOut, stdErr) + require.Contains(t, stdErr, "Pulled "+ref) + defer e2e.cleanFiles(out) + + // Verify the package was pulled. + require.FileExists(t, out) + + // Test pull w/ bad ref. + stdOut, stdErr, err = e2e.execZarfCommand("package", "pull", "oci://"+badRef.String(), "--insecure") + require.Error(t, err, stdOut, stdErr) +} + +func (suite *RegistryClientTestSuite) Test_2_Deploy() { + t := suite.T() + t.Log("E2E: Package Deploy oci://") + + // Build the fully qualified reference. + suite.Reference.Repository = "helm-oci-chart" + suite.Reference.Reference = fmt.Sprintf("0.0.1-%s", e2e.arch) + ref := suite.Reference.String() + + // Deploy the package via OCI. + stdOut, stdErr, err := e2e.execZarfCommand("package", "deploy", "oci://"+ref, "--insecure", "--confirm") + require.NoError(t, err, stdOut, stdErr) + require.Contains(t, stdErr, "Pulled "+ref) + + stdOut, stdErr, err = e2e.execZarfCommand("tools", "kubectl", "get", "pods", "-n=helm-oci-demo", "--no-headers") + require.NoError(t, err, stdErr) + require.Contains(t, string(stdOut), "podinfo-") + + // Test deploy w/ bad ref. + _, stdErr, err = e2e.execZarfCommand("package", "deploy", "oci://"+badRef.String(), "--insecure", "--confirm") + require.Error(t, err, stdErr) +} + +func (suite *RegistryClientTestSuite) Test_3_Inspect() { + t := suite.T() + t.Log("E2E: Package Inspect oci://") + + suite.Reference.Repository = "helm-oci-chart" + suite.Reference.Reference = fmt.Sprintf("0.0.1-%s", e2e.arch) + ref := suite.Reference.String() + stdOut, stdErr, err := e2e.execZarfCommand("package", "inspect", "oci://"+ref, "--insecure") + require.NoError(t, err, stdOut, stdErr) + require.Contains(t, stdErr, "Loaded Zarf Package oci://"+ref) + + // Test inspect w/ bad ref. + _, stdErr, err = e2e.execZarfCommand("package", "inspect", "oci://"+badRef.String(), "--insecure") + require.Error(t, err, stdErr) +} + +func TestRegistryClientTestSuite(t *testing.T) { + suite.Run(t, new(RegistryClientTestSuite)) +} diff --git a/src/types/api.go b/src/types/api.go index 80be071dcb..2932e2d294 100644 --- a/src/types/api.go +++ b/src/types/api.go @@ -6,16 +6,16 @@ package types // RestAPI is the struct that is used to marshal/unmarshal the top-level API objects. type RestAPI struct { - ZarfPackage ZarfPackage `json:"zarfPackage"` - ZarfState ZarfState `json:"zarfState"` - ZarfCommonOptions ZarfCommonOptions `json:"zarfCommonOptions"` - ZarfCreateOptions ZarfCreateOptions `json:"zarfCreateOptions"` - ZarfDeployOptions ZarfDeployOptions `json:"zarfDeployOptions"` - ZarfInitOptions ZarfInitOptions `json:"zarfInitOptions"` - ConnectStrings ConnectStrings `json:"connectStrings"` - ClusterSummary ClusterSummary `json:"clusterSummary"` - DeployedPackage DeployedPackage `json:"deployedPackage"` - APIZarfPackage APIZarfPackage `json:"apiZarfPackage"` + ZarfPackage ZarfPackage `json:"zarfPackage"` + ZarfState ZarfState `json:"zarfState"` + ZarfCommonOptions ZarfCommonOptions `json:"zarfCommonOptions"` + ZarfCreateOptions ZarfCreateOptions `json:"zarfCreateOptions"` + ZarfDeployOptions ZarfDeployOptions `json:"zarfDeployOptions"` + ZarfInitOptions ZarfInitOptions `json:"zarfInitOptions"` + ConnectStrings ConnectStrings `json:"connectStrings"` + ClusterSummary ClusterSummary `json:"clusterSummary"` + DeployedPackage DeployedPackage `json:"deployedPackage"` + APIZarfPackage APIZarfPackage `json:"apiZarfPackage"` APIZarfDeployPayload APIZarfDeployPayload `json:"apiZarfDeployPayload"` } @@ -37,4 +37,4 @@ type APIZarfPackage struct { type APIZarfDeployPayload struct { DeployOpts ZarfDeployOptions `json:"deployOpts"` InitOpts *ZarfInitOptions `json:"initOpts,omitempty"` -} \ No newline at end of file +} diff --git a/src/types/packager.go b/src/types/packager.go index 5c0c4f18f3..15ace0bccb 100644 --- a/src/types/packager.go +++ b/src/types/packager.go @@ -15,6 +15,9 @@ type PackagerConfig struct { // InitOpts tracks user-defined values for the active Zarf initialization. InitOpts ZarfInitOptions + // PublishOpts tracks user-defined options used to publish the package + PublishOpts ZarfPublishOptions + // Track if CLI prompts should be generated IsInteractive bool diff --git a/src/types/runtime.go b/src/types/runtime.go index 6e6ef24f8f..0671e8cd0f 100644 --- a/src/types/runtime.go +++ b/src/types/runtime.go @@ -4,6 +4,11 @@ // Package types contains all the types used by Zarf. package types +import ( + "oras.land/oras-go/v2" + "oras.land/oras-go/v2/registry" +) + // ZarfCommonOptions tracks the user-defined preferences used across commands. type ZarfCommonOptions struct { Confirm bool `json:"confirm" jsonschema:"description=Verify that Zarf should perform an action"` @@ -21,6 +26,14 @@ type ZarfDeployOptions struct { SetVariables map[string]string `json:"setVariables" jsonschema:"description=Key-Value map of variable names and their corresponding values that will be used to template against the Zarf package being used"` } +// ZarfPublishOptions tracks the user-defined preferences during a package publish. +type ZarfPublishOptions struct { + Reference registry.Reference `jsonschema:"description=Remote registry reference"` + CopyOptions oras.CopyOptions `jsonschema:"description=Options for the copy operation"` + PackOptions oras.PackOptions `jsonschema:"description=Options for the pack operation"` + PackagePath string `json:"packagePath" jsonschema:"description=Location where a Zarf package to publish can be found"` +} + // ZarfInitOptions tracks the user-defined options during cluster initialization. type ZarfInitOptions struct { // Zarf init is installing the k3s component @@ -85,6 +98,7 @@ type TempPaths struct { SeedImage string Images string Components string + SbomTar string Sboms string ZarfYaml string }