Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added remote sync updates #399

Closed
wants to merge 8 commits into from
Closed

added remote sync updates #399

wants to merge 8 commits into from

Conversation

zackbradys
Copy link
Member

@zackbradys zackbradys commented Feb 3, 2025 •
edited
Loading

Please check below, if the PR fulfills these requirements:

  • Commit(s) and code follow the repositories guidelines.
  • Test(s) have been added or updated to support these change(s).
  • Doc(s) have been added or updated to support these change(s).

Associated Links:

Types of Changes:

  • Feature

Proposed Changes:

  • [Feature] Added support for hauler store sync to accept multiple local and remote archives of http/https
  • [Feature] Improved logging for related commands and outputs of hauler store sync
  • [Feature] defaulted local file of `hauler-manifest.yaml (new behavior) to matches other commands...

Verification/Testing of Changes:

zackbradys@Zacks-MacBook-Pro test % ls -la
total 40
drwxr-xr-x    5 zackbradys  staff    160 Feb  3 02:41 .
drwxr-xr-x  213 zackbradys  staff   6816 Feb  2 10:17 ..
-rwxr-xr-x    1 zackbradys  staff   1266 Feb  3 02:38 hauler-manifest.yaml
-rw-r--r--    1 zackbradys  staff  14970 Feb  3 02:33 rancher-cluster-templates-0.5.2.tgz
drwxr-xr-x    8 zackbradys  staff    256 Feb  3 00:29 tmp
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync         
2025-02-03 02:41:24 INF processing manifest [hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:41:24 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:41:24 INF adding 'image' [busybox] to the store
2025-02-03 02:41:30 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:41:30 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:41:31 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:41:31 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:41:33 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:41:33 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:41:33 INF adding 'chart' [rancher] to the store
2025-02-03 02:41:35 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:41:35 INF adding 'chart' [rancher] to the store
2025-02-03 02:41:36 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:41:36 INF adding 'chart' [rancher] to the store
2025-02-03 02:41:38 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:41:38 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:41:38 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:41:38 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:41:39 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:41:39 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:41:39 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:41:39 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:41:39 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:41:39 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:41:39 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:41:39 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:41:39 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:41:39 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -l debug
2025-02-03 02:41:55 DBG running cli command [hauler store sync]
2025-02-03 02:41:55 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:41:55 DBG using temporary directory at [/var/folders/5f/lyy8pc6s62dbckp8sy31ft5r0000gn/T/hauler1037744451]
2025-02-03 02:41:55 INF processing manifest [hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:41:55 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:41:55 INF adding 'image' [busybox] to the store
2025-02-03 02:41:56 DBG multi-arch image [true]
2025-02-03 02:42:00 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:42:00 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:42:00 DBG multi-arch image [true]
2025-02-03 02:42:00 DBG platform for image [linux/amd64]
2025-02-03 02:42:00 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:42:00 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:42:01 DBG multi-arch image [false]
2025-02-03 02:42:02 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:42:02 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:42:02 INF adding 'chart' [rancher] to the store
2025-02-03 02:42:04 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:42:04 INF adding 'chart' [rancher] to the store
2025-02-03 02:42:05 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:42:05 INF adding 'chart' [rancher] to the store
2025-02-03 02:42:07 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:42:07 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:42:08 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:42:08 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:42:08 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:42:08 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:42:08 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:42:08 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:42:08 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:42:08 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:42:08 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:42:09 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:42:09 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:42:09 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml         
2025-02-03 02:43:17 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:43:17 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:43:17 INF adding 'image' [busybox] to the store
2025-02-03 02:43:23 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:43:23 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:43:24 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:43:24 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:43:26 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:43:26 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:43:26 INF adding 'chart' [rancher] to the store
2025-02-03 02:43:27 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:43:27 INF adding 'chart' [rancher] to the store
2025-02-03 02:43:29 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:43:29 INF adding 'chart' [rancher] to the store
2025-02-03 02:43:30 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:43:30 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:43:31 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:43:31 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:43:31 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:43:31 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:43:31 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:43:31 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:43:31 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:43:31 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:43:32 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:43:32 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:43:32 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:43:32 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store                                                                                  
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -l debug -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml
2025-02-03 02:44:13 DBG running cli command [hauler store sync]
2025-02-03 02:44:13 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:44:13 DBG using temporary directory at [/var/folders/5f/lyy8pc6s62dbckp8sy31ft5r0000gn/T/hauler3233246142]
2025-02-03 02:44:13 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:44:13 DBG detected remote manifest... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml]
2025-02-03 02:44:13 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:44:13 INF adding 'image' [busybox] to the store
2025-02-03 02:44:14 DBG multi-arch image [true]
2025-02-03 02:44:18 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:44:18 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:44:18 DBG multi-arch image [true]
2025-02-03 02:44:18 DBG platform for image [linux/amd64]
2025-02-03 02:44:19 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:44:19 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:44:19 DBG multi-arch image [false]
2025-02-03 02:44:21 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:44:21 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:44:21 INF adding 'chart' [rancher] to the store
2025-02-03 02:44:23 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:44:23 INF adding 'chart' [rancher] to the store
2025-02-03 02:44:24 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:44:24 INF adding 'chart' [rancher] to the store
2025-02-03 02:44:26 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:44:26 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:44:27 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:44:27 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:44:27 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:44:27 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:44:27 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:44:27 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:44:27 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:44:27 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:44:27 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:44:28 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:44:28 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:44:28 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store                
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info

+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % export HAULER_TEMP_DIR=/Users/zackbradys/Downloads/RGS/test/tmp
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -l debug -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml
2025-02-03 02:45:38 DBG running cli command [hauler store sync]
2025-02-03 02:45:38 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:45:38 DBG using temporary directory at [/Users/zackbradys/Downloads/RGS/test/tmp/hauler61401760]
2025-02-03 02:45:38 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:45:38 DBG detected remote manifest... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml]
2025-02-03 02:45:39 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:45:39 INF adding 'image' [busybox] to the store
2025-02-03 02:45:39 DBG multi-arch image [true]
2025-02-03 02:45:43 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:45:43 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:45:44 DBG multi-arch image [true]
2025-02-03 02:45:44 DBG platform for image [linux/amd64]
2025-02-03 02:45:44 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:45:44 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:45:45 DBG multi-arch image [false]
2025-02-03 02:45:46 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:45:46 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:45:46 INF adding 'chart' [rancher] to the store
2025-02-03 02:45:48 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:45:48 INF adding 'chart' [rancher] to the store
2025-02-03 02:45:50 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:45:50 INF adding 'chart' [rancher] to the store
2025-02-03 02:45:51 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:45:51 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:45:52 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:45:52 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:45:52 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:45:52 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:45:52 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:45:52 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:45:52 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:45:52 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:45:53 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:45:53 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:45:53 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:45:53 INF successfully added 'file' [hauler/rke2-install.sh:latest]
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % unset HAULER_TEMP_DIR
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store sync -l debug -t /Users/zackbradys/Downloads/RGS/test/tmp -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml 
2025-02-03 02:46:34 DBG running cli command [hauler store sync]
2025-02-03 02:46:34 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:46:34 DBG using temporary directory at [/Users/zackbradys/Downloads/RGS/test/tmp/hauler1717808495]
2025-02-03 02:46:34 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 02:46:34 DBG detected remote manifest... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml]
2025-02-03 02:46:35 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 02:46:35 INF adding 'image' [busybox] to the store
2025-02-03 02:46:35 DBG multi-arch image [true]
2025-02-03 02:46:39 INF successfully added 'image' [index.docker.io/library/busybox:latest]
2025-02-03 02:46:39 INF adding 'image' [busybox:stable] to the store
2025-02-03 02:46:39 DBG multi-arch image [true]
2025-02-03 02:46:39 DBG platform for image [linux/amd64]
2025-02-03 02:46:40 INF successfully added 'image' [index.docker.io/library/busybox:stable]
2025-02-03 02:46:40 INF adding 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 02:46:40 DBG multi-arch image [false]
2025-02-03 02:46:42 INF successfully added 'image' [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 02:46:42 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 02:46:42 INF adding 'chart' [rancher] to the store
2025-02-03 02:46:43 INF successfully added 'chart' [hauler/rancher:2.10.2]
2025-02-03 02:46:43 INF adding 'chart' [rancher] to the store
2025-02-03 02:46:45 INF successfully added 'chart' [hauler/rancher:2.8.4]
2025-02-03 02:46:45 INF adding 'chart' [rancher] to the store
2025-02-03 02:46:46 INF successfully added 'chart' [hauler/rancher:2.8.3]
2025-02-03 02:46:46 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 02:46:47 INF successfully added 'chart' [hauler/hauler-helm:1.1.1]
2025-02-03 02:46:47 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 02:46:47 INF successfully added 'chart' [hauler/hauler-helm:1.0.6]
2025-02-03 02:46:47 INF adding 'chart' [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 02:46:48 INF successfully added 'chart' [hauler/hauler-helm:1.0.4]
2025-02-03 02:46:48 INF adding 'chart' [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 02:46:48 INF successfully added 'chart' [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 02:46:48 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 02:46:48 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 02:46:48 INF successfully added 'file' [hauler/install.sh:latest]
2025-02-03 02:46:48 INF adding 'file' [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 02:46:48 INF successfully added 'file' [hauler/rke2-install.sh:latest]

Additional Context:

  • N/A

@zackbradys zackbradys requested a review from a team February 3, 2025 07:53
@zackbradys zackbradys self-assigned this Feb 3, 2025
@zackbradys zackbradys added enhancement New feature or request size/M Denotes an issue/PR requiring a relatively moderate amount of work labels Feb 3, 2025
@zackbradys zackbradys added this to the Hauler v1.2.0 milestone Feb 3, 2025
@zackbradys zackbradys linked an issue Feb 3, 2025 that may be closed by this pull request
[feature] Allow hauler to load and sync remote files #208
Closed
amartin120
amartin120 reviewed Feb 3, 2025
View reviewed changes
internal/flags/store.go Show resolved Hide resolved
internal/flags/sync.go Outdated Show resolved Hide resolved
cmd/hauler/cli/store/sync.go Show resolved Hide resolved
@zackbradys
Copy link
Member Author 

zackbradys@Zacks-MacBook-Pro hauler % hauler store sync -f hauler-manifest-pipeline.yaml -f https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml
2025-02-03 20:32:29 INF processing manifest [hauler-manifest-pipeline.yaml] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/store]
2025-02-03 20:32:29 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 20:32:29 INF adding image [busybox] to the store
2025-02-03 20:32:35 INF successfully added image [index.docker.io/library/busybox:latest]
2025-02-03 20:32:35 INF adding image [busybox:stable] to the store
2025-02-03 20:32:36 INF successfully added image [index.docker.io/library/busybox:stable]
2025-02-03 20:32:36 INF adding image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 20:32:38 INF successfully added image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 20:32:38 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 20:32:38 INF adding chart [rancher] to the store
2025-02-03 20:32:40 INF successfully added chart [hauler/rancher:2.10.2]
2025-02-03 20:32:40 INF adding chart [rancher] to the store
2025-02-03 20:32:41 INF successfully added chart [hauler/rancher:2.8.4]
2025-02-03 20:32:41 INF adding chart [rancher] to the store
2025-02-03 20:32:42 INF successfully added chart [hauler/rancher:2.8.3]
2025-02-03 20:32:42 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 20:32:43 INF successfully added chart [hauler/hauler-helm:1.1.1]
2025-02-03 20:32:43 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 20:32:43 INF successfully added chart [hauler/hauler-helm:1.0.6]
2025-02-03 20:32:43 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 20:32:44 INF successfully added chart [hauler/hauler-helm:1.0.4]
2025-02-03 20:32:44 INF adding chart [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 20:32:44 INF successfully added chart [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 20:32:44 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 20:32:44 INF adding file [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 20:32:44 INF successfully added file [hauler/install.sh:latest]
2025-02-03 20:32:44 INF adding file [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 20:32:44 INF successfully added file [hauler/rke2-install.sh:latest]
2025-02-03 20:32:44 INF adding file [testdata/hauler-manifest.yaml] to the store as [hauler/hauler-manifest.yaml:latest]
2025-02-03 20:32:44 INF successfully added file [hauler/hauler-manifest.yaml:latest]
2025-02-03 20:32:44 INF adding file [testdata/hauler-manifest.yaml] to the store as [hauler/hauler-manifest-local.yaml:latest]
2025-02-03 20:32:44 INF successfully added file [hauler/hauler-manifest-local.yaml:latest]
2025-02-03 20:32:44 INF processing completed successfully
2025-02-03 20:32:44 INF processing manifest [https://carbide.s3.us-gov-east-1.amazonaws.com/tests/hauler-manifest.yaml] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/store]
2025-02-03 20:32:45 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Images]
2025-02-03 20:32:45 INF adding image [busybox] to the store
2025-02-03 20:32:50 INF successfully added image [index.docker.io/library/busybox:latest]
2025-02-03 20:32:50 INF adding image [busybox:stable] to the store
2025-02-03 20:32:51 INF successfully added image [index.docker.io/library/busybox:stable]
2025-02-03 20:32:51 INF adding image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] to the store
2025-02-03 20:32:53 INF successfully added image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]
2025-02-03 20:32:53 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Charts]
2025-02-03 20:32:53 INF adding chart [rancher] to the store
2025-02-03 20:32:54 INF successfully added chart [hauler/rancher:2.10.2]
2025-02-03 20:32:54 INF adding chart [rancher] to the store
2025-02-03 20:32:56 INF successfully added chart [hauler/rancher:2.8.4]
2025-02-03 20:32:56 INF adding chart [rancher] to the store
2025-02-03 20:32:57 INF successfully added chart [hauler/rancher:2.8.3]
2025-02-03 20:32:57 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.1.1
Digest: sha256:3957fc2bf560f2ccd64e38b59651b097dddb38863124745df264053cae1b807a
2025-02-03 20:32:58 INF successfully added chart [hauler/hauler-helm:1.1.1]
2025-02-03 20:32:58 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.6
Digest: sha256:c4befb2f4939f4b077e3fa3986ab11090a603ac8dc7f8f4ac61dc7a896d20455
2025-02-03 20:32:58 INF successfully added chart [hauler/hauler-helm:1.0.6]
2025-02-03 20:32:58 INF adding chart [hauler-helm] to the store
Pulled: ghcr.io/hauler-dev/hauler-helm:1.0.4
Digest: sha256:6a0a7317980309f4551404bd11ee43d1a9d1f1fe97294c7ab051625e435c3d81
2025-02-03 20:32:58 INF successfully added chart [hauler/hauler-helm:1.0.4]
2025-02-03 20:32:58 INF adding chart [rancher-cluster-templates-0.5.2.tgz] to the store
2025-02-03 20:32:58 INF successfully added chart [hauler/rancher-cluster-templates:0.5.2]
2025-02-03 20:32:58 INF syncing [content.hauler.cattle.io/v1alpha1, Kind=Files]
2025-02-03 20:32:58 INF adding file [https://get.rke2.io/install.sh] to the store as [hauler/install.sh:latest]
2025-02-03 20:32:58 INF successfully added file [hauler/install.sh:latest]
2025-02-03 20:32:58 INF adding file [https://get.rke2.io/install.sh] to the store as [hauler/rke2-install.sh:latest]
2025-02-03 20:32:59 INF successfully added file [hauler/rke2-install.sh:latest]
2025-02-03 20:32:59 INF processing completed successfully

@zackbradys zackbradys requested review from amartin120 and a team February 4, 2025 04:26
@amartin120
Copy link
Contributor

Thanks for the updates and the back and forth. Please update the PR description to reflect the latest details and this should be good to go.

@zackbradys
Copy link
Member Author

updated description as requested! @amartin120

@zackbradys zackbradys closed this Feb 4, 2025
@zackbradys zackbradys deleted the remote-sync-updates branch February 5, 2025 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amartin120 amartin120 Awaiting requested review from amartin120

Assignees

@zackbradys zackbradys

Labels
enhancement New feature or request size/M Denotes an issue/PR requiring a relatively moderate amount of work
Projects
Hauler
Status: Resolved
Milestone
Hauler v1.2.0
Development

Successfully merging this pull request may close these issues.

[feature] Allow hauler to load and sync remote files
2 participants
@zackbradys @amartin120