Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added remote load updates and cleaned up load and save #397

Closed
wants to merge 12 commits into from
Closed

added remote load updates and cleaned up load and save #397

wants to merge 12 commits into from

Conversation

zackbradys
Copy link
Member

@zackbradys zackbradys commented Feb 3, 2025 •
edited
Loading

Please check below, if the PR fulfills these requirements:

  • Commit(s) and code follow the repositories guidelines.
  • Test(s) have been added or updated to support these change(s).
  • Doc(s) have been added or updated to support these change(s).

Associated Links:

Types of Changes:

  • Feature
  • Breaking Change

Proposed Changes:

  • [Feature] Added support for hauler store load to accept multiple local and remote archives of http/https
  • [Feature] Improved logging for related commands and outputs of hauler store load/hauler store save
  • [Feature] Defaulted local file of haul.tar.zst to match the functionality between both commands
  • [Breaking Change] Removed the flag of --filename/-f for hauler store save (argument)

Verification/Testing of Changes:

zackbradys@Zacks-MacBook-Pro test % ls -la
total 1124216
drwxr-xr-x    5 zackbradys  staff        160 Feb  3 00:09 .
drwxr-xr-x  213 zackbradys  staff       6816 Feb  2 10:17 ..
-rw-r--r--    1 zackbradys  staff  575594527 Feb  3 00:01 haul.tar.zst
drwxr-xr-x    4 zackbradys  staff        128 Feb  3 00:09 store
drwxr-xr-x    7 zackbradys  staff        224 Feb  3 00:09 tmp
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store load -h
Load a content store from a store archive

Usage:
  hauler store load [flags]

Flags:
  -h, --help              help for load
  -t, --tempdir string    (Optional) Override the default temporary directiory determined by the OS

Global Flags:
  -d, --haulerdir string   Set the location of the hauler directory (default $HOME/.hauler)
      --ignore-errors      Ignore/Bypass errors (i.e. warn on error) (defaults false)
  -l, --log-level string   Set the logging level (i.e. info, debug, warn) (default "info")
  -r, --retries int        Set the number of retries for operations (default 3)
  -s, --store string       Set the directory to use for the content store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store load       
2025-02-03 00:12:32 INF loading archive [haul.tar.zst] to store [store]
2025-02-03 00:12:33 INF unarchiving completed successfully
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+---------------------------------------+-------+-----------------+----------+----------+
| REFERENCE                             | TYPE  | PLATFORM        | # LAYERS | SIZE     |
+---------------------------------------+-------+-----------------+----------+----------+
| index.docker.io/library/alpine:latest | image | linux/386       |        1 | 3.5 MB   |
|                                       | image | linux/amd64     |        1 | 3.6 MB   |
|                                       | image | linux/arm       |        1 | 3.1 MB   |
|                                       | image | linux/arm       |        1 | 3.4 MB   |
|                                       | image | linux/arm64     |        1 | 4.0 MB   |
|                                       | image | linux/ppc64le   |        1 | 3.6 MB   |
|                                       | image | linux/riscv64   |        1 | 3.4 MB   |
|                                       | image | linux/s390x     |        1 | 3.5 MB   |
|                                       | image | unknown/unknown |        2 | 84.4 kB  |
|                                       | image | unknown/unknown |        2 | 84.3 kB  |
|                                       | image | unknown/unknown |        2 | 84.2 kB  |
|                                       | image | unknown/unknown |        2 | 84.3 kB  |
|                                       | image | unknown/unknown |        2 | 82.4 kB  |
|                                       | image | unknown/unknown |        1 | 5.5 kB   |
|                                       | image | unknown/unknown |        2 | 82.4 kB  |
|                                       | image | unknown/unknown |        2 | 82.3 kB  |
| index.docker.io/library/nginx:latest  | image | linux/386       |        7 | 70.4 MB  |
|                                       | image | linux/amd64     |        7 | 72.1 MB  |
|                                       | image | linux/arm       |        7 | 62.4 MB  |
|                                       | image | linux/arm       |        7 | 60.7 MB  |
|                                       | image | linux/arm64     |        7 | 68.5 MB  |
|                                       | image | linux/mips64le  |        7 | 68.1 MB  |
|                                       | image | linux/ppc64le   |        7 | 76.7 MB  |
|                                       | image | linux/s390x     |        7 | 66.7 MB  |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        1 | 34.5 kB  |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
+---------------------------------------+-------+-----------------+----------+----------+
|                                                                    TOTAL   | 595.2 MB |
+---------------------------------------+-------+-----------------+----------+----------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info   
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store load https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst
2025-02-03 00:12:49 INF loading archive [https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst] to store [store]
2025-02-03 00:12:51 INF unarchiving completed successfully
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info                                                                 
+---------------------------------------+-------+-----------------+----------+---------+
| REFERENCE                             | TYPE  | PLATFORM        | # LAYERS | SIZE    |
+---------------------------------------+-------+-----------------+----------+---------+
| index.docker.io/library/alpine:latest | image | linux/386       |        1 | 3.5 MB  |
|                                       | image | linux/amd64     |        1 | 3.6 MB  |
|                                       | image | linux/arm       |        1 | 3.1 MB  |
|                                       | image | linux/arm       |        1 | 3.4 MB  |
|                                       | image | linux/arm64     |        1 | 4.0 MB  |
|                                       | image | linux/ppc64le   |        1 | 3.6 MB  |
|                                       | image | linux/riscv64   |        1 | 3.4 MB  |
|                                       | image | linux/s390x     |        1 | 3.5 MB  |
|                                       | image | unknown/unknown |        2 | 84.4 kB |
|                                       | image | unknown/unknown |        2 | 84.3 kB |
|                                       | image | unknown/unknown |        2 | 84.2 kB |
|                                       | image | unknown/unknown |        2 | 84.3 kB |
|                                       | image | unknown/unknown |        2 | 82.4 kB |
|                                       | image | unknown/unknown |        1 | 5.5 kB  |
|                                       | image | unknown/unknown |        2 | 82.4 kB |
|                                       | image | unknown/unknown |        2 | 82.3 kB |
+---------------------------------------+-------+-----------------+----------+---------+
|                                                                    TOTAL   | 28.5 MB |
+---------------------------------------+-------+-----------------+----------+---------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store load -l debug -t /Users/zackbradys/Downloads/RGS/test/tmp https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst
2025-02-03 00:13:58 DBG running cli command [hauler store load]
2025-02-03 00:13:58 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 00:13:58 INF loading archive [https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst] to store [store]
2025-02-03 00:13:58 DBG using temporary directory [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:13:58 DBG detected remote archive... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst]
2025-02-03 00:14:00 DBG unarchiving temporary archive [/Users/zackbradys/Downloads/RGS/test/tmp/stores.tar.zst] to temporary store [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:14:00 DBG creating directory [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:14:00 INF unarchiving completed successfully
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+---------------------------------------+-------+-----------------+----------+---------+
| REFERENCE                             | TYPE  | PLATFORM        | # LAYERS | SIZE    |
+---------------------------------------+-------+-----------------+----------+---------+
| index.docker.io/library/alpine:latest | image | linux/386       |        1 | 3.5 MB  |
|                                       | image | linux/amd64     |        1 | 3.6 MB  |
|                                       | image | linux/arm       |        1 | 3.1 MB  |
|                                       | image | linux/arm       |        1 | 3.4 MB  |
|                                       | image | linux/arm64     |        1 | 4.0 MB  |
|                                       | image | linux/ppc64le   |        1 | 3.6 MB  |
|                                       | image | linux/riscv64   |        1 | 3.4 MB  |
|                                       | image | linux/s390x     |        1 | 3.5 MB  |
|                                       | image | unknown/unknown |        2 | 84.4 kB |
|                                       | image | unknown/unknown |        2 | 84.3 kB |
|                                       | image | unknown/unknown |        2 | 84.2 kB |
|                                       | image | unknown/unknown |        2 | 84.3 kB |
|                                       | image | unknown/unknown |        2 | 82.4 kB |
|                                       | image | unknown/unknown |        1 | 5.5 kB  |
|                                       | image | unknown/unknown |        2 | 82.4 kB |
|                                       | image | unknown/unknown |        2 | 82.3 kB |
+---------------------------------------+-------+-----------------+----------+---------+
|                                                                    TOTAL   | 28.5 MB |
+---------------------------------------+-------+-----------------+----------+---------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info                                                                                                                      
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % export HAULER_TEMP_DIR=/Users/zackbradys/Downloads/RGS/test/tmp
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store load -l debug https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst                                         
2025-02-03 00:14:29 DBG running cli command [hauler store load]
2025-02-03 00:14:29 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 00:14:29 INF loading archive [https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst] to store [store]
2025-02-03 00:14:29 DBG using temporary directory [/Users/zackbradys/Downloads/RGS/test/tmp/hauler3685394917]
2025-02-03 00:14:29 DBG detected remote archive... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst]
2025-02-03 00:14:30 DBG unarchiving temporary archive [/Users/zackbradys/Downloads/RGS/test/tmp/hauler3685394917/stores.tar.zst] to temporary store [/Users/zackbradys/Downloads/RGS/test/tmp/hauler3685394917]
2025-02-03 00:14:30 DBG creating directory [/Users/zackbradys/Downloads/RGS/test/tmp/hauler3685394917]
2025-02-03 00:14:30 INF unarchiving completed successfully
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+---------------------------------------+-------+-----------------+----------+---------+
| REFERENCE                             | TYPE  | PLATFORM        | # LAYERS | SIZE    |
+---------------------------------------+-------+-----------------+----------+---------+
| index.docker.io/library/alpine:latest | image | linux/386       |        1 | 3.5 MB  |
|                                       | image | linux/amd64     |        1 | 3.6 MB  |
|                                       | image | linux/arm       |        1 | 3.1 MB  |
|                                       | image | linux/arm       |        1 | 3.4 MB  |
|                                       | image | linux/arm64     |        1 | 4.0 MB  |
|                                       | image | linux/ppc64le   |        1 | 3.6 MB  |
|                                       | image | linux/riscv64   |        1 | 3.4 MB  |
|                                       | image | linux/s390x     |        1 | 3.5 MB  |
|                                       | image | unknown/unknown |        2 | 84.4 kB |
|                                       | image | unknown/unknown |        2 | 84.3 kB |
|                                       | image | unknown/unknown |        2 | 84.2 kB |
|                                       | image | unknown/unknown |        2 | 84.3 kB |
|                                       | image | unknown/unknown |        2 | 82.4 kB |
|                                       | image | unknown/unknown |        1 | 5.5 kB  |
|                                       | image | unknown/unknown |        2 | 82.4 kB |
|                                       | image | unknown/unknown |        2 | 82.3 kB |
+---------------------------------------+-------+-----------------+----------+---------+
|                                                                    TOTAL   | 28.5 MB |
+---------------------------------------+-------+-----------------+----------+---------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % rm -rf store 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info                                                                                                                      
+-----------+------+----------+----------+------+
| REFERENCE | TYPE | PLATFORM | # LAYERS | SIZE |
+-----------+------+----------+----------+------+
+-----------+------+----------+----------+------+
|                                TOTAL   | 0 B  |
+-----------+------+----------+----------+------+
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store load -l debug -t /Users/zackbradys/Downloads/RGS/test/tmp https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst https://carbide.s3.us-gov-east-1.amazonaws.com/store.tar.zst
2025-02-03 00:39:13 DBG running cli command [hauler store load]
2025-02-03 00:39:13 DBG using store at [/Users/zackbradys/Downloads/RGS/test/store]
2025-02-03 00:39:13 INF loading archive [https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst] to store [store]
2025-02-03 00:39:13 DBG using temporary directory [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:39:13 DBG detected remote archive... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/stores.tar.zst]
2025-02-03 00:39:14 DBG unarchiving temporary archive [/Users/zackbradys/Downloads/RGS/test/tmp/stores.tar.zst] to temporary store [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:39:14 DBG creating directory [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:39:14 INF unarchiving completed successfully
2025-02-03 00:39:14 INF loading archive [https://carbide.s3.us-gov-east-1.amazonaws.com/store.tar.zst] to store [store]
2025-02-03 00:39:14 DBG using temporary directory [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:39:14 DBG detected remote archive... starting download... [https://carbide.s3.us-gov-east-1.amazonaws.com/store.tar.zst]
2025-02-03 00:39:35 DBG unarchiving temporary archive [/Users/zackbradys/Downloads/RGS/test/tmp/store.tar.zst] to temporary store [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:39:35 DBG creating directory [/Users/zackbradys/Downloads/RGS/test/tmp]
2025-02-03 00:39:36 INF unarchiving completed successfully
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % 
zackbradys@Zacks-MacBook-Pro test % hauler store info
+---------------------------------------+-------+-----------------+----------+----------+
| REFERENCE                             | TYPE  | PLATFORM        | # LAYERS | SIZE     |
+---------------------------------------+-------+-----------------+----------+----------+
| index.docker.io/library/alpine:latest | image | linux/386       |        1 | 3.5 MB   |
|                                       | image | linux/amd64     |        1 | 3.6 MB   |
|                                       | image | linux/arm       |        1 | 3.1 MB   |
|                                       | image | linux/arm       |        1 | 3.4 MB   |
|                                       | image | linux/arm64     |        1 | 4.0 MB   |
|                                       | image | linux/ppc64le   |        1 | 3.6 MB   |
|                                       | image | linux/riscv64   |        1 | 3.4 MB   |
|                                       | image | linux/s390x     |        1 | 3.5 MB   |
|                                       | image | unknown/unknown |        2 | 84.4 kB  |
|                                       | image | unknown/unknown |        2 | 84.3 kB  |
|                                       | image | unknown/unknown |        2 | 84.2 kB  |
|                                       | image | unknown/unknown |        2 | 84.3 kB  |
|                                       | image | unknown/unknown |        2 | 82.4 kB  |
|                                       | image | unknown/unknown |        1 | 5.5 kB   |
|                                       | image | unknown/unknown |        2 | 82.4 kB  |
|                                       | image | unknown/unknown |        2 | 82.3 kB  |
| index.docker.io/library/nginx:latest  | image | linux/386       |        7 | 70.4 MB  |
|                                       | image | linux/amd64     |        7 | 72.1 MB  |
|                                       | image | linux/arm       |        7 | 62.4 MB  |
|                                       | image | linux/arm       |        7 | 60.7 MB  |
|                                       | image | linux/arm64     |        7 | 68.5 MB  |
|                                       | image | linux/mips64le  |        7 | 68.1 MB  |
|                                       | image | linux/ppc64le   |        7 | 76.7 MB  |
|                                       | image | linux/s390x     |        7 | 66.7 MB  |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        1 | 34.5 kB  |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
|                                       | image | unknown/unknown |        2 | 3.0 MB   |
+---------------------------------------+-------+-----------------+----------+----------+
|                                                                    TOTAL   | 595.2 MB |
+---------------------------------------+-------+-----------------+----------+----------+

Additional Context:

  • N/A

@zackbradys zackbradys requested a review from a team February 3, 2025 05:41
@zackbradys zackbradys self-assigned this Feb 3, 2025
@zackbradys zackbradys added enhancement New feature or request size/L Denotes an issue/PR requiring a relatively large amount of work labels Feb 3, 2025
@zackbradys zackbradys added this to the Hauler v1.2.0 milestone Feb 3, 2025
@zackbradys zackbradys linked an issue Feb 3, 2025 that may be closed by this pull request
[feature] Allow hauler to load and sync remote files #208
Closed
amartin120
amartin120 reviewed Feb 3, 2025
View reviewed changes
.github/workflows/tests.yaml Outdated Show resolved Hide resolved
cmd/hauler/cli/store/save.go Show resolved Hide resolved
cmd/hauler/cli/store/save.go Outdated Show resolved Hide resolved
@zackbradys zackbradys changed the title added remote load updates added remote load updates and cleaned up load and save Feb 3, 2025
@zackbradys
Copy link
Member Author

zackbradys commented Feb 3, 2025 •
edited
Loading 

zackbradys@Zacks-MacBook-Pro hauler % rm -rf store
zackbradys@Zacks-MacBook-Pro hauler % hauler store load -h                                         
Load a content store from a store archive

Usage:
  hauler store load [flags]

Flags:
  -h, --help               help for load
  -t, --tempdir string     (Optional) Override the default temporary directory determined by the OS

Global Flags:
  -d, --haulerdir string   Set the location of the hauler directory (default $HOME/.hauler)
      --ignore-errors      Ignore/Bypass errors (i.e. warn on error) (defaults false)
  -l, --log-level string   Set the logging level (i.e. info, debug, warn) (default "info")
  -r, --retries int        Set the number of retries for operations (default 3)
  -s, --store string       Set the directory to use for the content store
zackbradys@Zacks-MacBook-Pro hauler % cp testdata/haul.tar.zst . 
zackbradys@Zacks-MacBook-Pro hauler % hauler store load
2025-02-03 10:59:35 INF loading archive [haul.tar.zst] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/store]
2025-02-03 10:59:35 INF unarchiving completed successfully
zackbradys@Zacks-MacBook-Pro hauler %         
zackbradys@Zacks-MacBook-Pro hauler % hauler store info
+------------------------------------------------------------------------------------------------+-------+-----------------+----------+----------+
| REFERENCE                                                                                      | TYPE  | PLATFORM        | # LAYERS | SIZE     |
+------------------------------------------------------------------------------------------------+-------+-----------------+----------+----------+
| gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5 | image | linux/amd64     |        2 | 8.0 MB   |
| hauler/hauler-helm:1.0.8                                                                       | chart | -               |        1 | 3.8 kB   |
| hauler/hauler-manifest.yaml:latest                                                             | file  | -               |        1 | 823 B    |
| hauler/install.sh:latest                                                                       | file  | -               |        1 | 25.0 kB  |
| hauler/rancher:2.8.5                                                                           | chart | -               |        1 | 15.1 kB  |
| index.docker.io/library/busybox:latest                                                         | image | linux/386       |        1 | 2.2 MB   |
|                                                                                                | image | linux/amd64     |        1 | 2.2 MB   |
|                                                                                                | image | linux/arm       |        1 | 1.8 MB   |
|                                                                                                | image | linux/arm       |        1 | 948.4 kB |
|                                                                                                | image | linux/arm       |        1 | 1.6 MB   |
|                                                                                                | image | linux/arm64     |        1 | 1.8 MB   |
|                                                                                                | image | linux/mips64le  |        1 | 2.1 MB   |
|                                                                                                | image | linux/ppc64le   |        1 | 2.5 MB   |
|                                                                                                | image | linux/riscv64   |        1 | 927.7 kB |
|                                                                                                | image | linux/s390x     |        1 | 1.9 MB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
| index.docker.io/library/busybox:stable                                                         | image | linux/amd64     |        1 | 2.2 MB   |
+------------------------------------------------------------------------------------------------+-------+-----------------+----------+----------+
|                                                                                                                             TOTAL   | 28.0 MB  |
+------------------------------------------------------------------------------------------------+-------+-----------------+----------+----------+
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % rm -rf store 
zackbradys@Zacks-MacBook-Pro hauler % hauler store load -l debug
2025-02-03 10:59:56 DBG running cli command [hauler store load]
2025-02-03 10:59:56 DBG using store at [/Users/zackbradys/Documents/Coding/GitHub/hauler/store]
2025-02-03 10:59:56 INF loading archive [haul.tar.zst] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/store]
2025-02-03 10:59:56 DBG using temporary directory [/var/folders/5f/lyy8pc6s62dbckp8sy31ft5r0000gn/T/hauler953561023]
2025-02-03 10:59:56 DBG unarchiving temporary archive [haul.tar.zst] to temporary store [/var/folders/5f/lyy8pc6s62dbckp8sy31ft5r0000gn/T/hauler953561023]
2025-02-03 10:59:56 DBG creating directory [/var/folders/5f/lyy8pc6s62dbckp8sy31ft5r0000gn/T/hauler953561023]
2025-02-03 10:59:56 INF unarchiving completed successfully
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % hauler store info
+------------------------------------------------------------------------------------------------+-------+-----------------+----------+----------+
| REFERENCE                                                                                      | TYPE  | PLATFORM        | # LAYERS | SIZE     |
+------------------------------------------------------------------------------------------------+-------+-----------------+----------+----------+
| gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5 | image | linux/amd64     |        2 | 8.0 MB   |
| hauler/hauler-helm:1.0.8                                                                       | chart | -               |        1 | 3.8 kB   |
| hauler/hauler-manifest.yaml:latest                                                             | file  | -               |        1 | 823 B    |
| hauler/install.sh:latest                                                                       | file  | -               |        1 | 25.0 kB  |
| hauler/rancher:2.8.5                                                                           | chart | -               |        1 | 15.1 kB  |
| index.docker.io/library/busybox:latest                                                         | image | linux/386       |        1 | 2.2 MB   |
|                                                                                                | image | linux/amd64     |        1 | 2.2 MB   |
|                                                                                                | image | linux/arm       |        1 | 1.8 MB   |
|                                                                                                | image | linux/arm       |        1 | 948.4 kB |
|                                                                                                | image | linux/arm       |        1 | 1.6 MB   |
|                                                                                                | image | linux/arm64     |        1 | 1.8 MB   |
|                                                                                                | image | linux/mips64le  |        1 | 2.1 MB   |
|                                                                                                | image | linux/ppc64le   |        1 | 2.5 MB   |
|                                                                                                | image | linux/riscv64   |        1 | 927.7 kB |
|                                                                                                | image | linux/s390x     |        1 | 1.9 MB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
|                                                                                                | image | unknown/unknown |        1 | 1.9 kB   |
| index.docker.io/library/busybox:stable                                                         | image | linux/amd64     |        1 | 2.2 MB   |
+------------------------------------------------------------------------------------------------+-------+-----------------+----------+----------+
|                                                                                                                             TOTAL   | 28.0 MB  |
+------------------------------------------------------------------------------------------------+-------+-----------------+----------+----------+
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % rm -rf store 
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % mkdir tmp                           
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % hauler store load -l debug -s temp -t /Users/zackbradys/Documents/Coding/GitHub/hauler/tmp haul.tar.zst testdata/haul.tar.zst 
2025-02-03 11:01:11 DBG running cli command [hauler store load]
2025-02-03 11:01:11 DBG using store at [/Users/zackbradys/Documents/Coding/GitHub/hauler/temp]
2025-02-03 11:01:11 INF loading archive [haul.tar.zst] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/temp]
2025-02-03 11:01:11 DBG using temporary directory [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp]
2025-02-03 11:01:11 DBG unarchiving temporary archive [haul.tar.zst] to temporary store [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp]
2025-02-03 11:01:11 DBG creating directory [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp]
2025-02-03 11:01:11 INF unarchiving completed successfully
2025-02-03 11:01:11 INF loading archive [testdata/haul.tar.zst] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/temp]
2025-02-03 11:01:11 DBG using temporary directory [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp]
2025-02-03 11:01:11 DBG unarchiving temporary archive [testdata/haul.tar.zst] to temporary store [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp]
2025-02-03 11:01:11 DBG creating directory [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp]
2025-02-03 11:01:11 INF unarchiving completed successfully
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % export HAULER_STORE_DIR=/Users/zackbradys/Documents/Coding/GitHub/hauler/temp2
zackbradys@Zacks-MacBook-Pro hauler % export HAULER_TEMP_DIR=/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp2  
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % hauler store load -l debug haul.tar.zst testdata/haul.tar.zst
2025-02-03 11:02:36 DBG running cli command [hauler store load]
2025-02-03 11:02:36 DBG using store at [/Users/zackbradys/Documents/Coding/GitHub/hauler/temp2]
2025-02-03 11:02:36 INF loading archive [haul.tar.zst] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/temp2]
2025-02-03 11:02:36 DBG using temporary directory [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp2/hauler160357038]
2025-02-03 11:02:36 DBG unarchiving temporary archive [haul.tar.zst] to temporary store [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp2/hauler160357038]
2025-02-03 11:02:36 DBG creating directory [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp2/hauler160357038]
2025-02-03 11:02:36 INF unarchiving completed successfully
2025-02-03 11:02:36 INF loading archive [testdata/haul.tar.zst] to store [/Users/zackbradys/Documents/Coding/GitHub/hauler/temp2]
2025-02-03 11:02:36 DBG using temporary directory [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp2/hauler3113089594]
2025-02-03 11:02:36 DBG unarchiving temporary archive [testdata/haul.tar.zst] to temporary store [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp2/hauler3113089594]
2025-02-03 11:02:36 DBG creating directory [/Users/zackbradys/Documents/Coding/GitHub/hauler/tmp2/hauler3113089594]
2025-02-03 11:02:36 INF unarchiving completed successfully

@zackbradys
Copy link
Member Author 

zackbradys@Zacks-MacBook-Pro hauler % hauler store save
2025-02-03 11:09:54 WRN specify an export platform to prevent potential platform mismatch on import of index [index.docker.io/library/busybox:latest]
2025-02-03 11:09:54 INF saved store [/Users/zackbradys/Documents/Coding/GitHub/hauler/store] -> [/Users/zackbradys/Documents/Coding/GitHub/hauler/haul.tar.zst]
zackbradys@Zacks-MacBook-Pro hauler % 
zackbradys@Zacks-MacBook-Pro hauler % hauler store save -l debug
2025-02-03 11:09:57 DBG running cli command [hauler store save]
2025-02-03 11:09:57 DBG using store at [/Users/zackbradys/Documents/Coding/GitHub/hauler/store]
2025-02-03 11:09:57 DBG descriptor [sha256:d50b7c9f77ecb424481ee733421a2a1d154e49dde6a7f56996cd440958819b21] = [application/vnd.oci.image.manifest.v1+json]
2025-02-03 11:09:57 DBG descriptor [sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140] = [application/vnd.oci.image.index.v1+json]
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140]... type=[application/vnd.oci.image.index.v1+json]... size=[10200]
2025-02-03 11:09:57 WRN specify an export platform to prevent potential platform mismatch on import of index [index.docker.io/library/busybox:latest]
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:9186e638ccc30c5d1a2efd5a2cd632f49bb5013f164f6f85c48ed6fce90fe38f]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140], platform=[unknown/unknown]: matches unknown platform... skipping...
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:13e8bc253c581305aca9ae1fa8562f591769abddd70d877c88bf99fc3679f821]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140], platform=[unknown/unknown]: matches unknown platform... skipping...
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:17934c55824eff3072923db6cd7c3ba00359e57b730e5581e7273c610738f910]: type=application/vnd.oci.image.manifest.v1+json, size=608
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:5b5c31ebfa56e5245ba8666f899fbf64eeb51caa344608d8b938b2ab7d9fe442]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140], platform=[unknown/unknown]: matches unknown platform... skipping...
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:71e065368796c7368a99a072019b9fe73e28e225ae9882430579ec49a1e46235]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140], platform=[unknown/unknown]: matches unknown platform... skipping...
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:88d91ebb7b59921fe3736426a7bb0033a168ba0e2382af5e4b102efabf622240]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140], platform=[unknown/unknown]: matches unknown platform... skipping...
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:5d4fe778cdb7f30d46b87cd1c3cb8ab9b4facea1701838363e27f8f6a61ee611]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:de60096c7ffc12dc44e5af1c0ba38db72e9e8994e65f2fdf034ae5a855b6e1f3]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140], platform=[unknown/unknown]: matches unknown platform... skipping...
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:2d91f6c14d238a12d20fa53250a1c5c9f5322c8457904103fb04184fad393077]: type=application/vnd.oci.image.manifest.v1+json, size=608
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140], platform=[unknown/unknown]: matches unknown platform... skipping...
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:e38676b0e3c3595eab09f46f195bffa17936b35f608dbf53abaf7729d65ca835]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG index [index.docker.io/library/busybox:latest]: digest=[sha256:c230832bd3b0be59a6c47ed64294f9ce71e91b327957920b6929a0caa8353140], platform=[unknown/unknown]: matches unknown platform... skipping...
2025-02-03 11:09:57 DBG descriptor [sha256:9186e638ccc30c5d1a2efd5a2cd632f49bb5013f164f6f85c48ed6fce90fe38f] = [application/vnd.oci.image.manifest.v1+json]
2025-02-03 11:09:57 DBG image [index.docker.io/library/busybox@sha256:9186e638ccc30c5d1a2efd5a2cd632f49bb5013f164f6f85c48ed6fce90fe38f]: type=application/vnd.oci.image.manifest.v1+json, size=610
2025-02-03 11:09:57 DBG descriptor [sha256:fe07eb6fec5e42ab3f5316287db8be5ac050e6f0090683b6ce2dfd855c5d90c3] = [application/vnd.oci.image.manifest.v1+json]
2025-02-03 11:09:57 DBG descriptor [sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5] = [application/vnd.docker.distribution.manifest.v2+json]
2025-02-03 11:09:57 DBG image [gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5]: type=application/vnd.docker.distribution.manifest.v2+json, size=737
2025-02-03 11:09:57 DBG descriptor [sha256:385a130f54c0295333e1f8192e5982d0c5af69e19d79ef1244e39f21a8029743] = [application/vnd.oci.image.manifest.v1+json]
2025-02-03 11:09:57 DBG descriptor [sha256:8b9303c0a5cc71fe4141ff8de7d05d28b15e817bc45f42ad0dc2c159bc592839] = [application/vnd.oci.image.manifest.v1+json]
2025-02-03 11:09:57 DBG starting the archival process for [.]
2025-02-03 11:09:57 DBG removing existing output file: [/Users/zackbradys/Documents/Coding/GitHub/hauler/haul.tar.zst]
2025-02-03 11:09:57 DBG mapping files in directory [.]
2025-02-03 11:09:57 DBG successfully mapped files for directory [.]
2025-02-03 11:09:57 DBG creating output file [/Users/zackbradys/Documents/Coding/GitHub/hauler/haul.tar.zst]
2025-02-03 11:09:57 DBG defining the archive format: [archives.Tar]/[archives.Zstd]
2025-02-03 11:09:57 DBG starting archive for [/Users/zackbradys/Documents/Coding/GitHub/hauler/haul.tar.zst]
2025-02-03 11:09:57 DBG archive created successfully [/Users/zackbradys/Documents/Coding/GitHub/hauler/haul.tar.zst]
2025-02-03 11:09:57 DBG closing output file [/Users/zackbradys/Documents/Coding/GitHub/hauler/haul.tar.zst]
2025-02-03 11:09:57 INF saved store [/Users/zackbradys/Documents/Coding/GitHub/hauler/store] -> [/Users/zackbradys/Documents/Coding/GitHub/hauler/haul.tar.zst]

@zackbradys zackbradys requested review from amartin120 and a team February 4, 2025 04:27
amartin120
amartin120 reviewed Feb 4, 2025
View reviewed changes
Copy link
Contributor

@amartin120 amartin120 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still stand by hauler store save using a flag for the filename because it's optional. I don't really like optional args if I can help it. I understand that hauler is currently a mixed bag.

If anything, hauler store load should be the thing that's changed to match save for consistency, not the other way around.

We can discuss on our call later if you'd like.

@zackbradys zackbradys closed this Feb 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amartin120 amartin120 amartin120 left review comments

Assignees

@zackbradys zackbradys

Labels
enhancement New feature or request size/L Denotes an issue/PR requiring a relatively large amount of work
Projects
Hauler
Status: Resolved
Milestone
Hauler v1.2.0
Development

Successfully merging this pull request may close these issues.

[feature] Allow hauler to load and sync remote files
2 participants
@zackbradys @amartin120