Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add optional embedding of container images #239

Merged
merged 22 commits into from
Aug 28, 2024
Merged

Add optional embedding of container images #239

merged 22 commits into from
Aug 28, 2024

Commits on Aug 27, 2024

  1. Update linuxkit to 1.5.0: 

    This version has volume support that is needed
for embedding images.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    6235ab6 View commit details
    Browse the repository at this point in the history

  2. Fix Docker build warnings: 

    Docker was warning about the case of `FROM`
not matching `AS`.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    ff15b8a View commit details
    Browse the repository at this point in the history

  3. Add optional embedding of container images into DinD: 

    This helps use cases where images already existing
in the DinD cache is needed. Air gap envs, latency
constrained/concerned envs, etc.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    9fc15e8 View commit details
    Browse the repository at this point in the history

  4. Add note on embedded images being optional: 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    9318be6 View commit details
    Browse the repository at this point in the history

  5. Add platform option to pull-images.sh: 

    This makes it so that the correct architecture for
the embedded images are pulled.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    f68dc4b View commit details
    Browse the repository at this point in the history

  6. Don't git track the images.txt file: 

    There is an example file already. The images.txt
will be something the user creates for themselves.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    d481896 View commit details
    Browse the repository at this point in the history

  7. Update the embed process: 

    This removes the need to mv the
directory of embedded images. This is
accomplished by bind mount (rw) the
read only images location onto the /var/lib/docker
directory in the Hook-docker container.

This means that start up doesn't need to wait for
the mv command to complete. So startup doesn't incur
any delay like it was with the mv. This also means that
we can embed a lot more images with having start up issue.
In testing, I found that if enough images, compared to the
amount of memory available, were embedded then HookOS would
not boot up. It would max out on memory. It's possible with
enough time that it would have booted but i didnt wait longer
than about 30min.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    4165fe2 View commit details
    Browse the repository at this point in the history

  8. Change the way the images are presented to hook-docker: 

    Instead of having hook-docker know about mounting the
embedded images, the images get mounted with the right
permissions and made available to hook-docker at the
"usual" location. This decouples this embedding process
from hook-docker. This should allow the two to only be
coupled by the mount point of /var/run/images.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    72fde59 View commit details
    Browse the repository at this point in the history

  9. Add newlines to all files without them: 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    385941e View commit details
    Browse the repository at this point in the history

  10. Remove .dockerignore: 

    It adds an unnecessary place to need to be aware of
when adding or renaming files.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    d0d5e0f View commit details
    Browse the repository at this point in the history

  11. Make the docker:dind image configurable: 

    Allows users to specify specific versions of
docker:dind to use.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    04dc1bc View commit details
    Browse the repository at this point in the history

  12. Refactor script to pull images: 

    This pulls images from the local docker
client instead of from the DinD container.
This will allow for registries that need logged
into and any proxying that might be needed to occur
during an image pull.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    4bd7ee6 View commit details
    Browse the repository at this point in the history

  13. Fix multi arch: 

    When existing images in the local Docker image
cache existed for an image the `--platform` arg
doesn't matter. This means that when an existing
amd64 image is already in the cache the arm64
pulls will not happen. To fix this we always delete
the image before pulling.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    882aa21 View commit details
    Browse the repository at this point in the history

  14. Load images from inside DinD: 

    This resolves issues with needing to
mount the docker.sock and needing sudo.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    b490bac View commit details
    Browse the repository at this point in the history

  15. Add note on Docker storage driver: 

    Because hook-docker uses the overlay2
storage driver, the local docker client
using pull-images.sh must also use the
overlay2 storage driver.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    4a8f7ce View commit details
    Browse the repository at this point in the history

  16. Remove note on host Docker storage driver: 

    The host Docker storage driver actually doesn't
matter at all because we use DinD. This was my
mistake.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    58eb394 View commit details
    Browse the repository at this point in the history

  17. Update images.txt format: 

    This adds the ability to remove the "source
image" tag from the final embedded images.
This leaves only the "additional tag".

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    de8a316 View commit details
    Browse the repository at this point in the history

  18. Add some code comments 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    67a2d6c View commit details
    Browse the repository at this point in the history

  19. Check that DinD uses the overlay2 storage driver: 

    As hook-docker uses the overlay2 storage driver
the DinD image must use the overlay2 storage driver too.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    845d6b2 View commit details
    Browse the repository at this point in the history

  20. Add ssl certs to SSH container: 

    This resolves an issue with things
like apk not working in the ssh container.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    6be6b09 View commit details
    Browse the repository at this point in the history

  21. Fix build process: 

    Remove tabs that were causing the build
to fail.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    50cdc1d View commit details
    Browse the repository at this point in the history

  22. Update hook-bootkit: 

    Update go.mod dependencies. Check for tink-worker
image and don't fail the image pull if it doesn't
exist. With embedded images, the tink worker could
potentially already exist in the local Docker image
cache. And the image name could be something
unreachable via the network
(for example: 127.0.0.1/embedded/tink-worker).

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    78f2850 View commit details
    Browse the repository at this point in the history