How to reuse docker build on further steps/jobs without pushing to registry?

[h2ogeek]

I would l like to be able to reuse the Docker image that I built in a step in the upcoming ones. I want to avoid pushing that image to a registry. I have not been able to find out how to do that.

I attached part of my action.

- name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        id: buildx
        with:
          install: true
          buildkitd-flags: --debug
- name: Bake Docker Image
        run: |
          echo ${{secrets.NPM_TOKEN}}
          echo ${{secrets.NPM_TOKEN}} > NPM_TOKEN.txt
          cat NPM_TOKEN.txt
          docker build -t myimage:latest --progress=plain --secret id=NPM_TOKEN,src=NPM_TOKEN.txt .
          docker image ls -a
- name: Test
        run: docker image ls -a

[airtower-luna]

KhimairaCrypto:

I would l like to be able to reuse the Docker image that I built in a step in the upcoming ones.

Do you mean steps or jobs? The image should be available in subsequent steps of the same job, e.g. myimage:latest should be visible in the “Test” step. If it isn’t please share the details, ideally a link to the workflow log.

Using the image in another job won’t work without pushing to a registry. GitHub-hosted runner VMs are discarded after their job is complete, and if you use self-hosted runners the next job might be assigned to another one.

[h2ogeek]

Hi @airtower-luna,

At this stage, I am pretty open to either steps or jobs.

In my .yml file, I am listing the docker images at the step and the job level. The result is the same in both scenarios, and I can not see the new image `*/myimage: latest.

The goal is to build my custom/disposable image and be able to use it across all the different jobs. But based on your last comment, my only option is to do as much as possible at the step level.

here is the result of docker image ls -a on Bake Docker Image

REPOSITORY          TAG               IMAGE ID       CREATED        SIZE
100
moby/buildkit       buildx-stable-1   440639846006   38 hours ago   142MB
101
node                12-alpine         8dc0ee810c0a   3 days ago     91MB
102
node                14-alpine         6f5dba13ae83   3 days ago     119MB
103
node                16-alpine         59b389513e8a   3 days ago     111MB
104
alpine              3.12              24c8ece58a1a   4 days ago     5.58MB
89
alpine              3.13              20e452a0a81a   4 days ago     5.61MB
90
alpine              3.14              e04c818066af   4 days ago     5.59MB
91
node                12                543479162c86   9 days ago     918MB
92
node                14                7d03025aad90   9 days ago     945MB
93
node                16                424bc28f998d   9 days ago     906MB
94
buildpack-deps      stretch           de7321966554   10 days ago    835MB
95
buildpack-deps      buster            cbdff219ec50   10 days ago    804MB
96
buildpack-deps      bullseye          c93648763550   10 days ago    834MB
97
debian              9                 cb32206528ee   11 days ago    101MB
98
debian              10                76e02db62235   11 days ago    114MB
99
debian              11                d69c6cd3a20d   11 days ago    124MB
100
ubuntu              20.04             ff0fea8310f3   3 weeks ago    72.8MB
101
ubuntu              18.04             b67d6ac264e4   3 weeks ago    63.2MB
102
moby/buildkit       latest            ab57028900a2   4 weeks ago    142MB
103
tonistiigi/binfmt   latest            71d103174212   4 weeks ago    35.8MB
104
ubuntu              16.04             b6f507652425   7 months ago   135MB

[airtower-luna]

Most of those images look like they’re the ones cached on Actions runners. Without more details about your workflow (full job and its output) I don’t think it’s possible to say what may be going wrong, I can only say it works in my workflows. I use Docker commands directly though, without docker/setup-buildx-action.

[h2ogeek]

Please let me know what additional details would you like to know!. What logs do you want to see?. I list the images on as you can see the fresh new image is not there. The docker part of the workflow Is all there. You mentioned that you are not using BuildX. I can need to use buildX in order to be able to pass a secret. Without Adding BuildX that is not possible.

the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled
31
Error: Process completed with exit code 1.

here is the sample Dockefile that I use for testing purposes:

# syntax=docker/dockerfile:1.2
FROM alpine
shows secret from default secret location:
RUN --mount=type=secret,id=NPM_TOKEN cat /run/secrets/NPM_TOKEN
CMD ["/bin/sh"]

[h2ogeek]

I refactored my workflow to use docker/build-push-action@v2 and added the actions/cache@v2 too. I am still not able to see myimage:latest in the Lint and Test step.

name: first
on: [push]
env:

NPM_TOKEN: ${{secrets.NPM_TOKEN}}
jobs:

buildx:

runs-on: ubuntu-latest

steps:

- name: Checkout

uses: actions/checkout@v2

- name: Set up QEMU

uses: docker/setup-qemu-action@v1

- name: Set up Docker Buildx

uses: docker/setup-buildx-action@v1

id: buildx

with:

install: true

buildkitd-flags: --debug

- name: Cache Docker layers

uses: actions/cache@v2

with:

path: /tmp/.buildx-cache

key: ${{ runner.os }}-buildx-${{ github.sha }}

restore-keys: |

${{ runner.os }}-buildx-

- name: Bake Docker Image

uses: docker/build-push-action@v2

with:

context: .

builder: ${{ steps.buildx.outputs.name }}

push: false

tags: user/myimage:latest

secrets: |

"NPM_TOKEN=${{secrets.NPM_TOKEN}}"

cache-from: type=local,src=/tmp/.buildx-cache

cache-to: type=local,dest=/tmp/.buildx-cache-new

- name: Available platforms

run: echo ${{ steps.buildx.outputs.platforms }}

- name: Lint and Test

run: docker image ls -a

# This ugly bit is necessary if you don't want your cache to grow forever

# till it hits GitHub's limit of 5GB.

# Temp fix

# docker/build-push-action#252

# moby/buildkit#1896

- name: Move cache

run: |

rm -rf /tmp/.buildx-cache

mv /tmp/.buildx-cache-new /tmp/.buildx-cache

[h2ogeek]

I managed to save my docker image, by saving it as an artifact :-). This will allow me to use it in the following steps :-).

name: first
on: [push]
env:

NPM_TOKEN: ${{secrets.NPM_TOKEN}}
jobs:

build:

runs-on: ubuntu-latest

steps:

- name: Checkout

uses: actions/checkout@v2

- name: Set up QEMU

uses: docker/setup-qemu-action@v1

- name: Set up Docker Buildx

uses: docker/setup-buildx-action@v1

id: buildx

with:

install: true

buildkitd-flags: --debug

- name: Cache Docker layers

uses: actions/cache@v2

with:

path: /tmp/.buildx-cache

key: ${{ runner.os }}-buildx-${{ github.sha }}

restore-keys: |

${{ runner.os }}-buildx-

- name: Bake Docker Image

uses: docker/build-push-action@v2

with:

context: .

builder: ${{ steps.buildx.outputs.name }}

push: false

tags: user/myimage:latest

secrets: |

"NPM_TOKEN=${{secrets.NPM_TOKEN}}"

cache-from: type=local,src=/tmp/.buildx-cache

cache-to: type=local,dest=/tmp/.buildx-cache-new

outputs: type=docker,dest=/tmp/myimage.tar # export docker image

- name: Upload artifact

uses: actions/upload-artifact@v2

with:

name: myimage

path: /tmp/myimage.tar

# This ugly bit is necessary if you don't want your cache to grow forever

# till it hits GitHub's limit of 5GB.

# Temp fix

# docker/build-push-action#252

# moby/buildkit#1896

- name: Move cache

run: |

rm -rf /tmp/.buildx-cache

mv /tmp/.buildx-cache-new /tmp/.buildx-cache
use-image:

runs-on: ubuntu-latest

needs: build

steps:

- name: Set up Docker Buildx

uses: docker/setup-buildx-action@v1

- name: Download artifact

uses: actions/download-artifact@v2

with:

name: myimage

path: /tmp

- name: Load image

run: |

docker load --input /tmp/myimage.tar

docker image ls -a

[lgrosz]

Are you able to run a step a subsequent step with the docker/build-push-action@v2? When I attempt to run another step with that action whenever I reference the loaded docker image it attempts to find it from docker hub instead of the local machine.