Make Docker step less redundant

[bschilder]

Currently, I build the Docker container twice (once in the beginning of the Linux workflow, and again via the docker/build-push-action@v1 step. Would be more efficient to just push the first one.

[bschilder]

Should also upgrade the action. This step may help:

https://github.com/Bioconductor/bioconductor_docker/blob/RELEASE_3_16/.github/workflows/build_containers.yaml#L28

[bschilder]

Related issue:
docker/setup-buildx-action#251

Though their solution didn't help in my case:

uses: docker/setup-buildx-action@v2
      with:
        driver: docker
        install: true

Nor did mounting volumes in the matrix config:
https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/customizing-the-containers-used-by-jobs

 cont: 
            image: ghcr.io/bioconductor/bioconductor_docker:devel
            volumes: 
              - /usr/local/bin/docker:/usr/bin/docker
              - /var/run/docker.sock:/var/run/docker.sock
            

Still getting the same error:

Error: Unable to locate executable file: docker. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.

[bschilder]

Tried many variations of this, all of them ran into the same error regarding being unable to find Docker executable.

@almahmoud do you have any insights for how to overcome these issues on GHA?

#### Docker registry steps ####
    - name: 🐳🛠 Set up Docker Buildx
      if: |
        (!contains(github.event.head_commit.message, '[nodocker]')) && inputs.run_docker == 'true' && runner.os == 'Linux'
      uses: docker/setup-buildx-action@v3
        
    - name: 🐳🛠 Set up QEMU
      if: |
        (!contains(github.event.head_commit.message, '[nodocker]')) && inputs.run_docker == 'true' && runner.os == 'Linux'
      uses: docker/setup-qemu-action@v3
      
    - name: 🐳🔐 Log in to the Github Container registry - ghcr.io
      if : |
        (!contains(github.event.head_commit.message, '[nodocker]')) && inputs.run_docker == 'true' && runner.os == 'Linux' && inputs.docker_registry == 'ghcr.io'
      uses: docker/login-action@v3
      with:
        registry: ghcr.io
        username: ${{ github.repository_owner }}
        password: ${{ inputs.GITHUB_TOKEN }}
        
    - name: 🐳🔐 Log in to the Github Container registry - docker.io
      if: |
        (!contains(github.event.head_commit.message, '[nodocker]')) && inputs.run_docker == 'true' && runner.os == 'Linux' && inputs.docker_registry == 'docker.io'
      uses: docker/login-action@v3
      with:
        registry: ghcr.io
        username: ${{ inputs.docker_user }}
        password: ${{ inputs.DOCKER_TOKEN }}  
      
    - name: 🐳✏️ Create Docker file 
      if: |
        (!contains(github.event.head_commit.message, '[nodocker]')) && inputs.run_docker == 'true' && runner.os == 'Linux'
      run: |
        options(crayon.enabled = TRUE)
        if(!require("rworkflows", quietly=TRUE)) remotes::install_github("neurogenomics/rworkflows")
        path <- rworkflows::use_dockerfile()
      shell: Rscript {0} 
      
    - name: 🐳🚀 Build and push to Docker registry 
      if: |
        (!contains(github.event.head_commit.message, '[nodocker]')) && inputs.run_docker == 'true' && runner.os == 'Linux'
      uses: docker/build-push-action@v5
      with: 
        push: true
        tags: |
          ${{ inputs.docker_registry }}/${{ inputs.docker_org }}/${{ env.packageName }}:${{ env.packageVersion }},
          ${{ inputs.docker_registry }}/${{ inputs.docker_org }}/${{ env.packageName }}:latest
        build_args: |
          PKG=${{ env.packageNameOrig }},
          BASE_IMAGE=${{ matrix.config.cont }}

[almahmoud]

Can you clarify what you're trying to do?
Looking at the previous messages, my best guess is you're trying to use docker inside the bioconductor docker container? If so, you might need the Docker CLI installed manually since it's not by default in the container

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

I'm assuming GHA automatically mount the docker sockets for docker in docker to work but if they don't, you might need that too

[bschilder]

Can you clarify what you're trying to do? Looking at the previous messages, my best guess is you're trying to use docker inside the bioconductor docker container? If so, you might need the Docker CLI installed manually since it's not by default in the container

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

I'm assuming GHA automatically mount the docker sockets for docker in docker to work but if they don't, you might need that too

thanks, @almahmoud! Here is the premise of the issue:

• when running the rworkflows action on Ubuntu, I start by pulling the bioc docker container and run all subsequent steps within that container.
• those steps include not only installing the package that I'm testing (and its deps), but also all the other things I've done to customise the container environment throughout the action (e.g. installing conda, installing extra system deps)
• then, at the end of the action, I want to upload a version of the container that has all the modifications that were made by the action after the initial build. However, the only way I'm aware of pushing a container to a registry like DockerHub/GHCR is by making a Dockerfile that basically specifies everything that should go into the container that will be uploaded. In essence, this repeats all of the things I've already done previously with the action (installing R packages, installing system deps, conda, etc).
• This seems rather redundant, since it means if I want to truly replicate the environment in which my R package was tested, I would need to re-specify every one of the steps in the action within the Dockerfile. And I'm not even sure I could mimic every action step using the Dockerfile syntax.
• So my question is, is there instead a way to skip the Dockerfile step and just directly upload the container to DockerHub/GHCR at the end of the action?

I'm not sure if I explained that very well, so if it's not clear let me know and I'd be happy to explain my thought process a bit better on a call.

[bschilder]

A suggestion that came up in the most recent Bioc Cloud Methods WG meeting: Add the COPY --from=base / / step in the Dockerfile. I don't fully understand what this does yet but will delve into it more.

• https://github.com/Bioconductor/bioconductor_docker/blob/devel/Dockerfile#L53

[almahmoud]

The context of the the COPY there, is that we run through an arbitrary number of steps in the build stage, then essentially restart from the base image and copy the entirety of the contents from the end of the build stage into the base image, resulting in a single layer for the diff between the base and built image, essentially squashing all Bioconductor additions into a single layer over the rocker base image.

More generally, some hopefully helpful pointers:

• Docker multi-stage builds would probably help if you're looking to potentially create different stages for build vs test vs log extraction.
• Another example of a stage for extracting only logs/artifacts is in gha-build, notably here where desired export files are copied into a stage created from scratch.
• Also, if you pursue the multi-stage build approach, it might be useful to also look at docker buildx bake which could allow you to target and export multiple stages in a single run.