docker version 18 in buildspec of test_commands gives 'Unknown runtime version'

[lennertcc]

Hello,

Today I upgraded from 1.20 to 1.27.0.

I use test_commands for some post deployment steps and that started failing.

The buildspec that fails is

version: 0.2
phases:
  install:
      runtime-versions:
        docker: 18
  build:
    commands:
      - yum install -y sudo && yum install -y expect
      - wget https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.27.0
      - mv ./copilot-linux-v1.27.0 ./copilot-linux
      - chmod +x ./copilot-linux
      - unbuffer ./copilot-linux svc exec -c "python manage.py collectstatic --no-input --skip-checks" -n app -e test --yes
      - unbuffer ./copilot-linux svc exec -c "python manage.py migrate" -n app -e test --yes

The error I'm getting with that:

[Container] 2023/03/31 18:53:17 Waiting for agent ping
--
2 | [Container] 2023/03/31 18:53:18 Waiting for DOWNLOAD_SOURCE
3 | [Container] 2023/03/31 18:53:21 Phase is DOWNLOAD_SOURCE
4 | [Container] 2023/03/31 18:53:21 CODEBUILD_SRC_DIR=/codebuild/output/src759560563/src
5 | [Container] 2023/03/31 18:53:21 YAML location is /codebuild/readonly/buildspec.yml
6 | [Container] 2023/03/31 18:53:21 No commands found for phase name: install
7 | [Container] 2023/03/31 18:53:21 Setting HTTP client timeout to higher timeout for S3 source
8 | [Container] 2023/03/31 18:53:21 Processing environment variables
9 | [Container] 2023/03/31 18:53:21 Selecting 'docker' runtime version '18' based on manual selections...
10 | [Container] 2023/03/31 18:53:24 Phase complete: DOWNLOAD_SOURCE State: FAILED
11 | [Container] 2023/03/31 18:53:24 Phase context status code: YAML_FILE_ERROR Message: Unknown runtime version named '18' of docker. This build image has the following versions: 20

Seems like it sets docker version 18 in the buildspec whereas version 20 is available in the build image.

Comparing that to the generated buildspec for the build stage generated by v1.27.0 that also sets docker to 20 like so:

version: 0.2
phases:
  install:
    runtime-versions:
      docker: 20
      ruby: 3.1
      nodejs: 16

I did run copilot pipeline update and all the changes for updating to v1.27.0 are committed to main.

Manually changing the buildspec of the build project to use docker: 20 solves the issue with logging:

[Container] 2023/03/31 20:13:20 Waiting for agent ping
--
2 | [Container] 2023/03/31 20:13:21 Waiting for DOWNLOAD_SOURCE
3 | [Container] 2023/03/31 20:13:25 Phase is DOWNLOAD_SOURCE
4 | [Container] 2023/03/31 20:13:25 CODEBUILD_SRC_DIR=/codebuild/output/src591659411/src
5 | [Container] 2023/03/31 20:13:25 YAML location is /codebuild/readonly/buildspec.yml
6 | [Container] 2023/03/31 20:13:25 No commands found for phase name: install
7 | [Container] 2023/03/31 20:13:25 Setting HTTP client timeout to higher timeout for S3 source
8 | [Container] 2023/03/31 20:13:25 Processing environment variables
9 | [Container] 2023/03/31 20:13:25 Selecting 'docker' runtime version '20' based on manual selections...
10 | [Container] 2023/03/31 20:13:25 Running command echo "Specifying docker version in buildspec is deprecated. Using docker $DOCKER_VERSION"
11 | Specifying docker version in buildspec is deprecated. Using docker 20.10.22

[efekarakus]

Hi @lennertcc !!

Thank you so much for reporting this issue. It looks like we have to update this section of the CloudFormation template for the pipeline:

copilot-cli/internal/pkg/template/templates/cicd/pipeline_cfn.yml

Lines 302 to 304 in e1afe39

	install:
	runtime-versions:
	docker: 18

and completely remove docker in runtime-versions as its no longer needed to specify

[lominming]

Getting this error right now. The lines

copilot-cli/internal/pkg/template/templates/cicd/pipeline_cfn.yml

Lines 302 to 304 in e1afe39

	install:
	runtime-versions:
	docker: 18

may need to bump the version to 20?

[Container] 2023/04/30 06:57:20 Processing environment variables
--
9 | [Container] 2023/04/30 06:57:20 Selecting 'docker' runtime version '18' based on manual selections...
10 | [Container] 2023/04/30 06:57:22 Phase complete: DOWNLOAD_SOURCE State: FAILED
11 | [Container] 2023/04/30 06:57:22 Phase context status code: YAML_FILE_ERROR Message: Unknown runtime version named '18' of docker. This build image has the following versions: 20

[Lou1415926]

@lominming Hey there! It's been fixed in #4723, but the fix isn't rolled out yet! We will try to release the fix ASAP. For the time being, to get around this bug you can try to modify the buildspec for the test command CodeBuild project and remove ⬇️ lines altogether

install:
  runtime-versions:
    docker: 18

just like what https://github.com/aws/copilot-cli/pull/4723/files#diff-b23d8ebe6f7575eb66b41d872d02a08228ae1952ed3705bfb5b5ad4a8b27c403 did.

[lominming]

The buildspec only included: (No docker was created by copilot CLI)

runtime-versions:
   ruby: 3.1

I tried adding docker: 20 but it did not work either.

[efekarakus]

Hi @lominming, the buildspec.yml for test_commands isn't written to the disk. I believe @Lou1415926 was trying to say that for now, you'd need to navigate to the CodeBuild project in the console and modify the buildspec to remove the the install instructions.

Apologies for the inconvenience 🙇

[lominming]

@efekarakus @Lou1415926 Is there a command similar to copilot svc package for build/pipeline so I can actually get the generated buildspec? I cannot find how to grab the current buildspec used in the console under CodeBuilder. Let me know where I can grab the buildspec.

At the console, when I try to edit buildspec, it just says Using the buildspec.yml in the source code root directory without ability to see a buildspec.

[efekarakus]

Let me know where I can grab the buildspec.

1. If you navigate to CodeBuild, you should see a project that starts with BuildTestCommands
   

2. In the Edit dropdown, choose "Buildspec"
   

3. And finally update the Buildspec by deleting the lines:
   

From the error message above, it looks like perhaps the wrong CodeBuild project's buildspec is trying to be edited 💭

[lominming]

There's no project that starts with BuildTestCommands.
I only have 1 project that starts with pipeline-myapp

When I click on insert build commands, there's nothing.

[lominming]

I haven't change anything recently and suddenly I am getting which I think could help to shed light on what is going on. Previously without test_commands it will build fine, and now it triggers this error regardless of whether I use test_commands.

Error:

Success:

Compare to my last successful build:
(Note: it manages to find the YAML file)

[lominming]

I figured out why I got the YAML file missing error, it is because when I was trying to edit the buildspec, I accidentially deleted the path to the YAML file.

This is the original config by copilot pipeline. It appears to point the buildspec.yml that is generated, but you were saying that that is not the buildspec to modify?

To recap, the file copilot/pipelines/repo-main/buildspec.yml that is originally generated does not have docker: 18 in the runtime-versions. Adding docker: 20 does not help either.

[efekarakus]

Hi @lominming !

nice, I'm glad you found the issue 🚀 I'm struggling to follow if there is still an error or if after adding the buildspec file path the issue is fixed?

The buildspec should not have a docker entry anymore in the install phase

[lominming]

@efekarakus Sorry for the confusion, I am still getting the original error:

---

Sorry that I am still a bit unclear. It would be great if you could help clarify the following statements again.

1) There is a buildspec.yml in copilot/pipelines/repo-main/buildspec.yml
---> This file has NO docker entry (it had a old ruby version that I had to update to 3.1)
---> This buildspec appears to be the buildspec that CodeBuild is pointing to.
---> Question: Based on what you said earlier "the buildspec.yml for test_commands isn't written to the disk", so this is NOT the buildspec I should edit?

2. If that is not the buildspec I should be editing...
   ---> Question: Where can I find the correct buildpsec that I should be editing?

3. If that is the correct buildspec...
   ----> Question: Why is it complaining about docker: 18 when the entry is not in it?
   ----> Manually adding docker: 20 does not help.

---

For reference, the buildspec.yml generated by copilot in copilot/pipelines/repo-main/buildspec.yml

# Buildspec runs in the build stage of your environment pipeline to generate the environment CloudFormation stack config.
version: 0.2
phases:
  install:
    runtime-versions:
      # docker: 20 ###### NOTE: This entry is not there. Adding docker: 20 does not help either.
      ruby: 3.1 ###### This was originally 2.7, had to update to 3.1
    commands:
      - echo "cd into $CODEBUILD_SRC_DIR"
      - cd $CODEBUILD_SRC_DIR
      # Download the copilot linux binary. You can update this version.
      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.27.0
      - mv ./copilot-linux-v1.27.0 ./copilot-linux
      - chmod +x ./copilot-linux
  build:
    commands:
      - echo "Run your tests"
      # - make test
  post_build:
    commands:
      - ls -l
      - export COLOR="false"
      - export CI="true"
      - pipeline=$(cat $CODEBUILD_SRC_DIR/copilot/pipelines/repo-main/manifest.yml | ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))')
      - stages=$(echo $pipeline | jq -r '.stages[].name')
      # Generate the cloudformation templates.
      - >
        for env in $stages; do
          ./copilot-linux env package -n $env --output-dir './infrastructure' --upload-assets --force;
          if [ $? -ne 0 ]; then
            echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
            exit 1;
          fi
        done;
      - ls -lah ./infrastructure
artifacts:
  files:
    - "infrastructure/*"

[KollaAdithya]

Hey @lominming !
Can you please provide below information

• which version of copilot are you using?

Note: If you redeploy pipeline using copilot pipeline update and copilot pipeline deploy then everytime you need to edit the buildspec generated for TestCommands

Let me clarify there are two different buildspec over here and I am currently using copilot v1.27.0

1. copilot generated buildspec - copilot/pipelines/repo-main/buildspec.yml . This is the buildspec used for Build stage of the pipeline.

Link to the copilot generated buildspec

This file has NO docker entry (it had a old ruby version that I had to update to 3.1)

It will have pre-populated docker version as 20 and ruby version 3.1 with copilot version 1.27.0.

2. Buildspec for TestCommands. This is generated by our pipeline CloudFormation template

Edit the TestCommands buildspec to remove the docker version from install instructions

If that is not the buildspec I should be editing...
---> Question: Where can I find the correct buildpsec that I should be editing?

You have to edit the buildspec of BuildTestcommands in codebuild console.

You can go to CodeBuild >Buildprojects in codebuild console and find the Buildproject starting with name TestCommands
and follow the instructions as mentioned by @efekarakus over here to remove docker version.

[lominming]

Updating my previous error that could be helpful to others. copilot pipeline init has 2 types of pipeline creations: workload and environment. It was not very clear which one should be used. I have been using the workload which is not what I want (and I suspect not what most people want).

copilot pipeline -> environment is the one that allows you to receive code changes from github (or other sources), build it, deploy to test, test, then deploy to prod.

[lominming]

@KollaAdithya I am now facing another issue from a recent copilot pipeline deploy.

What I did:

• setup copilot pipeline init with environment
• stages are github > build > deploy to test + test_commands > deploy to prod
• inside copilot/pipelines/<repo>/buildspec.yml, I added these lines for tests commands

test_commands: [
      echo 'running tests',
      npm install --force,
      npm run test
    ]

• It appears that it is not running in the correct directory?

[dannyrandall]

Hey @lominming! Just wanted to clarify something from #4721 (comment) - Environment pipelines deploy Copilot environments, which don't involve any of your code (like copilot env deploy). Workload pipelines deploy Copilot workloads into environments, and build/push your code as part of deploying each service (like copilot svc deploy or copilot job deploy). Both pipeline types support triggering based on git commits.

So for example an environment pipeline would have the steps:

1. Deploy the beta environment
2. Deploy the prod environment

A workload pipeline would have the steps:

1. Deploy the services foo and bar into environment beta
2. Deploy the services foo and bar into environment prod

Does this help clarify anything for you? I just want to make sure that an environment pipeline is what you're expecting. If you're deploying code for a service that you wrote, you want a workload pipeline.