Question: What is the best way to run database migrations with copilot pipeline?

[adricu]

Hi!

I have a Django application that is deployed using copilot and I am using copilot Pipelines to automate the release process of the app. But I am not able to figure out how to do the database migration step process (python manage.py migrate --noinput) in a safely manner...

Until now we were using Kubernetes and had a manifest like this one:

apiVersion: batch/v1
kind: Job
metadata:
  name: backend-migrate
  labels:
    name: backend
    type: migrate
spec:
  activeDeadlineSeconds: 3600
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: backend-migrate
        image: ECR-ACC/backend:$IMAGE_TAG
        command: ["/bin/sh"]
        args: ["-c", "python manage.py migrate --noinput"]

Help is very much appreciated.

Thanks,

Adrián

[guyschlider]

I'm using Github actions for the release process, but it should be pretty easy to do the same using Pipelines:

using hooks in your pipelines specs, you can define after_install script to run python manage.py migrate:

See example here at step 14:
https://girisagar46.github.io/build-aws-codepipeline-for-cicd

[wstewarttennes]

I asked the same question here: https://gitter.im/aws/copilot-cli?at=617ef102a41fd20699178d80 -- still haven't had success but trying to work through it and will post any clean solution I come up with.

[wstewarttennes]

Following up -- I am getting this to work by putting the following in the pipeline.yml file underneath test_commands:

      test_commands: 
        - yum update -y && yum install -y sudo
        - wget https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.12.0 
        - mv ./copilot-linux-v1.12.0 ./copilot-linux       
        - chmod +x ./copilot-linux        
        - ./copilot-linux svc exec -c "python manage.py migrate" -n <service-name> -e <environment-name> --yes

I do think a feature that runs commands after tests would be useful, as this isn't really a test command. Also, if your healthcheck depends on your migrations being run (which usually should be the case, I believe), it's possible (and likely) that your deploy step will fail before the migration completes, which leaves you in some bad water if the migration was only halfway done (duplicate column errors the next time you try to run it for django specifically).

[iamhopaul123]

Thank you @wstewarttennes for the great answer! Yes the request totally makes sense. Actually we have another issue just opened up for tracking this feature request: #3007. I'll close this issue for now. Feel free to reopen if #3007 doesn't look good to your use case.

[hugours]

What is your current workaround for db migrations ?

[Lou1415926]

Hello @hugoduportet!

Does the workaround mentioned in the above #2992 (comment) work for you? In this solution, if you need to do db migration with pipeline, you can take advantage of the test_commands field as a workaround.

In addition, we have opened an issue for this use case. If this sounds good to you, would you mind giving it a thumb-up to help us prioritize?

Let me know if this helps!

[adonig]

I don´t use copilot right now but I ran into the same issue with both App Runner and ECS. IMO a nice solution is to use lock-exec. It's a CLI distributed locking helper using a DynamoDB table.

I copy the binary to my docker container's /usr/local/bin folder, set up the DynamoDB table with AWS CDK and pass the table name to my service using an environment variable (LOCK_EXEC_TABLE). My docker-entrypoint.sh looks like this:

#!/bin/bash
set -euxo pipefail

lock-exec run key 'python3 manage.py migrate --noinput' --table $LOCK_EXEC_TABLE

python3 manage.py collectstatic --noinput

python3 manage.py createsuperuser --noinput 2>/dev/null || true

exec "$@"

It typically goes like this:

1. The first task creates a lock and runs the migrations.
2. The second task skips running the migrations because of the lock.
3. The first task finishes running the migrations and releases the lock.
4. The third task creates a lock but there are no migrations left so it releases the lock.

The table uses pay-per-request pricing so you can have like a million requests and it costs you about $1.

[gabrieljoelc]

I tried the test_commands workaround @wstewarttennes provided like so:

  - name: dev
    deployments:
      api:
    test_commands:
      - yum update -y && yum install -y sudo
      - wget https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.27.0
      - mv ./copilot-linux-v1.27.0 ./copilot-linux
      - chmod +x ./copilot-linux
      - ./copilot-linux svc exec -c "yarn db:migrate:apply" -n $COPILOT_SERVICE_NAME -e $COPILOT_ENVIRONMENT_NAME

but am running into a docker runtime-version issue with that copilot version:

Selecting 'docker' runtime version '18' based on manual selections...
[Container] 2023/05/09 20:53:23 Phase complete: DOWNLOAD_SOURCE State: FAILED
[Container] 2023/05/09 20:53:23 Phase context status code: YAML_FILE_ERROR Message: Unknown runtime version named '18' of docker. This build image has the following versions: 20

is there a way to select the docker runtime version when in the test_commands context?

[KollaAdithya]

Hello @gabrieljoelc !
Here is the related issue #4721
It has already fixed in #4723, but the fix has not been released yet. Till the fix gets released Can you navigate to the CodeBuild project in the console and modify the buildspec to remove the docker runtime version below in install instructions.

install:
  runtime-versions:
    docker: 20

[gabrieljoelc]

@KollaAdithya That's great, but my runtime-version.docker is set to 20 not 18.

[KollaAdithya]

you are right. It should be 20 if you are using version v1.27.0. you can just remove that docker runtime version in the install instructions.

[gabrieljoelc]

@KollaAdithya so removing runtime-versions.docker entirely will get me past the "Unknown runtime version named '18' of docker" error?

[KollaAdithya]

That's true. As mentioned over here #4721 (comment) you can navigate to the Codebuild console and edit the buildspec to remove the docker version should fix the error.

[gabrieljoelc]

@KollaAdithya thank you so much for your help!

to be extra clear, can I change it in the buildspec.yml that Copilot created for me and redeploy my pipeline instead of doing it in the console?

here's what the top of my copilot/pipelines/api-main/buildspec.yml looks like right now:

version: 0.2
phases:
  install:
    runtime-versions:
      docker: 20
      ruby: 3.1
      nodejs: 16

[KollaAdithya]

Hey @gabrieljoelc !

I provided more context on which buildspec you should edit in this comment.

Basically you do not need to edit the copilot generated buildspec. you just need to edit the TestCommands buildspec in the codebuild console to remove docker version.