access task run command's exit code

[nbarrera]

I got my CI's step executing a copilot task run, the task fails with exit 1 and I would like the CI step to fail and bail without executing the following step on the CI pipeline.

Here is the tail of the Dockerfile I 'm feeding the task run

ENTRYPOINT yarn migrate-up

Here is an task run output excerpt (note command failed with exit 1)

% DOCKER_DEFAULT_PLATFORM=linux/amd64 copilot task run   --follow   
✔ Proposing infrastructure changes for stack task-x-titles-runtime 
...
...
...
copilot-task/x-title yarn run v1.22.5
...
...
copilot-task/x-title ERROR: connect ECONNREFUSED 127.0.0.1:27017 MongoServerSelectionError: connect ECONNREFUSED 127.0.0.1:27017
copilot-task/x-title     at Timeout._onTimeout (/usr/src/app/node_modules/migrate-mongo/node_modules/mongodb/lib/core/sdam/topology.js:438:30)
copilot-task/x-title     at listOnTimeout (internal/timers.js:557:17)
copilot-task/x-title     at processTimers (internal/timers.js:500:7)
copilot-task/x-title error Command failed with exit code 1.
copilot-task/x-title info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Task has stopped.

When issued manually from my computer, output is the same and right after that I get the task run exit code like this (note the task run exit code is 0 even if the command's exit code was 1)

% echo $?
0

Is there a way to get what's the exit code of the task that ran inside the container?

If I run that dockerfile locally I can get that:

docker run d295e8ac6951
...
...
...
copilot-task/x-title error Command failed with exit code 1.
copilot-task/x-title info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
% echo $?
1

So, if the task run command does not forward the command's exit code.., could I get it somehow?

thank you in advance guys

[nbarrera]

I 've found some thought at the command design issue, but as far as I 've tested this is not implemented.

The CLI returns the task's container's exit code itself. So a failure ECS-side causes failure on CLI-side. Great for running within a pipeline.

[efekarakus]

Ooh hi @nbarrera !

That's a great feature request, you're right we don't have a way yet of reflecting the task exit code at the moment 🙇

Looks like we can achieve this by calling DescribeTasks and checking the exitCode: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Container.html#ECS-Type-Container-exitCode

[nbarrera]

Sorry @efekarakus, I think I may use your suggestion of calling DescribeTasks to temporarily workaround my problem in the mean time.

I would like to ask you:

Is there a way to get the task id (so I can pass it to DescribeTasks) from copilot once copilot task run --follow has finished?

[Lou1415926]

Hi @nbarrera!

Unfortunately we don't surface task ID at this moment😞 Right now you'd be able to access the task's ARN by calling ListTasks.

For example, if using aws cli, you can run

aws ecs list-tasks --desired-status STOPPED --family copilot-<task-name> --cluster <cluster-name> --started-by copilot-task

<task-name> will be either the one you've entered, or one that's defaulted to the working directory's name.

--cluster <cluster-name> can be omitted if your tasks are running in the default cluster. Otherwise, it will be the cluster that corresponds to your application + environment. The easiest way to get this value would be to run

aws resourcegroupstaggingapi get-resources --tag-filters Key=copilot-application,Value=<app> Key=copilot-environment,Values=<env> --resource-type-filters ecs:cluster

There may be more than one results in the response, depending on the --count you specified, and whether you have some previous copilot task run execution. Since ECS will delete any STOPPED tasks only after a period of time, you could get results from some previous run that's stopped but not for long enough that ECS has deleted them.

In that case, you may check the task definition version to find the latest deployment.

An alternative way is to run

aws resourcegroupstaggingapi get-resources --tag-filters Key=copilot-task,Values=<task-name> --resource-type-filters ecs:task

but this will only give you the RUNNING tasks instead of the STOPPED one :( if you're able to run this command after the task is running, and feed the ARN to DescribeTasks after --follow finishes, this is the solution with less run and API call.

I'm sorry that there's so much steps for a simple tasks 😞 I will update the answer if I find a simpler solution too.

Meanwhile, thank you for the great feature request!!!

[coopsmoss]

I'd also love to have this feature!

But in the mean time if anyone else is wondering about this, I think this is the work around I'm going to go with, it's a little janky but should do the trick.

1. Run your copilot task using --follow and save the output to a file | tee $COPILOT_TASK_OUTPUT
2. Print out a special string if your task completes successfully ( echo "MIGRATION_SUCCEEDED" )
3. Check to see if that special string is in the saved output and based on that you know if the task succeeded
4. Delete your buffer file if you want

copilot_task.sh (The code you actually run on AWS)

#!/bin/bash
set -e    # causes the script to fail as a whole if any part of it fails
python3 database_migrations.py  # do whatever you have to do
echo "MIGRATION_SUCCEEDED"  # print out your special string (make sure this never prints if your task fails)

launch_copilot_task.sh (The code you use to launch the task)

#!/bin/bash
COPILOT_TASK_OUTPUT="task_output_tmp.txt"     # The buffer file
# runs task and save's output to buffer
copilot task run --follow --command copilot_task.sh | tee $COPILOT_TASK_OUTPUT 

# Check for the appearance of special string
if cat $COPILOT_TASK_OUTPUT | grep -q "MIGRATION_SUCCEEDED"; then  
  echo "Migration Successful";
  rm $COPILOT_TASK_OUTPUT  # delete buffer file (optional)
else
  echo "Migration Failed";
  rm $COPILOT_TASK_OUTPUT  # delete buffer file (optional)
  exit 1
fi

That's all there is to it!
Just make sure that special string is consistent in both files or this all breaks down pretty quickly.

(Note! This is an example and I haven't tested this in production!)

[efekarakus]

From gitter as well:

Hello 👋 ,
I am trying to setup a CI/CD pipeline for a Rails web app using AWS copilot. I'd appreciate your help in figuring out the best way to run database migrations.

I've read through aws/copilot-cli#2992 which suggests to use test_command as a workaround. However, it looks like test_command would run only after the service has been deployed. I'd prefer to run the migrations first and deploy the service only if the migration succeed.

As an alternate approach, I was looking into using Github Action instead of Code Pipeline. I was thinking of running the migration as one-off task using the task run and then running svc deploy only if the migration task succeed. However, I couldn't figure out a way to know if the migration task succeeded or resulted in an error. I could see the logs with the --follow flag, but not sure how I could use those logs to automatically stop the Github Action workflow if there are any errors.
https://gitter.im/aws/copilot-cli?at=6274f61ccd938f6ea2332a60

This feature request makes a lot of sense !

An alternative design, could be something like :

$ copilot task show -n <task group name>
<display metadata around containers created with the task>

but I think it makes sense to exit with a non-zero value if any of the tasks with --follow fails

[raethlo]

Hey all 👋🏻 solving a similar issue as the OP - super cool to see this is already implemented in #3620 🙏🏻 is there an estimate of when you want to ship the next release? Thanks

[dannyrandall]

Hey @raethlo! Glad to see you'll be able to make use of this feature.🎉 We are planning on releasing it this month!😊

[Lou1415926]

Closing this issue since the feature has been released v1.20.0! Feel free to reopen for further questions.