Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1.65.0: action fails with exit code 141 pipefail #269

Closed
baumac opened this issue May 18, 2023 · 11 comments
Closed

v1.65.0: action fails with exit code 141 pipefail #269

baumac opened this issue May 18, 2023 · 11 comments
Assignees

Comments

@baumac
Copy link
Contributor

baumac commented May 18, 2023

Summary

We updated the action from v.1.62.0 -> v1.65.0 and now the action is failing ~50% of the time with Docker Action run completed with exit code 141 which we believe is the result of a pipe error occurring on line 85 of entrypoint.sh.

Debug Logs

##[debug]Evaluating: needs.bump-version.outputs.new-version
##[debug]Evaluating Index:
##[debug]..Evaluating Index:
##[debug]....Evaluating Index:
##[debug]......Evaluating needs:
##[debug]......=> Object
##[debug]......Evaluating String:
##[debug]......=> 'bump-version'
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'outputs'
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'new-version'
##[debug]=> '0.0.416'
##[debug]Result: '0.0.416'
##[debug]Evaluating: secrets.GH_TOKEN
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'GH_TOKEN'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating condition for step: 'Create GH Tag'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Create GH Tag
##[debug]Loading inputs
##[debug]Loading env
Run anothrNick/[email protected]
  env:
    DOCKER_BASE_IMAGE_NAME: baumac/test
    DOCKER_OCI_ARCHIVE: docker-archive
    GITHUB_TOKEN: ***
    CUSTOM_TAG: 0.0.416
    VERBOSE: true
/usr/bin/docker run --name ed866e737379fcd0fc48c5941d83db8a81dea6_9f19db --label ed866e --workdir /github/workspace --rm -e "DOCKER_BASE_IMAGE_NAME" -e "DOCKER_OCI_ARCHIVE" -e "GITHUB_TOKEN" -e "CUSTOM_TAG" -e "VERBOSE" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_DEBUG" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/baumac/test":"/github/workspace" ed866e:737379fcd0fc48c5941d83db8a81dea6
++ git rev-parse --abbrev-ref HEAD
*** CONFIGURATION ***
	DEFAULT_BUMP: minor
	DEFAULT_BRANCH: 
	WITH_V: false
	RELEASE_BRANCHES: master,main
	CUSTOM_TAG: 0.0.416
	SOURCE: .
	DRY_RUN: false
	INITIAL_VERSION: 0.0.0
	TAG_CONTEXT: repo
	PRERELEASE: false
	PRERELEASE_SUFFIX: beta
	VERBOSE: true
	MAJOR_STRING_TOKEN: #major
	MINOR_STRING_TOKEN: #minor
	PATCH_STRING_TOKEN: #patch
	NONE_STRING_TOKEN: #none
	BRANCH_HISTORY: compare
pre_release = false
+ current_branch=main
+ pre_release=false
+ IFS=,
+ read -ra branch
+ for b in "${branch[@]}"
+ [[ main == \m\a\s\t\e\r ]]
+ '[' master '!=' master ']'
+ for b in "${branch[@]}"
+ [[ main == \m\a\i\n ]]
+ pre_release=false
+ '[' main '!=' main ']'
+ echo 'pre_release = false'
+ git fetch --tags
From https://github.com/baumac/test
   e51e64f..bde24f1  main                    -> origin/main
 * [new branch]      prod-us                 -> origin/main-deploy
 * [new tag]         0.0.4                   -> 0.0.4
 * [new tag]         0.0.414                 -> 0.0.414
+ tagFmt='^v?[0-9]+\.[0-9]+\.[0-9]+$'
+ preTagFmt='^v?[0-9]+\.[0-9]+\.[0-9]+(-beta\.[0-9]+)$'
+ case "$tag_context" in
++ head -n 1
++ grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+$'
++ git for-each-ref --sort=-v:refname --format '%(refname:lstrip=2)'
++ true
+ tag=0.0.414
##[debug]Docker Action run completed with exit code 141
##[debug]Finishing: Create GH Tag
@baumac baumac changed the title Action fails with exit code 141 pipefail for v1.65.0 Action fails with exit code 141 pipefail after upgrading to v1.65.0 May 18, 2023
@baumac baumac changed the title Action fails with exit code 141 pipefail after upgrading to v1.65.0 v1.65.0: action fails with exit code 141 pipefail May 18, 2023
@sbe-arg
Copy link
Collaborator

sbe-arg commented May 19, 2023

whats the result of this?

preTagFmt="^v?[0-9]+\.[0-9]+\.[0-9]+(-$suffix\.[0-9]+)$"
git for-each-ref --sort=-v:refname --format '%(refname:lstrip=2)' | (grep -E "$preTagFmt" || true) | head -n 1
echo $?

also can you share more context of how your workflow operates? I see you run based on customtags instead of auto discovery

also what do you mean that fails 50% of the time?

@sbe-arg
Copy link
Collaborator

sbe-arg commented May 19, 2023

have a look at this https://unix.stackexchange.com/questions/580117/debugging-sporadic-141-shell-script-errors regarding head command and number of filters... I see u have 400 patch tags do they all exist in the repo history?

@sbe-arg
Copy link
Collaborator

sbe-arg commented May 19, 2023

To me sounds the head command is signaling 141 after line 84 not necessarily after line 85 (85 does not even start)

@sbe-arg
Copy link
Collaborator

sbe-arg commented May 19, 2023

if this is the case we need to figure out a way to handle SIGPIPE as feature request (in between clearup some tags) or remain in 1.62 or lower for now.

Interesting scenario

@sbe-arg sbe-arg self-assigned this May 19, 2023
@baumac
Copy link
Contributor Author

baumac commented May 19, 2023

whats the result of this?

preTagFmt="^v?[0-9]+\.[0-9]+\.[0-9]+(-$suffix\.[0-9]+)$"
git for-each-ref --sort=-v:refname --format '%(refname:lstrip=2)' | (grep -E "$preTagFmt" || true) | head -n 1
echo $?

When ran locally the result of this is 0. I'd imagine that it's not zero when the pipeline fails, but I can't confirm this. I'll fork this repo see if I can capture the output of this today in the pipeline.

also can you share more context of how your workflow operates? I see you run based on customtags instead of auto discovery

Yup in short our workflow:

  1. Checks out the source code
  2. Creates the new version and stores it in ${{ bump-version.outputs.new-version }} (uses actions-ecosystem/[email protected])
  3. Creates and pushes a docker image using the new version
  4. Creates the git tag using the new version (anothrNick/[email protected])
  5. Pushes the new git tag

also what do you mean that fails 50% of the time?

The github-tag-action is flakey. Approximately half the time the action will succeed, and the other half of the time it will fail.

I see u have 400 patch tags do they all exist in the repo history?

Yup they all exist in the repo history, the real output of git fetch --tags is all 400 tags in the semvar format. For readability I removed them from the job output.

@baumac
Copy link
Contributor Author

baumac commented May 19, 2023

@sbe-arg thanks for taking a look at this so quickly, it's very much appreciated 🙇

@sbe-arg
Copy link
Collaborator

sbe-arg commented May 19, 2023

Ill try to debug further next week, need to simulate 500 tags or so to get a realistic scenario XD

@baumac
Copy link
Contributor Author

baumac commented May 19, 2023

@sbe-arg I setup a repo with 998 tags to repro the error. Note, I also forked the action, updated it to not exit immediately on piperrors, and to log the pipe exit status for the command causing the error.

Test repo with 998 tags: https://github.com/baumac/actions-test-repo
github-tag-action fork: https://github.com/baumac/github-tag-action/tree/1.65.1

Here's a GH actions run that produces the Pipefailure in the logs: https://github.com/baumac/actions-test-repo/actions/runs/5024209249/jobs/9009683359

Relevant snippet:

tagFmt='^v?[0-9]+\.[0-9]+\.[0-9]+$'
+ preTagFmt='^v?[0-9]+\.[0-9]+\.[0-9]+(-beta\.[0-9]+)$'
+ git for-each-ref --sort=-v:refname --format '%(refname:lstrip=2)'
+ head -n 1
+ grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+$'
+ true
+ printf '%s\n' 141 141 0 0

@sbe-arg
Copy link
Collaborator

sbe-arg commented May 19, 2023

Maybe add a sleep between tag and pretag greps to make sure is happening in tag.

But we can probably do a $? 141 || true as dirty fix to ignore sigpipe

@sbe-arg
Copy link
Collaborator

sbe-arg commented May 20, 2023

Another simple way to fix this is split the tag pretag one lines into 2 functions

Where
1- get tags into a var
2- once tags are in var, grep
3- once grep finish, head

Because pipe with grep is making the git command to still piping info when grep already finishes with 0 so the pipe returns a 141

@baumac
Copy link
Contributor Author

baumac commented May 20, 2023

Created #270 which implements your suggested fix of using vars.

Was able to test it out, and everything appears to be working 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants