Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to remove the deb-s3 lockfile if the command fails #6652

Merged
merged 5 commits into from
Nov 21, 2020
Merged

Attempt to remove the deb-s3 lockfile if the command fails #6652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9cb9194
Add script to remove deb-s3 lockfile if it is > 5 mins old
mrmr1993 Nov 11, 2020
f295de7
Use clear-deb-s3-lockfile.sh script to try releasing the lockfile
mrmr1993 Nov 11, 2020
24782bd
Merge branch 'develop' into feature/try-purge-s3-lockfile
mrmr1993 Nov 11, 2020
2c4f182
Merge branch 'develop' into feature/try-purge-s3-lockfile
lk86 Nov 20, 2020
a9c5cdf
Merge branch 'develop' into feature/try-purge-s3-lockfile
mergify[bot] Nov 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion buildkite/scripts/publish-deb.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,11 @@ esac

echo "Publishing debs: ${DEBS}"
set -x
${DEBS3} --codename "${CODENAME}" "${DEBS}"
# Upload the deb files to s3.
# If this fails, attempt to remove the lockfile and retry.
${DEBS3} --codename "${CODENAME}" "${DEBS}" \
|| ( scripts/clear-deb-s3-lockfile.sh \
&& ${DEBS3} --codename "${CODENAME}" "${DEBS}")
set +x
echo "Exporting Variables: "

Expand Down
6 changes: 5 additions & 1 deletion scripts/archive/build-release-archives.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,11 @@ else
echo "Publishing debs:"
ls mina-*.deb
set -x
${DEBS3} --codename ${CODENAME} --component main mina-*.deb
# Upload the deb files to s3.
# If this fails, attempt to remove the lockfile and retry.
${DEBS3} --codename ${CODENAME} --component main mina-*.deb \
|| ( scripts/clear-deb-s3-lockfile.sh \
&& ${DEBS3} --codename main mina-*.deb)
fi

###
Expand Down
18 changes: 18 additions & 0 deletions scripts/clear-deb-s3-lockfile.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/bin/bash

S3_LOCKFILE_DATE="$(aws s3 ls s3://packages.o1test.net/dists/unstable/main/binary-/lockfile | awk '{print $1 " " $2}')"
if [[ "$S3_LOCKFILE_DATE" == "" ]]; then
echo "Could not get the lockfile timestamp from the S3 bucket. Have you set your AWS credentials correctly?"
exit 1
fi
S3_LOCKFILE_SECONDS=$(date -d "$S3_LOCKFILE_DATE" +%s)
NOW_SECONDS=$(date -d +%s)
TIME_DIFF=$(($NOW_SECONDS - $S3_LOCKFILE_SECONDS))
if [[ $TIME_DIFF > 300 ]]; then
echo "Lockfile has been held for > 5 mins. The deb-s3 instance is likely to have died. Deleting lockfile.."
aws s3 rm s3://packages.o1test.net/dists/unstable/main/binary-/lockfile
echo "Lockfile deleted"
else
echo "Lockfile is younger than 5 mins. There may be a deb-s3 instance actively using it. Refusing to delete."
exit 1
fi
6 changes: 5 additions & 1 deletion scripts/publish-deb.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,11 @@ case "$CIRCLE_JOB" in
build-artifacts--testnet_postake_medium_curves | FORCED)
echo "Publishing debs: ${DEBS}"
set -x
${DEBS3} --codename "${CODENAME}" "${DEBS}"
# Upload the deb files to s3.
# If this fails, attempt to remove the lockfile and retry.
${DEBS3} --codename "${CODENAME}" "${DEBS}" \
|| ( scripts/clear-deb-s3-lockfile.sh \
&& ${DEBS3} --codename "${CODENAME}" "${DEBS}")
echo "Exporting Variables: "
# Export Variables for Downstream Steps
echo "export CODA_DEB_REPO=$CODENAME" >> /tmp/DOCKER_DEPLOY_ENV
Expand Down