Refactor Jenkins pipeline #2005
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Remove usage of parallel in the build and package stages - Retain usage of parallel in the deploy stages - Separate build and package into separate scripts to support build-only stages - Add a build-only stage for dev/feature branches
melange396
approved these changes
Jul 29, 2024
There was a problem hiding this comment.
The deploy scripts are safe to keep parallel?
Do you wanna also remove the old https://github.com/cmu-delphi/covidcast-indicators/blob/main/jenkins/build-and-package.sh script?
Otherwise, LGTM!
|
Yeah, re deploy stages: I'd considered changing them as well, but thought we could see how they fare. They do have some infrequent issues, but they are typically minor and iirc always resolve on a retry, so maybe we keep 'em? I'll keep an eye out for failure. The build stage is the one that, at least in the context of this latest issue we ran into, would not resolve on any subsequent retry and is most necessary to address to unblock work. I'll yank out that original script as well 👍 |
minhkhul
approved these changes
Jul 29, 2024
This was referenced Sep 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Bug fix and feature
(This Slack thread is the starting point for this work.)
Main change:
Make the pipeline less likely to produce hard to debug errors by removing usage of
parallel. Though not 100% diagnosed, it seems like parallel can lead to some strange behavior that manifests as files and programs not being found, or Python package installation steps failing, etc. These are transient and often go away after a retry, but they are confusing and sometimes send people on unnecessary troubleshooting exercises.Ex issue:
A potentially significant downside will be the extra time the pipeline will take to complete. This won't be too bad for now, but time will scale with the number of indicators. There will be additional work done to migrate the pipeline to GH Actions in the short to medium term, where we should be able to run in parallel again.
Additional change(s):
Run the build script on dev/feature branches. This is useful to catch issues before the necessity of merging to
main.Changelog