Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use threads, cache Scala compiler in GH mvn workflow #9925

Merged
merged 20 commits into from
Dec 28, 2023
Merged

Use threads, cache Scala compiler in GH mvn workflow #9925

merged 20 commits into from
Dec 28, 2023

Conversation

gerashegalov
Copy link
Collaborator

@gerashegalov gerashegalov commented Dec 1, 2023 •
edited
Loading

  • Save the overhead of setting up the runner by folding the short-lived runner get-shim-versions-from-dist steps into cache-dependencies job
  • Add restore-keys to make sure cache-dependencies does not start on an empty m2, stable versions like for spark artifacts do not change
  • Use de.qaware.maven:go-offline-maven-plugin suggested by @revans2
  • Save scala compiler bridge into a cacheable location

Previous mvn workflow duration:

After this PR:

  • on cache hit: 5min
  • on cache miss: 7min

@gerashegalov gerashegalov changed the title [WIP] Use multi-threaded maven and go-offline-maven-plugin in GH work… [WIP] Use threads and go-offline-maven-plugin in GH mvn workflow Dec 1, 2023
@gerashegalov gerashegalov changed the title [WIP] Use threads and go-offline-maven-plugin in GH mvn workflow [WIP DON'T REVIEW] Use threads and go-offline-maven-plugin in GH mvn workflow Dec 1, 2023
@gerashegalov gerashegalov changed the title [WIP DON'T REVIEW] Use threads and go-offline-maven-plugin in GH mvn workflow [WIP DON'T REVIEW] Use threads and and cache Scala compiler in mvn workflow Dec 1, 2023
@gerashegalov gerashegalov marked this pull request as ready for review December 2, 2023 21:58
@gerashegalov gerashegalov changed the title [WIP DON'T REVIEW] Use threads and and cache Scala compiler in mvn workflow Use threads, cache Scala compiler in GH mvn workflow Dec 2, 2023
@gerashegalov
Copy link
Collaborator Author

build

YanxuanLiu
YanxuanLiu previously approved these changes Dec 4, 2023
View reviewed changes
Copy link
Collaborator

@YanxuanLiu YanxuanLiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

jlowe
jlowe reviewed Dec 4, 2023
View reviewed changes
integration_tests/run_pyspark_from_build.sh Outdated Show resolved Hide resolved
jdk-profiles/pom.xml Outdated Show resolved Hide resolved
@sameerz sameerz added the build Related to CI / CD or cleanly building label Dec 5, 2023
@gerashegalov gerashegalov self-assigned this Dec 11, 2023
@gerashegalov gerashegalov requested a review from jlowe December 11, 2023 19:13
@gerashegalov
review: single for loop
711a723 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov gerashegalov requested a review from jlowe December 27, 2023 22:05
@gerashegalov
Copy link
Collaborator Author

build

NvTimLiu
NvTimLiu approved these changes Dec 28, 2023
View reviewed changes
Copy link
Collaborator

@NvTimLiu NvTimLiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gerashegalov gerashegalov merged commit 30e20b9 into NVIDIA:branch-24.02 Dec 28, 2023
38 checks passed
@gerashegalov gerashegalov deleted the t1cMvnWorkflow branch December 28, 2023 16:16
@gerashegalov gerashegalov mentioned this pull request Dec 28, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlowe jlowe jlowe approved these changes

@NvTimLiu NvTimLiu NvTimLiu approved these changes

@YanxuanLiu YanxuanLiu YanxuanLiu left review comments

@revans2 revans2 Awaiting requested review from revans2 revans2 is a code owner

@tgravescs tgravescs Awaiting requested review from tgravescs tgravescs is a code owner

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@pxLi pxLi Awaiting requested review from pxLi

Assignees

@gerashegalov gerashegalov

Labels
build Related to CI / CD or cleanly building
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@gerashegalov @jlowe @NvTimLiu @YanxuanLiu @sameerz