Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: dynamic jobs #124175

Merged
merged 8 commits into from
Apr 24, 2024
Merged

CI: dynamic jobs #124175

merged 8 commits into from
Apr 24, 2024

Conversation

Kobzol
Copy link
Contributor

@Kobzol Kobzol commented Apr 19, 2024 •
edited
Loading

This PR modifies our CI workflows to be dynamic. This means that when a GitHub event is generated, we will run a Python script (calculate-job-matrix.py), which decides which CI jobs should be generated. These jobs are defined in src/ci/github-actions/jobs.yml).

This should provide a few benefits:

  • Once the migration to dynamic jobs is complete, we shouldn't need expand-yaml-anchors anymore.
  • The job table on PRs (and also the left job column on auto/try builds) should be much cleaner and contain only the jobs that are actually relevant/executed.
  • It should be much easier to support dynamic try builds, i.e. to run an arbitrary CI job on a try build.

See this Zulip discussion for more context.

r? @ghost

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Apr 19, 2024
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 3 times, most recently from b325a04 to 0e9eed1 Compare April 19, 2024 19:56
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 3 times, most recently from 55550c7 to 476d4ae Compare April 19, 2024 20:45
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 4 times, most recently from 65141ab to 8b945c3 Compare April 19, 2024 21:19
@Kobzol
Copy link
Contributor Author

Kobzol commented Apr 19, 2024

@bors try

bors added a commit to rust-lang-ci/rust that referenced this pull request Apr 19, 2024
@bors
Auto merge of rust-lang#124175 - Kobzol:ci-dynamic-job, r=<try>
5b35171 
CI: dynamic jobs

Opening as a draft to experiment with [dynamic CI jobs](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/job.20matrix.20re-ordered.20PR.20list).

r? `@ghost`
@bors
Copy link
Contributor

bors commented Apr 19, 2024

⌛ Trying commit 8b945c3 with merge 5b35171...

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from 97ca4d3 to 8f21998 Compare April 19, 2024 21:33
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@bors
Copy link
Contributor

bors commented Apr 20, 2024

☔ The latest upstream changes (presumably #124177) made this pull request unmergeable. Please resolve the merge conflicts.

@Kobzol Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from 0efaec5 to 545c12f Compare April 20, 2024 07:18
@Kobzol Kobzol force-pushed the ci-dynamic-job branch 2 times, most recently from dc3c544 to 4d3e89c Compare April 20, 2024 07:45
pietroalbini
pietroalbini reviewed Apr 22, 2024
View reviewed changes
Copy link
Member

@pietroalbini pietroalbini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few minor nits, but this otherwise looks great!

src/ci/github-actions/jobs.yml Outdated Show resolved Hide resolved
src/ci/github-actions/calculate-job-matrix.py Outdated Show resolved Hide resolved
src/ci/github-actions/jobs.yml Outdated Show resolved Hide resolved
src/ci/github-actions/ci.yml Show resolved Hide resolved
@Kobzol
Copy link
Contributor Author

Kobzol commented Apr 23, 2024

Thanks for the review, tried to fix all remarks in separate commits.

@pietroalbini
Copy link
Member

@bors r+

@bors
Copy link
Contributor

bors commented Apr 23, 2024

📌 Commit 2632c61 has been approved by pietroalbini

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 23, 2024
@Kobzol
Copy link
Contributor Author

Kobzol commented Apr 23, 2024

@bors rollup=never

(Who knows what this might do in the auto build).

RalfJung
RalfJung reviewed Apr 23, 2024
View reviewed changes
src/ci/github-actions/jobs.yml
<<: *job-linux-16c

# Jobs that run when you perform a try build (@bors try)
# These jobs automatically inherit envs.production, to avoid repeating
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't they inherit envs.try? (And similar for auto below)

Copy link
Contributor Author

@Kobzol Kobzol Apr 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point, I forgot to update the comment. I'll add it to a follow up PR.

@bors
Copy link
Contributor

bors commented Apr 24, 2024

⌛ Testing commit 2632c61 with merge e7da0fa...

@bors
Copy link
Contributor

bors commented Apr 24, 2024

☀️ Test successful - checks-actions
Approved by: pietroalbini
Pushing e7da0fa to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 24, 2024
@bors bors merged commit e7da0fa into rust-lang:master Apr 24, 2024
11 checks passed
@rustbot rustbot added this to the 1.79.0 milestone Apr 24, 2024
@bors bors mentioned this pull request Apr 24, 2024
Closed
@Kobzol Kobzol deleted the ci-dynamic-job branch April 24, 2024 09:07
@Kobzol
Copy link
Contributor Author

Kobzol commented Apr 24, 2024

Wow, it went through. Great.

This was referenced Apr 24, 2024
Merged
Closed
Merged
Closed
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (e7da0fa): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 673.216s -> 673.226s (0.00%)
Artifact size: 315.45 MiB -> 315.42 MiB (-0.01%)

@Zalathar Zalathar mentioned this pull request May 3, 2024
Merged
@Kobzol Kobzol mentioned this pull request May 6, 2024
Merged
@Kobzol Kobzol mentioned this pull request May 24, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pietroalbini pietroalbini pietroalbini left review comments

@RalfJung RalfJung RalfJung left review comments

Assignees

@pietroalbini pietroalbini

Labels
A-testsuite Area: The testsuite used to check the correctness of rustc merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
1.79.0
Development

Successfully merging this pull request may close these issues.
7 participants
@Kobzol @rust-log-analyzer @bors @pietroalbini @rust-timer @RalfJung @rustbot