Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

split change detector action into a separate job, or a separate workflow completely #2736

Closed
shahzadlone opened this issue Jun 17, 2024 · 0 comments · Fixed by #2789
Closed

split change detector action into a separate job, or a separate workflow completely #2736

shahzadlone opened this issue Jun 17, 2024 · 0 comments · Fixed by #2789
Labels
ci/build This is issue is about the build or CI system, and the administration of it. code quality Related to improving code quality

Comments

@shahzadlone
Copy link
Member

shahzadlone commented Jun 17, 2024 •
edited
Loading

Split change detector action into a separate job, or a separate workflow completely main due to the following reasons:

  • the coverage report job does not need to wait for change detector (which takes much longer, and doesn't utilize caching), currently it is blocked when change detector is running.
  • extra if statements in the test-and-upload-coverage.yml workflow for a different flow (no code coverage steps).
  • test-and-upload-coverage.yml is getting more complex and caching stuff doesn't apply to change detector either.
  • optimize change detector when it is it's own job or workflow (perhaps using caching in a different way).
@shahzadlone shahzadlone added ci/build This is issue is about the build or CI system, and the administration of it. code quality Related to improving code quality labels Jun 17, 2024
@shahzadlone shahzadlone mentioned this issue Jun 17, 2024
Merged
shahzadlone added a commit that referenced this issue Jun 18, 2024
@shahzadlone
ci: Cache dependencies to speed up test runs (#2732)
3c0a14a 
## Relevant issue(s)
Resolves #2735 

## Description
In this PR we cache go and rust build/dep states, and restore them if an
action run has same dependency graph. Go cache is around 877 MB and
cargo/rust cache is about 189MB (99MB for macos) so we can roughly have
room for 5 caches (x1 for linux and x1 for macos) before a cache is
dropped / lost.

Notes:
- Github gives us 10GB of cache storage by default.
- Rather than going with the standard practice of using lockfiles to
generate the caching key we will use the mod and toml files, in order to
maximize cache hits. Incase troublesome we can go back to lockfiles.
- This change does not help the change detector. IMO change detector
should move to a separate job, or a new workflow file. (opened
#2736)


### Demo
- Without cache run (current state):
https://github.com/sourcenetwork/defradb/actions/runs/9554784255?pr=2732
<img
src="https://github.com/sourcenetwork/defradb/assets/30120428/c0b7720a-bd3f-4a01-853f-43d7b135ab76"
width="400" />


- With cache run (after this PR):
https://github.com/sourcenetwork/defradb/actions/runs/9555174699
<img
src="https://github.com/sourcenetwork/defradb/assets/30120428/db9d0bab-7fcb-4255-99f1-bd1d83220795"
width="400" />




## How has this been tested?
- Manual
- `act` tool
@shahzadlone shahzadlone mentioned this issue Jun 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ci/build This is issue is about the build or CI system, and the administration of it. code quality Related to improving code quality
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ci(i): Make change detector an independent workflow shahzadlone/defradb
1 participant
@shahzadlone