feat(vcs): add new git repo scanning method to improve resolution speed #4642
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@edvald is this ready for review? There are a few test failures. |
|
I need to look at those, pushed just last night after working on this on my SF flight. I'd say it's ready for preliminary review maybe. |
edvald
force-pushed
the
git-repo-scan
branch
2 times, most recently
from
72e8734 to
a8f5255
Compare
thsig
reviewed
Jun 18, 2023
edvald
force-pushed
the
git-repo-scan
branch
3 times, most recently
from
4eb4a68 to
1b7bdc6
Compare
This adds a new experimental git repo scanning method, enabled by setting `scan.git.mode: repo` in the project config or by setting `GARDEN_GIT_SCAN_MODE=repo`. It can still do with quite a bit of optimization but the general idea is to do far fewer (albeit generally more expensive) calls to git. For now I don't think it makes sense to make this the default since there are pros and cons to either approach, but for large repositories with many actions/modules this is likely to be more efficient. I suggest we collect user feedback and do IRL tests before diving in and optimizing further. One obvious optimization will be to index the repository files more efficiently, instead of just filtering on a straight list of paths. It would also make sense to delay hashing files that aren't in the index until they are called upon.
|
I've rebased the PR. @thsig can we get this merged please? It does include performance optimizations even for the default scanning method. We can always iterate on the newer experimental version. |
thsig
approved these changes
Jul 6, 2023
|
Yeah, sounds good. I've just approved it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a new experimental git repo scanning method, enabled by setting
scan.git.mode: repoin the project config or by settingGARDEN_GIT_SCAN_MODE=repo.It can still do with quite a bit of optimization but the general idea is to
do far fewer (albeit generally more expensive) calls to git.
For now I don't think it makes sense to make this the default since there are
pros and cons to either approach, but for large repositories with many
actions/modules this is likely to be more efficient.
I suggest we collect user feedback and do IRL tests before diving in and
optimizing further. One obvious optimization will be to index the repository
files more efficiently, instead of just filtering on a straight list of paths.
It would also make sense to delay hashing files that aren't in the index until
they are called upon.
This is the first step to resolve #3452