Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sparse checkout docs repository in release step #1585

Merged
3 commits merged into from
May 18, 2021
Merged

Sparse checkout docs repository in release step #1585

3 commits merged into from
May 18, 2021

Conversation

benbp
Copy link
Member

@benbp benbp commented Apr 29, 2021

In our release stages, we do a full git clone of the docs repo. This can add up to 4 minutes of time to the release stage. Checking out only the files we need reduces that time to around 10 seconds.

@benbp benbp added the Central-EngSys This issue is owned by the Engineering System team. label Apr 29, 2021
@benbp benbp self-assigned this Apr 29, 2021
@benbp benbp requested a review from a team as a code owner April 29, 2021 20:49
@benbp benbp requested review from weshaggard and scbedd April 29, 2021 20:50
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp benbp requested a review from weshaggard May 4, 2021 20:11
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp benbp requested a review from weshaggard May 6, 2021 00:11
- pwsh: git checkout ${{ repo.Commitish }}
displayName: Sparse checkout at ${{ repo.Commitish }}
- pwsh: |
git checkout ${{ repo.Commitish }} # this will use the default branch if repo.Commitish is empty
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does it pick-up the default branch? Are you sure git checkout without parameters checks out the default branch? As best I can tell it will be a noop.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my testing (local and in-pipeline), it does. According to the docs:

You could omit , in which case the command degenerates to "check out the current branch", which is a glorified no-op with rather expensive side-effects to show only the tracking information, if exists, for the current branch.

Since we're in a state where sparse checkout paths are set but no blobs are downloaded, this seems to have the effect of grabbing all the relevant blobs.

I could switch it back to using the regex that we're using elsewhere, but as this seems to work I thought it was much cleaner.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this works than great. If not I was think we could use the default branch template and the $(DefaultBranch) variable in this place.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually only discovered this because I copied that DefaultBranch code over and forgot to change the remote from Source to origin, so it failed to set the variable and just ran git checkout, which worked!

displayName: Clone Documentation Repository
ignoreLASTEXITCODE: false

- ${{ if not(parameters.SparseCheckoutPaths) }}:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely like this approach better as it should allow both scenarios to work. It might actually be interesting make this an option in the sparse-checkout template itself if we find more patterns like this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah you mean perhaps replacing all our checkouts with calls to some checkout template, and we can have a parameter like sparse or full? I initially tried doing a sparse checkout with /* but that actually doubled the checkout time, hence why I went this route instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes having this split of full vs sparse clone might be a future improvement to consider.

Copy link
Member

@weshaggard weshaggard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On question about the default branch but otherwise looks good.

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@ghost
Copy link

ghost commented May 18, 2021

Hello @azure-sdk!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@ghost ghost merged commit 684233d into Azure:master May 18, 2021
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Central-EngSys This issue is owned by the Engineering System team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants