Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add caching support to verify-links #1562

Merged
1 commit merged into from
Apr 22, 2021
Merged

Conversation

weshaggard
Copy link
Member

@weshaggard weshaggard commented Apr 15, 2021

  • Update link checking pipeline to enable caching
  • Add loading cache file from a http endpoint
  • Add retry logic when hitting 429

Fixes most of Azure/azure-sdk#2403
Fixes #1439

@weshaggard weshaggard requested a review from sima-zhu April 15, 2021 22:24
@weshaggard weshaggard requested a review from a team as a code owner April 15, 2021 22:24
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@mitchdenny
Copy link
Contributor

No issues with this code. But I've been wondering whether we can have a more pluggable approach to these kinds of checks. Instead of plugging them into the pipelines themselves, we create a separate pipeline that runs the integrations indirectly. The rules for triggering the pipeline would be:

  1. Have any of the files that match a glob filter (from config file) changed?
  2. Has a parent commit had the pipeline run (successfully) within specified (configured) time period

I think this file matching/commit history aware system could also be used for kinds of validations that we want to run as well.

@weshaggard
Copy link
Member Author

I assume you are talking about the github.io link checker. I've generally when with scheduled builds for these type of type of things as that is the best option we have short of writing our own scheduler/triggering mechanism. I know we have skirted around that issue for a while and I'm not sure this is enough of a reason to push us over the edge. That said if you want to file an issue on the backlog to track that I'm not opposed to the idea we just need to figure out how we would integrate it.

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

- Update link checking pipeline to enable caching
- Add loading cache file from a http endpoint
- Add retry logic when hitting 429

Fix verify link renaming scenarios

- Correctly error if link format is incorrect
- Handle case where the replace regex is not passed
- Improve the error logging so it is easier to
  identify the broken links
@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@ghost
Copy link

ghost commented Apr 22, 2021

Hello @azure-sdk!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@ghost ghost merged commit ad902df into Azure:master Apr 22, 2021
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Link verification check error message is confusing
3 participants