Markdown Link check is failing for valid URLs

[dibyom]

Expected Behavior

Markdown link check is should pass!

Actual Behavior

Markdown link check is failing for valid URLs e.g. tektoncd/triggers#273
I can reproduce this locally from the Triggers repo.

Steps to Reproduce the Problem

liche -d $(pwd) -v $(find docs/ -name *.md)

Additional Info

Failing URLs:

tektoncd/triggers#275

I1212 16:03:05.422] 	ERROR	https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#events
I1212 16:03:05.422] 		Not Found (HTTP error 404)
I1212 16:03:05.422] 	ERROR	https://docs.gitlab.com/ee/user/project/integrations/webhooks.html/
I1212 16:03:05.422] 		Not Found (HTTP error 404)

tektoncd/pipeline#1736

I1212 08:38:19.837] docs/tasks.md
I1212 08:38:19.837] 	ERROR	https://en.wikipedia.org/wiki/Shebang_(Unix
I1212 08:38:19.838] 		Not Found (HTTP error 404)

tektoncd/pipeline#1696

I1211 16:59:30.399] ---- Checking links in the markdown files ----
I1211 16:59:30.399] ----------------------------------------------
I1211 16:59:32.799] docs/resources.md
I1211 16:59:32.800] 	ERROR	https://godoc.org/github.com/jenkins-x/go-scm/scm#State
I1211 16:59:32.800] 		Not Found (HTTP error 404)

[bobcatfish]

Is it possible the URLs are sometimes down? Maybe we can make a list of the URLs we are seeing fail.

Ideas:

• could make the markdown lint checker more tolerant (e.g. retry)
• make the markdown lint check not required

[dibyom]

Maybe related: raviqqe/liche#24
Does not explain why https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#events is failing though

[dibyom]

Problem might be in fasthttp

https://gist.github.com/dibyom/e546f71828d2d0886fc9c14b639f535d

[vdemeester]

Honestly, I would like to limit the check to internal files as it's way more useful than old files 👼

[dibyom]

Yeah, I don't think checking links only for changed markdown files on PRs is super useful -- if a markdown file is not touched for months, and in that time any of the links go stale, we would not catch it unless someone modifies the file.
Running this tool locally though, I did discover quite a few URLs that currently 404 or end with a . making them invalid.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.