400 Bad Response

[aclevername]

Hello!

Our team has started to notice failures when checking links on twitter.com. E.g.:

        ERROR   https://twitter.com/ashleymcnamara
                Bad Request (HTTP error 400)

When I access the above link in my browser it appears to be fine. Doing some research this might be related to legacy twitter being removed https://screenrant.com/twitter-legacy-nintendo-3ds-shut-down-date-december-2020/

Thanks,
Jake & @Callisto13

[MichaIng]

Same here. It's a Twitter internal DDoS protection or similar. Other hosts cause similar issues: #42

It is a natural issue when checking many links concurrently and/or shortly after another to the same host. The only solution I can think of is to allow defining not only the number of concurrent connections but also a maximum number of connections to the same host in a certain time range, so that liche waits for the timeout before continuing to check link with that host. But this is quite a larger task, I guess, especially since it needs to be configured on a per-host level to match the individual limits.

[aclevername]

Same here. It's a Twitter internal DDoS protection or similar. Other hosts cause similar issues: #42

It is a natural issue when checking many links concurrently and/or shortly after another to the same host. The only solution I can think of is to allow defining not only the number of concurrent connections but also a maximum number of connections to the same host in a certain time range, so that liche waits for the timeout before continuing to check link with that host. But this is quite a larger task, I guess, especially since it needs to be configured on a per-host level to match the individual limits.

I'm not sure if its related to that issue, I think its more likely caused by this https://news.ycombinator.com/item?id=25464280

[MichaIng]

Ah that is true. I assumed to quickly since Twitter was one of the hosts that often show up alone in list of failed links, but with a different error response indeed. Okay that 400 is new and can be replicated via curl --head https://twitter.com and curl https://twitter.com. Then it needs to be excluded for now.

[raviqqe]

Sorry, I decided to deprecate this project! Please consider using the alternatives listed there. Thank you all for your cooperation!

https://github.com/raviqqe/liche#deprecation-notice

[MichaIng]

Sad to hear. Nobody (okay I can only speak for myself) is expecting a tool to automatically handle all kind of website rate limitations, other anomalies etc. E.g. while it is indeed nice to have even GitHub token support in lychee, I would never add it to my own tool (if I would develop one) as finally users or myself would expect to implement individual handlers and options for every larger website, total insane IMO. All that is needed to address the issues/reasons you mention is some flexibility to customise which response code is handled how. But probably that is more work than I imagine and of course I respect the decision and with all the best, started with some nice holidays. Thanks for your great work that helped us detect a lot of broken URLs by now 👍 😃.