Adding links from crates.io yields an error

[JeffResc]

Describe the bug
Hi, thanks for the great project! Upon adding various links from crates.io, I have been receiving the error message The Link was added but an error occurred when trying to request the URL, for example an invalid certificate. Details can be found in the logs. I checked the logs and they say https://crates.io: https://crates.io is not reachable. HTTP request returned status code 404:. This is a bug because the page https://crates.io is not a 404.

To Reproduce
Steps to reproduce the behavior:

1. Find a link on crates.io, for example the homepage: https://crates.io
2. Add the link to LinkAce using the "Quick Add Link" feature on the homepage or otherwise
3. The error message will appear: The Link was added but an error occurred when trying to request the URL, for example an invalid certificate. Details can be found in the logs.

Expected behavior
The link should be added normally as its status is 200.

Screenshots

LinkAce setup (please complete the following information):

• Version: v1.10.1
• Installed via: Docker (Advanced)
• OS: Docker 20.10.14 running on Ubuntu 22.04 (Jammy Jellyfish) LTS in an LXC container on Debian GNU/Linux 11 (bullseye)

Desktop (please complete the following information if applicable):

• OS: Windows 11 Version 21H2
• Browser: Google Chrome Version 100.0.4896.127

Additional context
I can add links from other websites and that works as expected. I'm not sure if this error only applies to crates.io or if bug applies to other sites as well, but I figured I would submit it just to bring awareness.

[Kovah]

There's nothing I can do about that.
Crates seem to block all requests that are not coming from a real browser.
curl https://crates.io/crates/libc also returns a 404. That's stupid, because it likely breaks a lot of applications, but it's entirely the choice of the site maintainers.

You have the option to set a custom user agent (you can try that of your browser, for example), but chances are high that crates is still blocking anything from LinkAce.

Similar issue: #417

[JeffResc]

curl https://crates.io/crates/libc also returns a 404

I didn't even think to check running a cURL on it. That's unfortunate, but thanks for looking into this for me. Keep up the great work on this project 😄

[JeffResc]

Hi @Kovah —

I've found a solution as described in rust-lang/crates.io#788. One must use the Accept: text/html HTTP header in order to access the site. I don't see this type of customization available anywhere in the docs. Is this a planned feature or something you would be willing to add if a PR was made?

[Kovah]

I guess that's a reasonable addition. I will add it to the library which is used to generate the meta.
See Kovah/laravel-html-meta#3

[Kovah]

Should be fixed with new version 1.10.2 as it contains the small improvement.