Bluesky link previews should honor "robots" meta tags (or headers) in posted URLs

[mfb]

Steps to Reproduce

Find a URL with a robots meta tag (or header) that rejects robots, for example, <meta name="ROBOTS" content="NOINDEX, NOFOLLOW, NOARCHIVE">

In this case, when the URL is posted on Bluesky, I would propose that Bluesky not mirror/preview the link content into the post.

The reason for this is that a site may use such meta tags to mark content that is currently public, but not yet fully reviewed. Therefore it would be a good idea to not mirror the content, in case the content contains misinformation, spam, etc.

Attachments

No response

What platform(s) does this occur on?

Web (Desktop)

Device Info

No response

What version of the app are you using?

1.94.0

Additional Information

No response