Track changes to robots.txt

[kornelski]

Currently robots.txt of doc.rust-lang.org is not part of any repo, so there's no way to contribute any changes to it, such as needed for #44894 and countless dupes of this issue.

I propose adding it to this repo. I'm not in control of the infrastructure, so I can't help to automate deployment of it, but even just having the file under source control is IMHO a step forward.

[rust-highfive]

r? @QuietMisdreavus

(rust_highfive has picked a reviewer for you, use r? to override)

[QuietMisdreavus]

cc @rust-lang/infra in case this requires a change to RCS. Does the doc publishing code grab everything in the doc output? (Does bootstrap require a change to copy this file into the output when x.py doc is run?)

[QuietMisdreavus]

I also want to rope in @rust-lang/release since this will add another step to the stable release process. We probably want to include all the old versions in this listing, right? That will mean we need to keep it updated when a new version is released.

[QuietMisdreavus]

In general, i'm in favor of adding this to the repo, so if the teams i tagged are okay with whatever process changes are needed, r=me.

[kornelski]

Right, I do plan to add more entries to the robots.txt ASAP, but for this issue I've copied exactly what's live to discuss further changes in a separate issue.

[pietroalbini]

We probably want to include all the old versions in this listing, right? That will mean we need to keep it updated when a new version is released.

I played around with Google's validator (requires a Google account) and we could use wildcards to automatically include every release without having to update the file manually:

User-Agent: *
Disallow: /0.*
Disallow: /1.*

That would be great from a release team point of view.

[pietroalbini]

Does the doc publishing code grab everything in the doc output?

Yep, RCS copies all the contents of the tarball in the S3 bucket.

Does bootstrap require a change to copy this file into the output when x.py doc is run?

I think you need to copy the file here, but I'm not too familiar with rustbuild. Anyway, I'd prefer to land this only when it's automatically synchronized with the website.

[pietroalbini]

Discussed this at the infra team meeting, there should be no problem with deploying that file from the repo.

[steveklabnik]

So, one small thing I'm worried about here: does this mean we lose all of the google juice we've built up over the years? This just disallows the old pages, but doesn't redirect them to the new ones, right?

[kornelski]

This PR only commits to the repo what's already live, so it doesn't affect SEO.

As for excluding old docs in the future, I expect it to be beneficial for SEO, but perhaps that should be discussed in depth in a separate issue?

[Mark-Simulacrum]

r? @Mark-Simulacrum

Going to re-assign this to myself -- I'll try to check the dist process and make sure this doesn't cause any problems for us. Should hopefully have some time later this week.

[Mark-Simulacrum]

@bors r+

I think this should be fine; if not we can always revert.

[bors]

📌 Commit fbc4326 has been approved by Mark-Simulacrum