Add rustdoc option to make old docs "less important" to make search engine results more pertinent

[GuillaumeGomez]

I think about something like:

rustdoc --robots-options="less-important" # or whatever option it is for search engine robots

However, this option will need to be called for the version specific docs generation (I'm talking about doc.rust-lang.org/doc/1.x.y) and not on "not specific" docs ( doc.rust-lang.org/doc/stable, ``doc.rust-lang.org/doc/betaordoc.rust-lang.org/doc/nightly`). Maybe you know something about this @Mark-Simulacrum? (wild guess)

cc @QuietMisdreavus (does the option's name seems ok to you?)

[steveklabnik]

My understanding is that this would generate different HTML, meaning we'd have to ship new releases.

We already have a solution to this problem with #44687, IMO

[GuillaumeGomez]

We talked about it on IRC and apparently, a simple solution would be to just update the robots.txt once and for all.

[kornelski]

#68677