Block package docs from search-engine indexing

[cnunciato]

This change blocks all package docs from search-engine indexing by conditionally applying the block_external_search_index frontmatter property, rendering pages with a <meta name="robots" content="noindex,follow"> tag, which ensures they're crawled by search engines, but not indexed themselves.

Part of #3389.

[lukehoban]

Directionally this totally makes sense.

My short term worry is that these docs have quite strong SEO in many places. And I assume de-indexing them will just “throw away” that SEO value.

Is there anyway we can more gracefully “delegate” any index-strength from these pages to the new pages?

Relatedly - if we do make this change as is - can we measure the traffic impact this has (and revert if it’s significant?).

[github-actions]

Your site preview for commit 0d1bf72 is now available! 🍹

http://content-bucket-835a5f5.s3-website-us-east-1.amazonaws.com

Deploy previews powered by Pulumify

[cnunciato]

@lukehoban I believe our options are either noindex+follow, as in this PR, or redirect. And since this PR would remove the package docs from the index anyway, 301-redirecting might be the better option.

@zchase do you know of any other/better options?

[cnunciato]

@lukehoban after chatting with @zchase a bit out of band, we think the best thing we could do here is indeed 301 -- but obviously we aren't ready to do that yet, because there's still some content in package docs that we want users to be able to get to. If we merged this PR, dropping these pages from search-engine indexes (but not redirecting), we'll likely lose the SEO ranking of the package docs before being able to hand it over to the resource docs.

So we could prioritize moving toward getting the resource docs into shape to replace the package docs, and then 301 the package docs. (And so close this PR for now.) That'd mean leaving the package docs out there in the indexes for a bit longer, but with the new messaging at the top of those pages now, it might be fine.

Alternatively, we could just merge this PR, and work on making the resource docs as good as we can make them, knowing that over time, their ranking will improve. Eventually, we'll want to 301 regardless, but the resource docs are already fairly decently ranked for certain searches, it looks like -- from a cursory glance, within a few results down from the package docs, which is better than I'd have expected:

So I'd probably say let's just merge this and roll forward, if we agree on the ultimate aim.

[cnunciato]

Relatedly - if we do make this change as is - can we measure the traffic impact this has (and revert if it’s significant?).

We can and should measure the changes here, absolutely, and could certainly revert, although I can't think of a reason we'd want to do that, given where we're headed more generally.

[cnunciato]

How does everyone feel about this? I'd like to keep taking steps toward removing these docs if we can, so if there isn't strong opposition to delisting them, my preference would be to merge this. But if there is opposition, I want to be respectful of that, too. What do y'all think?

[praneetloke]

@cnunciato I would have not removed the indexing on these API docs just yet, just because it is unclear if the resource docs would take its place when that happens. I think the other PR where you added the notice at the top of the API docs to go to the resource docs was a good enough change for me for the time being. But I also don't know about all of our SEO efforts, and I have not participated in any of them before either.

As for this PR, to me it is clear that you think we should do this, and I don't have a strong reason to say otherwise, so I am good with it.

[justinvp]

So we could prioritize moving toward getting the resource docs into shape to replace the package docs, and then 301 the package docs. (And so close this PR for now.) That'd mean leaving the package docs out there in the indexes for a bit longer, but with the new messaging at the top of those pages now, it might be fine.

What work remains to get the resource docs into shape and how far away are we?

What is the long-term end-goal for these docs? When we start redirecting the existing language docs to the resource docs, does that mean we'll no longer have language-specific docs for Node.js and Python (for resource packages)? Does that mean we'd also get rid of the .NET docs? Go docs? Or would we aim to republish the Node.js and Python docs at some other location?

[cnunciato]

What is the long-term end-goal for these docs? When we start redirecting the existing language docs to the resource docs, does that mean we'll no longer have language-specific docs for Node.js and Python (for resource packages)? Does that mean we'd also get rid of the .NET docs? Go docs?

@justinvp The end-goal is to make the new docs as complete as we can (adding overlays, component resources, etc.) and have them become just "the docs" going forward, knowing many users will use their IDEs to get the kind of information provided by package docs today. So yes, it'd mean the language-specific Python and Node.js docs would go away, as would the .NET docs, to be replaced by the resource docs. There's more work to be done to get there, but that's the intention (unless of course there's a compelling use case for keeping them).

The Go docs would continue to be published to go.dev as they are today, though, since that process (based on my understanding, having spoken with Pat) is essentially automatic and also expected from the Go developer's perspective. We could publish the Node, Python and .NET docs to some other location, too, if that made sense -- but I'm not (at least not yet) hearing that that's something we should do, or that users of those languages would expect.

[cnunciato]

Remaining coverage gaps are collected here: #3528

[leezen]

Given where these rank SEO-wise, I think merging this as-is makes sense to me (vs. trying to do the 301).