Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add robots.txt #904

Merged
merged 3 commits into from
Dec 7, 2021
Merged

add robots.txt #904

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b6ffb96
init of robots.txt
geoffcline Dec 3, 2021
980737b
fixup
geoffcline Dec 3, 2021
69fa393
add index to robots.txt
geoffcline Dec 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions website/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ baseURL = "/"
disableKinds = ["taxonomy", "taxonomyTerm"]

# Language settings
enableRobotsTXT = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean the robots.txt that's committed was generated by Hugo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no. it just means that the committed robots.txt makes it into the live site.

contentDir = "content/en"
defaultContentLanguage = "en"
defaultContentLanguageInSubdir = false
Expand Down
8 changes: 8 additions & 0 deletions website/layouts/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
User-agent: *

Disallow: /v0.4.3-docs/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to get stale. We need a longer term solution to avoid updating this on every release. Perhaps we can simply allow /docs.


Allow: /docs/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this also include the homepage?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've read https://developers.google.com/search/docs/advanced/robots/robots_txt and don't really understand allow. it just seems to say google may index it, but it's also seems that everything not explictly disallowed is indexed.

Allow: /index.html

SITEMAP: https://karpenter.sh/sitemap.xml