Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a robots.txt file #293

Closed
AymenLoukil opened this issue Nov 1, 2019 · 4 comments · Fixed by #344
Closed

Create a robots.txt file #293

AymenLoukil opened this issue Nov 1, 2019 · 4 comments · Fixed by #344
Assignees
@rviscomi
Labels
development Building the Almanac tech stack SEO SEO related
Milestone
SHIP IT!

Comments

@AymenLoukil
Copy link
Contributor

We should create a robots.txt file

@AymenLoukil AymenLoukil added development Building the Almanac tech stack SEO SEO related labels Nov 1, 2019
@rviscomi rviscomi added this to the SHIP IT! milestone Nov 1, 2019
@rviscomi
Copy link
Member

rviscomi commented Nov 1, 2019

Can we add sitemap.xml to this issue or should that be done separately?

@rviscomi
Copy link
Member

rviscomi commented Nov 2, 2019

As far as what should go in the robots file, I don't think we want to disallow anyone from anywhere. So is it simply going to contain the following?

User-agent: *
Allow: /

@AymenLoukil
Copy link
Contributor Author

Can we add sitemap.xml to this issue or should that be done separately?

I think it should be done separately. When the sitemap will be ok we will modify the robots.txt to add its reference.

@AymenLoukil
Copy link
Contributor Author

As far as what should go in the robots file, I don't think we want to disallow anyone from anywhere. So is it simply going to contain the following?

User-agent: *
Allow: /

Yeah. For the moment we allow everything. Later we could adjust.

@tunetheweb tunetheweb mentioned this issue Nov 4, 2019
Closed
@rviscomi rviscomi self-assigned this Nov 6, 2019
@rviscomi rviscomi mentioned this issue Nov 6, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rviscomi rviscomi

Labels
development Building the Almanac tech stack SEO SEO related
Projects
None yet
Milestone
SHIP IT!
Development

Successfully merging a pull request may close this issue.

Serve a robots.txt file HTTPArchive/almanac.httparchive.org
2 participants
@rviscomi @AymenLoukil