Create a sitemap.xml file

[tunetheweb]

We should create a sitemap.xml file per supported language (e.g. https://almanac.httparchive.org/sitemap-en.xml for English) as part of the SEO recommendations discussed in #286.

This should be added to robots.txt being generated as part of #293 like so (multiple sitemaps are allowed):

User-agent: *
Allow: /
Sitemap: https://almanac.httparchive.org/sitemap-en.xml
Sitemap: https://almanac.httparchive.org/sitemap-jp.xml

This should be of the following format with one section per page for that language:

<urlset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
<url>
  <loc>https://almanac.httparchive.org/en/2019/</loc>
  <lastmod>2019-11-04</lastmod>
</url>
<url>
  <loc>https://almanac.httparchive.org/en/2019/table-of-contents</loc>
  <lastmod>2019-11-04</lastmod>
</url>
...etc. for rest of non-chapter pages
<url>
  <loc>https://almanac.httparchive.org/en/2019/markup</loc>
  <lastmod>2019-11-04</lastmod>
</url>
...etc. for rest of chapter pages
</urlset>

The lastmod date can be derived from the date discussed in #317 so this could be dynamically generated on the fly, or as part of npm run generate.

[AymenLoukil]

@bazzadp i would prefer to have a sitemap index that lists all the sitemaps.
So in the robots.txt we put just the sitemap index URL.

I wouldn't add the <changefreq></changefreq> attribute (not used by Google and not useful n our case for the other engines.)

So in the sitemap-en.xml we will loop through all the years pages ?

[tunetheweb]

@bazzadp i would prefer to have a sitemap index that lists all the sitemaps.
So in the robots.txt we put just the sitemap index URL.

Either works for me. Whichever is easiest to implement.

I wouldn't add the <changefreq></changefreq> attribute (not used by Google and not useful n our case for the other engines.)

Fair enough - removed.

So in the sitemap-en.xml we will loop through all the years pages ?

Good question and not sure. Could have sitemap-en-2019.xml as well if easier.

[tunetheweb]

I've done this as one sitemap.xml file and it handles all years and languages.