Improve SEO

[Throne3d]

Google doesn't seem to be fully indexing the site; fix that as much as possible

Also:

• /posts/:id?page=:page should be /posts/:id/page/:page, perhaps? See here for information on how.

[Marri]

Also fix that we block replies/search from being scraped (blocking results is fine, and the page isn't that important, but it is not in fact blockworthy)

[Throne3d]

Google still seems to not index second pages of threads (at least not well enough you can search for terms in them, whereas you can search for terms on the first pages).

Sitemaps may help: https://support.google.com/webmasters/answer/183668?hl=en&ref_topic=4581190. Related task is picking one of www.glowfic.com / glowfic.com to be our canonical address (I now vote the former since iirc we have weird DNS records on the last), per https://support.google.com/webmasters/answer/139066.

https://github.com/kjvarga/sitemap_generator#adding-links is a gem for it, and it looks like you generate the sitemap on a cronjob on a background worker. Could be useful? We can even pass last modified attributes, which may help with crawlers getting all over the site (but we may need to be careful around layout changes there).