server: better way to handle HTTP redirects

[shcheklein]

Every time we move a page to a new URL we need to create a proper redirect.

The only way to this now is to create a 4-5 lines entry in our custom server.js. This is def. not scalable. We need to find an easier way to handle this.

The first solution that comes to my mind is a separate file with a list of simple expressions, may be converter functions.

[jorgeorpinel]

separate file with a list of simple expressions...

And maybe add a note with the date the redirect was added (can also be checked with GitHub Blame but it's not always straightforward) for periodic cleaning up of this list.

Another option is to use DNS for this, not the app.

[Suor]

@jorgeorpinel I don't think those should be cleaned up unless they clash with something.

Another thing - do not forget to use HTTP code 301 for moved things, not 302.

[jorgeorpinel]

Actually @Suor we are having some trouble with some 301 "permanent" redirects that we now want to change again... They're cached by the browser and can't be changed again by us until that cache expires a long time later. 302 Found may be better for us since we are not so sure they are so permanent. See #768 (comment)

[shcheklein]

I think they are still 301s @jorgeorpinel . Even though we change the end location we don't change the fact of the redirect. So, unless we don't find a good solution for caching, I would keep it 301. But I haven't googled best practices around dealing with cache yet.

[jorgeorpinel]

Well, they both have an impact. 301 is better in general but we need to be sure the redirect is actually permanent and since we're in an active development phase... I think we should not use 301s as much. Many of our redirects are temporary, which is what 302 is for. As for SEO effects:

302 redirect informs search engines to continue indexing old pages. It may impact importantly your SEO results. When you are optimizing the new page URL, make certain to be vigilant about not having duplicate content on both old and new pages at a similar time. The temporary redirect is very useful when you bring your website visitors back to the old page after some time.

https://www.firstrankseoservices.com/blog/effects-of-301-and-302-redirect-on-seo/

[shcheklein]

Almost all our redirects are 301s in nature. We do not plan to bring your website visitors back to the old page after some time.. The only thing that is potentially not permanent is the URL we redirect to (not the fact of redirect). And probably the right way to handle this actually is to keep old 301s and add new ones (making a chain of 301s effectively).

Either we do that, or we can try no-cache as a hack.

Looks like 302s is not a good option in terms of SEO.

[jorgeorpinel]

p.s we don't HAVE to use 302 Found, there's also 303 See Other, and 307 Temporary Redirect.

https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_Redirection

[jorgeorpinel]

Full list of current redirects in server.js (from #768 (comment)) and their motivation:

• Enforce https protocol and remove www from host - SEO
• man.dvc.org/{cmd} -> dvc.org/doc/command-reference/{cmd}- Route design
  • replace - for / in {cmd} except for /get-url, /import-url - Backward compatibility
• {code/data/remote}.dvc.org -> corresponding S3 bucket- Route design
• path /(deb|rpm) -> corresponding S3 bucket- Route design
• path /(help|chat) -> Discord chat invite- Route design
• path /doc/commands-reference... -> /doc/command-reference... - Backward compatibility
• path /doc/tutorial/... -> /doc/tutorials/deep/... - Backward compatibility
• path /doc/tutorial -> /doc/tutorials - Backward compatibility
• path /doc/use-cases/data-and-model-files-versioning -> /doc/use-cases/versioning-data-and-model-files - Backward compatibility
• path /doc*/{item} -> /doc/{item} - Backward compatibility (also removes trailing /)

[jorgeorpinel]

p.p.s. also there's a new 308 Permanent Redirect code in HTTP 1.1 to replace 301, and is now supposedly supported by most major browsers that have significant market share. https://tools.ietf.org/html/rfc7538#page-3

[jorgeorpinel]

OK, well, as Next.js doesn't yet feature HTTP redirects the only way to do it is with a custom server script like we already have (/server.js) (which disables automatic static optimization unfortunately).

What I think we can do to improve this is to extract all the redirect logic to a separate module that loads the regex and targets from a JSON in order, and tests them all. That way we only ever change the JSON file in the future, which is kind of configuration instead of code (it could even live outside the Git repo).

[shcheklein]

extract all the redirect logic to a separate module that loads the regex and targets from a JSON in order, and tests them all

yes! that was the idea of this ticket

[jorgeorpinel]

OK got it, great. Actually though, yet another option I'd like to leave here for the record is to write and deploy a completely separate proxy server that exclusively does redirects, and that fetches our regular node app when there's no special redirect.

I'd still vote for the separate module at this point.

[jorgeorpinel]

p.s we don't HAVE to use 302 Found, there's also 303 See Other, and 307 Temporary Redirect.

On this, I vote for 303 See Other as discussed in #875 (review).

[shcheklein]

yep. 303 should be fine, not ideal as well but fine.