add an instruction to robots.txt to exclude docs search page

[akbarkz]

Done

• Added an instruction to robots.txt to exclude /server/docs/search* page since it's dynamic and doesn't need to be indexed by Google.

QA

• Can be QA-ed after deployment by checking logs in Google Search Console. Currently there are a lot of 5xx and 403 errors coming from the page as Google is trying to index all the different search queries.

Help

QA steps - Commit guidelines

[webteam-app]

Demo starting at https://ubuntu-com-13652.demos.haus

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.43%. Comparing base (5868739) to head (9183a77).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #13652   +/-   ##
=======================================
  Coverage   74.43%   74.43%           
=======================================
  Files         107      107           
  Lines        2848     2848           
  Branches      948      948           
=======================================
  Hits         2120     2120           
  Misses        704      704           
  Partials       24       24           

[carkod]

We may need to do this to every search endpoint if what was there Disallow: /search/* wasn't working. But I think we could use some kind of wildcard i.e. */docs/search/* (I made it up may need to be tested).