Improving search

[aaruni96]

It is a known issue that search in our documentation leaves something to be desired for. I have been able to change search results, whether or not that is an improvement is not always clear.

This issue is to collect opinions and use cases for search, and also for figuring out the best way to make these changes permanent. Please add all the ways search misbehaves (provides different results than you expected / wished for), so that I can experiment with how to get those use cases working, and hopefully fix other unreported problems as a byproduct.

How search works

Here's a short overview of how search works, based on what I have been able to understand this past week:

Search is provided to us by Documenter.jl, using the Minisearch javascript search engine. It is an in memory search engine, which reads a pre-created index. So, the only two ways to change its behaviour are :

• change minisearch config (in javascript)
• change the search index (just a dictionary in a separate file in javascript)

Based on config options while constructing the minisearch instance in documenter.js, it takes in the query string as input, and applies it to search_index.js. The results are then filtered using a filter function in documenter.js

Changes I've tried already

• Use "AND" to combine queries, instead of "OR"
  This is, in my opinion, the single most useful change. It means the search results get more refined, the longer your search string is. Right now, the more words you type in, the more results are returned, because it searches for quo OR mpolyring.
  Searching for "quo MPolyRing" gives 200+ results in the current documentation, v/s only 35 results with the change.
  Searching for "random polytopes" now actually returns "rand_cyclic_polytope" and "rand01_polytope" as results.
  I have already made a PR in documenter for this : Combine results with AND JuliaDocs/Documenter.jl#2514 .

• Disable results from Hecke/Nemo/AbstractAlgebra showing up in results
  This is a questionable change, at least in its current form. While it means only direct Oscar documentation shows up in results, QQBarField now shows 0 results, because it is only documented inside Nemo.
  Also, the page for Quotient modules no longer shows up in search, even though it is navigable via the sidebar, because its from AbstractAlgebra. Similarly, the (useful) method roots(R::QQBarField, f::ZZPolyRingElem) is not show in search, as it comes from Nemo. And while searching for "Quotient Rings" no longer links the result from Hecke, it also doesn't link to Generic Residue Rings, because that is from AbstractAlgebra. (so, no improvements from Documentation for quotient rings (residue rings) #1908 , just different bad results)

Making the changes permanent

Once we have decided what changes we actually want in the search engine, we should also decide how to make the changes persist. Things which benefits the search engine in general (like the first change) can be upstreamed to Documenter. But things like the second change (or hopefully, a more useful variant of the this), which are useless outside the context of Oscar, but still need to be reflected in assets/documenter.js cannot be upstreamed, and maybe we need a patch file to make changes at build time ?

[lgoettgens]

Disable results from Hecke/Nemo/AbstractAlgebra showing up in results

Can you maybe just disable all results from files that are not contained in the doc.main file, i.e. the list of files reachable via the menu?

[aaruni96]

The search index is a javascript file which assigns the entire index to a variable. Instead, it could just be the index in a JSON file, and it could be assigned to a variable from within documenter.js. Then we could write a simple script which deletes any entry in the index which is not also in doc.main.

I've opened an issue in Documenter.jl for this.

[aaruni96]

Current state of experimental search can be previewed at https://oscar-system.github.io/Oscar.jl/previews/PR4292/