Refactor bleve tokenizer usage

[srfrog]

This PR refactors bleve term and fulltext tokenizers used in predicate indexes.

• Adds support for base language stop tokens and stemmer filters of all bleve language tokenizers.
• Deprecates the language index error for languages that don't have fulltext tokenizer.
• Removes snowball dependency
• Deprecates contrib script that generates the stopword dictionaries
• Update bleve to the current stable version, while reducing its footprint in alpha.
• Moderate performance gains for fulltext tokenizing.

Closes #2622
Closes #2601

Replaces #2602

---

This change is 

[danielmai]

Reviewed 1 of 1 files at r3.
Reviewable status: 1 of 165 files reviewed, 2 unresolved discussions (waiting on @golangcibot, @srfrog, and @manishrjain)

---

wiki/content/query-language/index.md, line 371 at r3 (raw file):

Dgraph uses [bleve](https://github.com/blevesearch/bleve) for its full text search indexing. See also the bleve language specific [stop word lists](https://github.com/blevesearch/bleve/tree/master/analysis/lang).

Following table contains all supported languages, corresponding country-codes, stop words and stemming filtering support.

Mention "stemming" before "stop words" since that's the order they're presented in the table below.

[manishrjain]

Reviewable status: 1 of 165 files reviewed, 12 unresolved discussions (waiting on @golangcibot, @srfrog, and @manishrjain)

---

posting/index.go, line 65 at r3 (raw file):

				"Consider switching to hash for better performance.\n", attr)
		}
		toks, err := tok.BuildTokens(sv.Value, it.SetLang(lang))

getLangTokenizer(t, lang) a helper func.

---

tok/bleve.go, line 91 at r3 (raw file):

}

func getFullTextTokens(s, lang string) ([]string, error) {

Make this part of the fulltext tokenizer struct. So, we can keep the objects we're receiving from bleveCache in that struct, instead of creating new stopx and stemmerx structs.

---

tok/langbase.go, line 55 at r3 (raw file):

		return lang
	}
	tag, _ := language.Parse(lang)

err

x.Check(err)

---

tok/langbase.go, line 57 at r3 (raw file):

	tag, _ := language.Parse(lang)
	if tag != language.Und { // something like "x-klingon" should retag to "en"
		if base, conf := tag.Base(); conf > language.No {

Add some comments explaining what's going on.

---

tok/tok.go, line 84 at r2 (raw file):

	registerTokenizer(MonthTokenizer{})
	registerTokenizer(DayTokenizer{})
	registerTokenizer(TermTokenizer{})

Where did TermTokenizer go?

---

tok/tok.go, line 119 at r2 (raw file):

func registerTokenizer(t Tokenizer) {
	if _, found := tokenizers[t.Name()]; found {
		glog.V(3).Infof("Duplicate tokenizer: %s", t.Name())

This should be an assert failure.

---

tok/tok.go, line 123 at r2 (raw file):

	}
	if _, ok := types.TypeForName(t.Type()); !ok {
		glog.V(3).Infof("Invalid type %q for tokenizer %s", t.Type(), t.Name())

Assert failure again.

---

tok/tok.go, line 136 at r2 (raw file):

	return types.IndexGeoTokens(v.(geom.T))
}
func (t GeoTokenizer) SetLang(string) Tokenizer { return t }

This func seems very specific to FTS. Doesn't seem to make sense to have it in every tokenizer.

---

worker/stringfilter.go, line 110 at r3 (raw file):

	tokens, err := tok.BuildTokens(value.Value, tokenizer.SetLang(filter.lang))
	if err != nil {
		glog.V(3).Infof("error while building tokens: %s", err)

glog.Errorf(...)

---

worker/task.go, line 1271 at r2 (raw file):

			return nil, x.Errorf("Could not find tokenizer with name %q", tokerName)
		}
		fc.tokens, err = tok.BuildTokens(valToTok.Value, tokenizer.SetLang(langForFunc(q.Langs)))

getIfLang(...)

[srfrog]

Reviewable status: 0 of 167 files reviewed, 12 unresolved discussions (waiting on @danielmai, @golangcibot, and @manishrjain)

---

posting/index.go, line 40 at r1 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

maxBatchSize is unused

Done.

---

posting/index.go, line 65 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

getLangTokenizer(t, lang) a helper func.

Done.

---

tok/bleve.go, line 91 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Make this part of the fulltext tokenizer struct. So, we can keep the objects we're receiving from bleveCache in that struct, instead of creating new stopx and stemmerx structs.

Done.

---

tok/langbase.go, line 55 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

err

x.Check(err)

I checked and we can't use that because a syntax error is possible. And I think language syntax errors shouldnt terminate Dgraph, but instead use the default lang "en".

---

tok/langbase.go, line 57 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Add some comments explaining what's going on.

Done.

---

tok/tok.go, line 84 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Where did TermTokenizer go?

I've put them back in tok.go, and do the bleve setup separately in bleve.go.

---

tok/tok.go, line 119 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

This should be an assert failure.

Done.

---

tok/tok.go, line 123 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Assert failure again.

Done.

---

tok/tok.go, line 136 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

This func seems very specific to FTS. Doesn't seem to make sense to have it in every tokenizer.

Done.

---

wiki/content/query-language/index.md, line 371 at r3 (raw file):

Previously, danielmai (Daniel Mai) wrote…

Mention "stemming" before "stop words" since that's the order they're presented in the table below.

Done.

---

worker/stringfilter.go, line 110 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

glog.Errorf(...)

Done.

---

worker/task.go, line 1271 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

getIfLang(...)

Done.