Update lucene to version 8.11.2 #15
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,3 +15,5 @@ rebel.xml | |
| /.gradle/ | ||
| /build/ | ||
| atlassian-ide-plugin.xml | ||
| .DS_Store | ||
| local.properties | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| /** | ||
| * Distribution License: | ||
| * JSword is free software; you can redistribute it and/or modify it under | ||
| * the terms of the GNU Lesser General Public License, version 2.1 or later | ||
| * as published by the Free Software Foundation. This program is distributed | ||
| * in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even | ||
| * the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. | ||
| * See the GNU Lesser General Public License for more details. | ||
| * | ||
| * The License is available on the internet at: | ||
| * http://www.gnu.org/copyleft/lgpl.html | ||
| * or by writing to: | ||
| * Free Software Foundation, Inc. | ||
| * 59 Temple Place - Suite 330 | ||
| * Boston, MA 02111-1307, USA | ||
| * | ||
| * © CrossWire Bible Society, 2007 - 2016 | ||
| * | ||
| */ | ||
| package org.apache.lucene.analysis; | ||
|
|
||
| import org.apache.lucene.analysis.core.LetterTokenizer; | ||
| import org.apache.lucene.analysis.cz.CzechAnalyzer; | ||
|
|
||
| /** | ||
| * An Analyzer whose {@link TokenStream} is built from a | ||
| * {@link LetterTokenizer} filtered with {@link LowerCaseFilter and @link StopFilter} (optional). | ||
| * Stemming not implemented yet | ||
| * | ||
| * @see gnu.lgpl.License The GNU Lesser General Public License for details. | ||
| * @author Sijo Cherian | ||
| * @author DM SMITH | ||
| */ | ||
| final public class CzechLuceneAnalyzer extends AbstractBookAnalyzer { | ||
| public CzechLuceneAnalyzer() { | ||
| stopSet = CzechAnalyzer.getDefaultStopSet(); | ||
| } | ||
|
|
||
| @Override | ||
| protected TokenStreamComponents createComponents(String fieldName) { | ||
| Tokenizer source = new LetterTokenizer(); | ||
| TokenStream result = new LowerCaseFilter(source); | ||
| if (doStopWords && stopSet != null) { | ||
| result = new StopFilter(result, (CharArraySet) stopSet); | ||
| } | ||
| return new TokenStreamComponents(source, result); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| /** | ||
| * Distribution License: | ||
| * JSword is free software; you can redistribute it and/or modify it under | ||
| * the terms of the GNU Lesser General Public License, version 2.1 or later | ||
| * as published by the Free Software Foundation. This program is distributed | ||
| * in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even | ||
| * the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. | ||
| * See the GNU Lesser General Public License for more details. | ||
| * | ||
| * The License is available on the internet at: | ||
| * http://www.gnu.org/copyleft/lgpl.html | ||
| * or by writing to: | ||
| * Free Software Foundation, Inc. | ||
| * 59 Temple Place - Suite 330 | ||
| * Boston, MA 02111-1307, USA | ||
| * | ||
| * © CrossWire Bible Society, 2007 - 2016 | ||
| * | ||
| */ | ||
| package org.apache.lucene.analysis; | ||
|
|
||
| import org.apache.lucene.analysis.core.LetterTokenizer; | ||
| import org.apache.lucene.analysis.en.EnglishAnalyzer; | ||
| import org.apache.lucene.analysis.en.PorterStemFilter; | ||
|
|
||
| /** | ||
| * English Analyzer works like lucene SimpleAnalyzer + Stemming. | ||
| * (LowerCaseTokenizer > PorterStemFilter). Like the AbstractAnalyzer, | ||
| * {@link StopFilter} is off by default. | ||
| * | ||
| * | ||
| * @see gnu.lgpl.License The GNU Lesser General Public License for details. | ||
| * @author sijo cherian | ||
| */ | ||
| final public class EnglishLuceneAnalyzer extends AbstractBookAnalyzer { | ||
|
|
||
| public EnglishLuceneAnalyzer() { | ||
| stopSet = EnglishAnalyzer.ENGLISH_STOP_WORDS_SET; | ||
| } | ||
|
|
||
|
|
||
| /** | ||
| * Constructs a {@link LetterTokenizer} with {@link LowerCaseFilter} filtered by a language filter | ||
| * {@link StopFilter} and {@link PorterStemFilter} for English. | ||
| */ | ||
| @Override | ||
| protected TokenStreamComponents createComponents(String fieldName) { | ||
| Tokenizer source = new LetterTokenizer(); | ||
| TokenStream result = new LowerCaseFilter(source); | ||
|
|
||
| if (doStopWords && stopSet != null) { | ||
| result = new StopFilter(result, (CharArraySet) stopSet); | ||
| } | ||
|
|
||
| // Using Porter Stemmer | ||
| if (doStemming) { | ||
| result = new PorterStemFilter(result); | ||
| } | ||
|
|
||
| return new TokenStreamComponents(source, result); | ||
| } | ||
|
|
||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why are these moved away from our namespace (org.crosswire)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because I needed to access some protected methods of Lucene classes in order to implement AbstractBookAnalyzer. Access to protected is only allowed in the same namespace.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm then solution is somewhat hacky. Options to consider:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I found out that there's public abtract class StopWordAnalyzerBase that probably could be used as a base class. At least that is the baseclass within lucene core lib that is used for per-language classes there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question arises are all custom per-language analyzers still really needed or could we simplify code by using analyzers from lucene core directly.
AbstractBookAnalyzer carries book info and it is passed to some filter classes, but any of those does not seem to use that information. I am having a feeling that all that could be simplified greatly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, I'll look into it