Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add script_filter tokenfilter #33431

Merged
merged 6 commits into from
Sep 11, 2018
Merged

Add script_filter tokenfilter #33431

merged 6 commits into from
Sep 11, 2018

Conversation

romseygeek
Copy link
Contributor

This will allow users to filter out terms using scripted predicates, rather than having to write Java code and wiring things up via analysis plugins.

@romseygeek romseygeek self-assigned this Sep 5, 2018
@romseygeek romseygeek requested a review from colings86 September 5, 2018 16:29
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search-aggs

@romseygeek romseygeek mentioned this pull request Sep 6, 2018
Closed
nik9000
nik9000 approved these changes Sep 6, 2018
View reviewed changes
Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

...analysis-common/src/main/java/org/elasticsearch/analysis/common/AnalysisPredicateScript.java
}

public boolean isKeyword() {
return isKeyword;
return keywordAtt.isKeyword();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

rjernst
rjernst reviewed Sep 6, 2018
View reviewed changes
Copy link
Member

@rjernst rjernst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use consistent terminology in the class names and token filter name? I think we should use "predicate" terminology everywhere? This will leave the namespace open for other types of scripted token filters in the future.

docs/reference/analysis/tokenfilters/scriptfilter-tokenfilter.asciidoc Outdated
--------------------------------------------------
// CONSOLE

<1> This will skip tokens that are 5 characters long or less
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This description would make more sense with positive logic, since the predicate is positive based. So something like:

<1> This will emit tokens that are more than 5 characters long

@romseygeek
Copy link
Contributor Author

@rjernst so call it predicate_filter and PredicateFilterFactory, etc?

@rjernst
Copy link
Member

rjernst commented Sep 6, 2018

How about predicate_token_filter and PrediateTokenFilterScriptFactory?

rjernst
rjernst approved these changes Sep 8, 2018
View reviewed changes
Copy link
Member

@rjernst rjernst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @romseygeek. The new naming LGTM.

docs/reference/analysis/tokenfilters/predicate-tokenfilter.asciidoc Outdated
[[analysis-scriptfilter-tokenfilter]]
=== Scripted Filtering Token Filter
[[analysis-predicatefilter-tokenfilter]]
=== Predicate Token Script Filter
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Token Script Filter -> Token Filter Script

@romseygeek romseygeek merged commit f598297 into elastic:master Sep 11, 2018
romseygeek added a commit that referenced this pull request Sep 11, 2018
@romseygeek
Add predicate_token_filter (#33431)
0b1a7b9 
This allows users to filter out tokens from a TokenStream using painless scripts, 
instead of having to write specialised Java code and packaging it up into a plugin.

The commit also refactors the AnalysisPredicateScript.Token class so that it wraps
and makes read-only an AttributeSource.
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Sep 11, 2018
@jasontedor
Merge branch 'master' into upgrade-remote-cluster-settings
f63b095 
* master:
  Preserve cluster settings on full restart tests (elastic#33590)
  Use IndexWriter.getFlushingBytes() rather than tracking it ourselves (elastic#33582)
  Fix upgrading of list settings (elastic#33589)
  Add read-only Engine (elastic#33563)
  HLRC: Add ML get categories API (elastic#33465)
  SQL: Adds MONTHNAME, DAYNAME and QUARTER functions (elastic#33411)
  Add predicate_token_filter (elastic#33431)
  Fix Replace function. Adds more tests to all string functions. (elastic#33478)
  [ML] Rename input_fields to column_names in file structure (elastic#33568)
jasontedor added a commit to martijnvg/elasticsearch that referenced this pull request Sep 11, 2018
@jasontedor
Merge branch 'master' into pr/33371
4d77eb7 
* master: (91 commits)
  Preserve cluster settings on full restart tests (elastic#33590)
  Use IndexWriter.getFlushingBytes() rather than tracking it ourselves (elastic#33582)
  Fix upgrading of list settings (elastic#33589)
  Add read-only Engine (elastic#33563)
  HLRC: Add ML get categories API (elastic#33465)
  SQL: Adds MONTHNAME, DAYNAME and QUARTER functions (elastic#33411)
  Add predicate_token_filter (elastic#33431)
  Fix Replace function. Adds more tests to all string functions. (elastic#33478)
  [ML] Rename input_fields to column_names in file structure (elastic#33568)
  Add full cluster restart base class (elastic#33577)
  Validate list values for settings (elastic#33503)
  Copy and validatie soft-deletes setting on resize (elastic#33517)
  Test: Fix package name
  SQL: Fix result column names for arithmetic functions (elastic#33500)
  Upgrade to latest Lucene snapshot (elastic#33505)
  Enable not wiping cluster settings after REST test (elastic#33575)
  MINOR: Remove Dead Code in SearchScript (elastic#33569)
  [Test] Remove duplicate method in TestShardRouting (elastic#32815)
  mute test on windows
  Update beats template to include apm-server metrics (elastic#33286)
  ...
@romseygeek romseygeek deleted the scripted-stop-filter branch September 11, 2018 17:48
@Mpdreamz Mpdreamz mentioned this pull request Dec 13, 2018
Closed
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Dec 18, 2018
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431
473c49f
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Jan 2, 2019
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431
f15e6c7
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Jan 24, 2019
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431
8e77da6
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Jan 25, 2019
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431
795478b
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Jan 25, 2019
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431 (#3522)
6d9fa3a 
* support predicate_token_filter elastic/elasticsearch#33431

* add new file

* fix failing unit tests
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Jan 25, 2019
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431 (#3522)
6783ef0 
* support predicate_token_filter elastic/elasticsearch#33431

* add new file

* fix failing unit tests
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Jan 25, 2019
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431 (#3522)
6d5340b 
* support predicate_token_filter elastic/elasticsearch#33431

* add new file

* fix failing unit tests
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Jan 28, 2019
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431 (#3522)
aec0228 
* support predicate_token_filter elastic/elasticsearch#33431

* add new file

* fix failing unit tests

(cherry picked from commit 6d5340b)
Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Mar 29, 2019
@Mpdreamz
support predicate_token_filter elastic/elasticsearch#33431 (#3522)
d9d807d 
* support predicate_token_filter elastic/elasticsearch#33431

* add new file

* fix failing unit tests

(cherry picked from commit 6d5340b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rjernst rjernst rjernst approved these changes

@nik9000 nik9000 nik9000 approved these changes

@colings86 colings86 Awaiting requested review from colings86

Assignees

@romseygeek romseygeek

Labels
>feature :Search Relevance/Analysis How text is split into tokens v6.5.0 v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@romseygeek @elasticmachine @rjernst @nik9000 @colings86