Integration with elasticsearch v5

[rauizab]

Hi

I know that the new version v5 of elasticsearch has just being released, but do you know when it will be available the integration of elastalert with the new version? At the moment I am getting:

INFO:elastalert:Starting up
WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert/_search?size=1000 [status:400 request:0.013s]
WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_status/_search?_source_include=endtime%2Crule_name&size=1 [status:400 request:0.002s]
ERROR:root:Error querying for last run: TransportError(400, {u'line': 1, u'root_cause': [{u'reason': u'Unknown key for a START_OBJECT in [filter].', u'type': u'parsing_exception', u'line': 1, u'col': 12}], u'type': u'parsing_exception', u'reason': u'Unknown key for a START_OBJECT in [filter].', u'col': 12})
WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_error?op_type=create [status:400 request:0.003s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'})
Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1033, in writeback
doc_type=doc_type, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 227, in create
return self.index(index, doc_type, body, id=id, params=params, op_type='create')
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 257, in index
_make_path(index, doc_type, id), params=params, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 72, in perform_request
self._raise_error(response.status_code, raw_data)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 102, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'})
WARNING:elasticsearch:GET http://elasticsearch:9200/payment-*/_search?_source_include=eventtime%2C%2A%2CuserId&ignore_unavailable=true&size=10000 [status:400 request:0.003s]
ERROR:root:Error running query: TransportError(400, {u'line': 1, u'root_cause': [{u'reason': u'no [query] registered for [filtered]', u'type': u'parsing_exception', u'line': 1, u'col': 67}], u'type': u'parsing_exception', u'reason': u'no [query] registered for [filtered]', u'col': 67})
WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_error?op_type=create [status:400 request:0.003s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'})
Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1033, in writeback
doc_type=doc_type, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 227, in create
return self.index(index, doc_type, body, id=id, params=params, op_type='create')
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 257, in index
_make_path(index, doc_type, id), params=params, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 72, in perform_request
self._raise_error(response.status_code, raw_data)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 102, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'})
INFO:elastalert:Ran User1111MakePayment from 2016-11-02 08:20 UTC to 2016-11-02 13:20 UTC: 0 query hits, 0 matches, 0 alerts sent

Thanks for this great tool!!!!

[penekk]

ES 5 has been around for a while already, its the GA version that just came out, any ETA on the support would be lovely 👍

[watollop]

+1

[lrolsen]

+1

[vijayarulmuthu]

+1

On Sat, Nov 5, 2016 at 5:14 AM -0700, "lrolsen" <[email protected]mailto:[email protected]> wrote:

+1

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/790#issuecomment-258607858, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AH7j4V-D_DPsJ-KmIqww-sOMgZ5FaPG-ks5q7HMggaJpZM4KnOvN.

[sabueso]

@watollop say me to put +1 here

[pasqualguerrero]

+1

[tynor88]

+1

[bennneuh]

+1

[nick-george]

+1 Duplicate of #510

[AndrewPix]

+1

[Alexwii]

+1

[stumyp]

Is it known what is broken and how can we help ?

[bkeifer]

IRC pointed me to https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L159

[doublesea]

+1

[doublesea]

Is it known what is broken and how can we help ?

as i known, it is because "filter" is not available any more in ES 5.0.

There are many "filter" query in EA which will get the 400 HTTP response for the incorrect query clause.

[kojisaiki]

+1

[branchnetconsulting]

+1

[stumyp]

It looks like all needed is to rewrite this query:
https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L159
(thanks @bkeifer )

Documentation says that change should be simple: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/query-dsl-filtered-query.html

I'll try to find some time, my python is a bit rusty :)

[timwsuqld]

My reading of that doc suggests that the change should be:

diff --git a/elastalert/elastalert.py b/elastalert/elastalert.py
index 2cbd553..a13e739 100644
--- a/elastalert/elastalert.py
+++ b/elastalert/elastalert.py
@@ -156,7 +156,7 @@ class ElastAlerter():
         if starttime and endtime:
             es_filters['filter']['bool']['must'].insert(0, {'range': {timestamp_field: {'gt': starttime,
                                                                                         'lte': endtime}}})
-        query = {'query': {'filtered': es_filters}}
+        query = {'query': {'bool': es_filters}}
         if sort:
             query['sort'] = [{timestamp_field: {'order': 'desc' if desc else 'asc'}}]
         return query

I'll see if I can make it work in our ES5 environment

[stumyp]

@timwsuqld: plus if statement to detect ES version

[timwsuqld]

@stumyp Good point. Any idea if we already detect ES version anywhere?

[stumyp]

@timwsuqld : So far I saw only this: 6358435

Not really a version detection :)

[timwsuqld]

I'm also not sure if the elasticsearch library (https://pypi.python.org/pypi/elasticsearch/5.0.1) is backwards compatible. My understanding is that it should be, so we can use version 5 of the library with older ES clusters.

[stumyp]

From what they say on the link you gave: major version of library must match ES version.
Means, elastalert should split versions too or just warn users for incompatibility ?

[Qmando]

I believe this patch must also be applied, because id must be explicitly set to None.

diff --git a/elastalert/elastalert.py b/elastalert/elastalert.py
index 2cbd553..fd49223 100644
--- a/elastalert/elastalert.py
+++ b/elastalert/elastalert.py
@@ -841,7 +841,8 @@ class ElastAlerter():

         res = es.create(index='kibana-int',
                         doc_type='temp',
-                        body=db_body)
+                        body=db_body,
+                        id=None)

         # Return dashboard URL
         kibana_url = rule.get('kibana_url')
@@ -1015,7 +1016,7 @@ class ElastAlerter():
         if self.writeback_es:
             try:
                 res = self.writeback_es.create(index=self.writeback_index,
-                                               doc_type=doc_type, body=body)
+                                               doc_type=doc_type, body=body, id=None)
                 return res
             except ElasticsearchException as e:
                 logging.exception("Error writing alert info to Elasticsearch: %s" % (e))

[timwsuqld]

For what it's worth, I appear to have no problems using the elasticsearch 2.4.0 library, and the index creation worked without a problem, as do my filters with my patch. I'm sure there are incompatibilities between the 2.4.0 library and ES 5, but maybe the things that we need won't actually hit those incompatibilities?

[Qmando]

The filters work fine? The current format is

query:
  filtered:
    filter:
      bool:
        must: [filters from rule here]

The docs say that filtered has been deprecated. I'm not really an expert on the query DSL though.

Another breaking change: No more fields
https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L243
That should be stored_fields instead.

search_type=count is removed too
https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L318
Instead, you have to add size: 0

These are just what stood out when scanning the breaking changes page

[ThomasdOtreppe]

For reference: elastic/elasticsearch-py#476

[stumyp]

@Qmando , @timwsuqld : I think good way to test compatibility and see if any deprecated queries left is to run your elastalert instance on cluster with enabled deprecation logging:

curl -XPUT localhost:9200/_cluster/settings -d '{   "transient": { "logger": { "deprecation": "DEBUG" } } }'

This is what I have in my logs:

a lot of this:

[2016-12-28 16:01:55,259][DEBUG][deprecation.common       ] Deprecated field [ignore_unmapped] used, replaced by [unmapped_type]

and this:

[2016-12-28 16:01:56,658][DEBUG][deprecation.index.query  ] The [query] filter is deprecated, you can now use queries as filters directly.
[2016-12-28 16:01:56,658][DEBUG][deprecation.index.query  ] The [and] query is deprecated, please use a [bool] query instead with [must] clauses.
[2016-12-28 16:01:56,658][DEBUG][deprecation.index.query  ] The [filtered] query is deprecated, please use a [bool] query instead with a [must] clause for the query part and a [filter] clause for the filter part.

[PeterDefendo]

@Qmando Thanks adding --config /path/to/config.yaml resolved the issue.

[bHoskins07]

@tfgm-bud What elasticsearch python module version are you using? Also, can you post some of those simple queries. Thank You!

[tfgm-bud]

@bHoskins07 Not sure what you are asking with "What elasticsearch python module version are you using?". I'm using the github version and changing to the support_es5 branch:

git checkout origin/support_es5

And a simple query would be:

filter:
- term:
    env: "live"
- query_string:
    query: 'msgtype:activity AND ("transaction" adjusted unchanged cancelled)'

Not sure how much that really helps you though....

[Qmando]

@tfgm-bud To find the version of the elasticsearch module, type $ pip freeze | grep elasticsearch

[tfgm-bud]

@Qmando oh, I mis-read @bHoskins07 question and was thinking he was asking about elastalert.

elasticsearch==2.4.0

[PeterDefendo]

Im new to this but why does this work:

filter:
- query_string:
    query: "received_from: Prod AND (level: FATAL OR level: ERROR)"

But this doesn't:

filter:
- and:
    - term:
        received_from: "Prod"
    - or:
        - term:
            level: "FATAL"
        - term:
            level: "ERROR"

Aren't they the same thing?

The latter gives the following error {'message': "Error running query: TransportError(400, u'parsing_exception', u'[and] query malformed, no start_object after query name')"...

I'm guessing this is the same issue as @Hronom has.

[bHoskins07]

Thank You @tfgm-bud. I was testing 5.1.1 using elastalert-test-rule which was not working. When I switched to $ python -m elastalert.elastalert --rule it worked. Also, I am using python elasticsearch module 5.0.1

[tfgm-bud]

@bHoskins07 - I see what you mean. I upgraded to elasticsearch-5.0.1 with sudo pip install --upgrade elasticsearch and elastalert-test-rule fails:

$ elastalert-test-rule --config /etc/elastalert/config.yaml /etc/elastalert/rules/news.yaml 
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3015, in <module>
    @_call_aside
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2999, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3028, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 657, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 670, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 849, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'elasticsearch<3.0.0' distribution was not found and is required by elastalert

I made the following change to try to fix it and it works for me:

user@elk:~/elastalert[support_es5]$ git diff
diff --git a/setup.py b/setup.py
index 1f433c1..ebf817a 100644
--- a/setup.py
+++ b/setup.py
@@ -23,7 +23,7 @@ setup(
     package_data={'elastalert': ['schema.yaml']},
     install_requires=[
         'argparse',
-        'elasticsearch<3.0.0',  # Elastalert is not yet compatible with ES5
+        'elasticsearch',
         'jira==0.32',  # jira.exceptions is missing from later versions
         'jsonschema',
         'mock',

@timwsuqld can you incorporate this into your branch? -- Bud

[deepthi0311]

@Qmando , I had the same issue while working with master branch. After checking out support_es5 branch and switching to that branch, it works perfectly fine. Can you please merge this branch to master branch and let me know after merging?

[aatchison]

Wahoo! I'm watching and waiting for ES5!

[dadadom]

I added an issue with the branch as a comment in the PR. Is that OK? Should I mention the problem here or should I create a separate issue for that branch?

[deepthi0311]

Hi, Yes, thanks for adding it in the PR. No need to create a separate issue for that branch. Thanks and regards, Deepthi D From: Dominik Sandjaja [mailto:[email protected]] Sent: Monday, January 9, 2017 8:53 PM To: Yelp/elastalert <[email protected]> Cc: Deepthi D <[email protected]>; Comment <[email protected]> Subject: Re: [Yelp/elastalert] Integration with elasticsearch v5 (#790) I added an issue with the branch as a comment in the PR. Is that OK? Should I mention the problem here or should I create a separate issue for that branch? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#790 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AWWRWKEbKy_zcbgsbmW228UAHZqtCuVMks5rQlC4gaJpZM4KnOvN>.

[ssinhaonline]

@Qmando thank you for the support_es5 branch. Works for me with changes to query as suggested by @doublesea

[rptete]

then, I would try elastalert on ES5?

[dreeco]

+1

[ssinhaonline]

@rptete Go right ahead.

[hollowimage]

+1. is there an eta for merging the support_es5 branch into mainstream?

[nsgeorgi]

Please notify when you merge this branch to the master one. Thanks

[Qmando]

This has been merged with master and released.

[yanncarlier]

Im running elasticsearch 5.2.2
wen I run elastalert-create-index i have this error:

root@logstash:/data/elastalert# elastalert-create-index
Traceback (most recent call last):
File "/usr/local/bin/elastalert-create-index", line 6, in
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3037, in
@_call_aside
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3021, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3050, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 655, in _build_master
ws.require(requires)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 969, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 855, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jira>=1.0.10' distribution was not found and is required by elastalert
root@logstash:/data/elastalert#

[yanncarlier]

after
pip install -r requirements.txt
pip install functools32

its working, thanks