Support es5

[Qmando]

Initial es5 support.

create -> index everywhere.
Tests assume ES 2.
Queries use two formats, one for ES<5 and one for ES>=5.

Note:

• test_rule.py not changed yet.
• Old query string queries need to be changed.
• Aggregations don't work yet.
• Not very well tested, very much a work in progress

[k2xl]

nice work. LGTM

[Alexwii]

Hello!

I have an error when I test my config file and after all seem good...:

elastalert-test-rule test2.yaml
Successfully loaded Testing rule

WARNING:elasticsearch:GET http://127.0.0.1:9200/logstash-*/snmptrap/_count?ignore_unavailable=true [status:400 request:0.003s]
Error querying Elasticsearch:
RequestError(400, u'parsing_exception', {u'status': 400, u'error': {u'line': 1, u'root_cause': [{u'reason': u'no [query] registered for [filtered]', u'type': u'parsing_exception', u'line': 1, u'col': 24}], u'type': u'parsing_exception', u'reason': u'no [query] registered for [filtered]', u'col': 24}})
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent. To send them, use --verbose.
INFO:elastalert:Queried rule Testing rule from 2016-11-29 16:25 CET to 2016-11-29 17:10 CET: 2 / 2 hits
INFO:elastalert:Queried rule Testing rule from 2016-11-29 17:10 CET to 2016-11-29 17:55 CET: 6 / 4 hits
INFO:elastalert:Queried rule Testing rule from 2016-11-29 17:55 CET to 2016-11-29 18:40 CET: 9 / 3 hits
...

When I add option --alert, I have so much logs... And it wants interrogate default settings:

elastalert_error - {'message': "Error while querying for alert silence status: ConnectionError(HTTPConnectionPool(host='es', port=14900): Max retries exceeded with url: /wb/silence/_search?_source_include=until%2Cexponent&size=1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x2033310>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPConnectionPool(host='es', port=14900): Max retries exceeded with url: /wb/silence/_search?_source_include=until%2Cexponent&size=1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x2033310>: Failed to establish a new connection: [Errno -2] Name or service not known',)))", 'traceback': ['Traceback (most recent call last):', '  File "/usr/lib/python2.6/site-packages/elastalert/elastalert.py", line 1265, in is_silenced', "    size=1, body=query, _source_include=['until', 'exponent'])", '  File "/usr/lib/python2.6/site-packages/elastalert-0.1.3-py2.6.egg/elasticsearch/client/utils.py", line 69, in _wrapped', '    return func(*args, params=params, **kwargs)', '  File "/usr/lib/python2.6/site-packages/elastalert-0.1.3-py2.6.egg/elasticsearch/client/__init__.py", line 539, in search', "    doc_type, '_search'), params=params, body=body)", '  File "/usr/lib/python2.6/site-packages/elastalert-0.1.3-py2.6.egg/elasticsearch/transport.py", line 327, in perform_request', '    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)', '  File "/usr/lib/python2.6/site-packages/elastalert-0.1.3-py2.6.egg/elasticsearch/connection/http_requests.py", line 79, in perform_request', "    raise ConnectionError('N/A', str(e), e)", "ConnectionError: ConnectionError(HTTPConnectionPool(host='es', port=14900): Max retries exceeded with url: /wb/silence/_search?_source_include=until%2Cexponent&size=1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x2033310>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPConnectionPool(host='es', port=14900): Max retries exceeded with url: /wb/silence/_search?_source_include=until%2Cexponent&size=1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x2033310>: Failed to establish a new connection: [Errno -2] Name or service not known',)))"], 'data': {'rule': 'Testing rule._silence'}} 

[Alexwii]

Ok, I had just needed to precise the global conf with

--config /opt/elastalert/config.yaml

[matthillary]

I'm getting this error while running this branch against my es5:

Error running your filter:
RequestError(400, u'parsing_exception', {u'status': 400, u'error': {u'line': 1, u'root_cause': [{u'reason': u'no [query] registered for [filtered]', u'type': u'parsing_exception', u'line': 1, u'col': 68}], u'type': u'parsing_exception', u'reason': u'no [query] registered for [filtered]', u'col': 68}})

My filter looks like this:

filter:
- query:
    query_string:
      query: "field: blargh"

any ideas?

Thanks!

[Qmando]

@matthillary Are you using use_count_query or use_terms_query?

[matthillary]

@Qmando Huge thanks for the quick reply!

The type == any (http://elastalert.readthedocs.io/en/latest/ruletypes.html#any).

It worked with the previous ES version, but not with ES5. Hopefully that gives enough information to help! Let me know what else I can provide to help troubleshoot!

Thanks!

[Qmando]

Is that a no to using use_count_query or use_terms_query? I need to see the full rule to understand where this error is coming from.

[matthillary]

@Qmando -- I'm not using either -- it's an any alert type and is not using either of those parameters (use_count_query or use_terms_query)

here's the full rule minus the alert info:

es_host: localhost
es_port: 9200

name: private_info_access

type: any

index: audit_events-*

filter:
- query:
    query_string:
      query: "entity_name: PrivateInfo"

alert:
- "email"
- "slack"

I hope that helps! Let me know if you need anything else!

I'll dig in more tomorrow as well to see if I can get more info on the error. Thanks for taking time to get this up to support for ES5!

[Alexwii]

Can you test that ? I had this same error before and now it's ok

filter:
 - query_string:
     query: "entity_name:PrivateInfo"

[matthillary]

Works perfectly! Thanks @Alexwii!

[Qmando]

Hmm, I thought this commit would have fixed that bug
bb09323
Did you have the most recent commit checked out?

[dadadom]

The following rule configuration throws an error when being used on the ES5 branch with ES5:

type: spike
index: mylogs-*
threshold_cur: 25
timeframe:
  minutes: 5
spike_height: 3
spike_type: up

doc_type: logs
use_terms_query: true
terms_size: 50

query_key: server_id

include:
- loglevel

filter:
- term:
    loglevel: ERROR
- not:
    terms:
      classname: [Classname1, Classname2]

alert:
- slack

The error is:

WARNING:elasticsearch:GET http://1.2.3.4:9200/mylogs-2017.01.08/logs/_search?ignore_unavailable=true&size=0 [status:400 request:0.021s]
ERROR:root:Error running query: TransportError(400, {u'line': 1, u'root_cause': [{u'reason': u'no [query] registered for [not]', u'type': u'parsing_exception', u'line': 1, u'col': 195}], u'type': u'parsing_exception', u'reason': u'no [query] registered for [not]', u'col': 195})

[Update]
The rule also does not work without the doc_type and use_terms_query parameters. Seems to be the -not: part of the filter which breaks in general, not only for terms/count queries.
Which kind of makes sense, since the query is within the must part of the bool query and if I understood it correctly, all not conditions should be in the must_not part.

[fengyingjie2018]

s_host: 127.0.0.1

es_port: 9200

name: Example rule

use_strftine_index: true

type: frequency

index: :cache-diags*

num_events: 1

timeframe:
hours: 1

minutes: 1

filter:

• query_string:
  query: "message:*truncated"
  smtp_host: smtp.163.com
  smtp_prot: 25
  smtp_ssl: false

error :
WARNING:elasticsearch:GET http://127.0.0.1:9200/elastalert_status/elastalert/_search?size=1000 [status:400 request:0.269s]
ERROR:root:Error finding recent pending alerts: TransportError(400, u'search_phase_execution_exception', u'No mapping found for [alert_time] in order to sort on') {'sort': {'alert_time': {'order': 'asc'}}, 'query': {'bool': {'filter': {'range': {'alert_time': {'to': '2018-01-26T06:17:37.337638Z', 'from': '2018-01-25T06:17:37.337614Z'}}}, 'must': {'query_string': {'query': '!exists:aggregate_id AND alert_sent:false'}}}}}

这是为什么呢？那里有错误么

[Qmando]

@fengyingjie2018
你首先需要使用elastalert-create-index

您可以尝试删除索引并重新创建它

有时它不完全工作