ELK - Elasticsearch or Kibana analyzer (contribution survey)

[Tux-Panik]

Request Type

Analyzer

Description

ELK stack is a famous solution to collect, store, index and query data.
Widely used in investigations or to store logs, an analyzer could help to directly interrogate a remote Elasticsearch or Kibana instance from TheHive or Cortex.

Complementary information

I would like to contribute to develop this new analyzer, but the last time I tried I failed.
Feel free to contact me if you want to work on this topic.
Kibana APIs: https://www.elastic.co/guide/en/kibana/master/api.html
Elasticsearch APIs: https://www.elastic.co/guide/en/elasticsearch/reference/1.4/_the_search_api.html

Regards,
Julien

[nicpenning]

This would be great!

Query hashes, IP addresses, urls, file names, etc.. In elastic and return the documents that are a hit. Might be tough because different logs have different field names to search on but if you can configure which observables will query which index and fields it would be efficient and extremely powerful.

I have been wanting to do this, but didn't think about making it an analyzer. Makes sense.

[jeffrey-e]

I think the analyzer should adhere to the upcoming Elastic Common Schema (https://github.com/elastic/ecs), then it should be rather universal to use. If no progress has been made on the point where I need it, I can start building it :)

[Tux-Panik]

Hi,
Sorry I was in vacations previous week.
I'll be happy to work with you and I agree that ECS could be a good approach.
I'll make available a fork to work on it.

Thanks, Regards,

[nicpenning]

Any updates on this? I would be happy to test.

[Tux-Panik]

Hi, unfortunately not on my side due to lack of time.
I just took the time, today, to upload the JSON analyzer structure: https://github.com/Tux-Panik/Cortex-Analyzers/tree/master/analyzers/ELK

Kind regards,
Julien

[vpiserchia]

Hello everyone,
please have a look to mine implementation here:

https://github.com/vpiserchia/CortexELKAnalyzer

any comments, PR is more than welcome

regards
vito

[lsoumille]

Hi,

I've implemented one if you are interested https://github.com/lsoumille/elasticsearch-analyzer

Regards,
Lucas

[vpiserchia]

@lsoumille I really like it as it provides a query field. I think we could join the effort, as they have both strengths and weeknesses:

mine Vs yours:

• is simpler in handling specific data types
• can handle different fields in the same flavour at once
• DOES NOT handle yet ssl/certs

yours Vs mine:

• has support for user's query
• it supports basic auth and ssl/certs

I think that I'll take inspiration from your work to support generic queries.

[DennisLund]

I have developed an ElasticSearch DHCP-Responder that might be of interest...
https://github.com/DennisLund/ElasticSearch-Cortex_Responder

This Responder is quite specialized, as it only fetches DHCP-assign and DHCP-Renew events within a specified timeframe, But I believe that it could be used as a template for other similar (specialized) responders.
This Responder can also perform automatic enrichment through TheHive API (create new observables, and edit the message-field of the observable that the responder was run from).

[Tux-Panik]

Dear all,

Thank you so much for your contributions.

@lsoumille: I had an issue configuring the analyzer in Cortex.
I switched the "multi" filed to false in "Elasticsearch_Query.json" not to have several unmodifiable rows.
"multi": false,

Then, I reached an error due to missing 'elasticsearch' dependencie:
Invalid output
Traceback (most recent call last):
File "elasticsearch-analyzer/elasticsearch_analyzer.py", line 5, in <module>
from elasticsearch import Elasticsearch
ImportError: No module named elasticsearch

@vpiserchia: I had some issues running the analyzer, probably because my Cortex Docker doesn't have 'elasticsearch' installed. Hereunder is the error message:
Invalid output
Traceback (most recent call last):
File "Elasticsearch/elk.py", line 4, in <module>
from elasticsearch import Elasticsearch
ModuleNotFoundError: No module named 'elasticsearch'

For both, unfortunatelly, as I don't have the default root password of the Docker image, I'm not able to install this requirement:
Collecting elasticsearch
Downloading https://files.pythonhosted.org/packages/ae/43
/38329621bcca6f0b97e1cc36fb3cef889414a1960fcdc83a41e26b496634/elasticsearch-7.0.2-
py2.py3-
none-any.whl (83kB)
100% |████████████████████████████████| 92kB 680kB/s
Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from elasticsearch) (1.24.1)
Installing collected packages: elasticsearch
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/sbin/.local'
Check the permissions.

I'll try to troubleshoot this permission issue next week.
Kind regards,
Julien

[vpiserchia]

@Tux-Panik
the analyzers comes with its own requirements file; you just need to pip install it after login, I only tested on python3, so something like this should help (provided you called the containers cortex):

docker cp /path/to/git/repo/CortexELKAnalyzer cortex:/opt/Cortex-Analyzers/analyzers/Elasticsearch

cd /opt/Cortex-Analyzers/analyzers/Elasticsearch
pip3 install -r requirements.txt

then for login, usually you can bash into it with something like:

docker exec -it -u root cortex /bin/bash

regards
/v

PS: I didn't check all the commands and there might be typos; I'm sure you can find guides online...

[nicpenning]

@lsoumille
I also tested yours. With everything as is the host field seemed to be broken:

I removed localhost from the field so I could put in the ElasticHost.

Unfortunately I also got a invalid output message.

What version of Cortex/TheHive does this work with? Also, what version of Elastic works?

Thanks!

Nic

[V1D1AN]

I have a problem with Domain.

It doesn't work when i do a search with toto1234.com , but if I search toto1234 ... it works.

I have change in elk.py:

        for field in fields:
            should.append({"term": {field: data}})
        if len(fields) > 1:

With:

        for field in fields:
            should.append({"match_phrase": {field: data}})
        if len(fields) > 1:

Now it works better.

[stiltz]

@lsoumille
I also tested yours. With everything as is the host field seemed to be broken:

I removed localhost from the field so I could put in the ElasticHost.

If you update multi from true to false in your json template (i.e.: Elasticsearch_NAME.json) and Elasticsearch_Query.json

{
      "name": "host",
      "description": "Elasticsearch Hostname",
      "type": "string",
      "multi": false,
      "required": true,
      "defaultValue": "localhost"
    },

Then you can input a host and it seems to work on 3.0.0-RC4

[nicpenning]

Just curious if anyone has been using this or have some updated versions. Still looking to get an analyzer for ElasticSearch queries against the different observables.

[LaZyDK]

I just pushed this analyzer to Github. That might be of use for all of you. It is based on the IP analyzer from @vpiserchia with some additions from @lsoumille. I rewrote the results and made short and long reports for - in my case - vulnerability data using some ECS contributed by @nicpenning.
https://github.com/LaZyDK/TheHive-Cortex-Analyzers/tree/master/Vulnerabilities

[Tux-Panik]

Thanks all for your contributions.
Sounds good.