[filebeat] VirusTotal Livehunt dataset - WIP

CHANGELOG-developer.next.asciidoc

@@ -60,6 +60,7 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
 
 ==== Added
 
+- Add VirusTotal Intelligence Live Hunt module. {issue}21541[21541] {pull}21815[21815]
 - Add configuration for APM instrumentation and expose the tracer trough the Beat object. {pull}17938[17938]
 - Make the behavior of clientWorker and netClientWorker consistent when error is returned from publisher pipeline
 - Metricset generator generates beta modules by default now. {pull}10657[10657]

CHANGELOG.next.asciidoc

@@ -37,6 +37,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Filebeat*
 
+- Introduce VirusTotal Intelligence Live Hunt module. {issue}21541[21541] {pull}21815[21815]
 - Improve ECS field mappings in panw module.  event.outcome now only contains success/failure per ECS specification. {issue}16025[16025] {pull}17910[17910]
 - Improve ECS categorization field mappings for nginx module. http.request.referrer only populated when nginx sets a value {issue}16174[16174] {pull}17844[17844]
 - Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . {issue}16180[16180] {pull}17982[17982]

x-pack/filebeat/filebeat.reference.yml

@@ -1645,6 +1645,16 @@ filebeat.modules:
     # can be added under this section.
     #input:
 
+#------------------------------ VirusTotal Module ------------------------------
+- module: virustotal
+  # All logs
+  livehunt:
+    enabled: true
+    # Set the VirusTotal private API key
+    var.apikey: ""
+    # Set retrieval limit, maximum 40, default 10
+    var.limit: 10
+
 #--------------------------------- Zeek Module ---------------------------------
 - module: zeek
   capture_loss:

x-pack/filebeat/module/virustotal/README.md

@@ -0,0 +1,101 @@
+# Development Workflow
+
+In all examples below, `filebeat.dev.yml` is a local development only configuration with VT API key, creds to Elastic Cloud instance, etc.
+
+My `filebeat.dev.yml` looks like this, with appropriate substitutions.
+
+```yml
+filebeat.modules:
+  - module: virustotal
+    livehunt:
+      enabled: True
+      var.input: httpjson   # httpjson or kafka
+
+      # If consuming events from VirusTotal httpjson
+      var.api_key: INSERT-YOUR-VT-API-KEY
+      var.vt_fileinfo_url: https://www.virustotal.com/gui/file/
+      var.limit: 40 # maximum 40, default is 10
+
+      # If consuming raw events from Kafka
+      var.kafka_brokers:
+        - 127.0.0.1:9093
+      var.kafka_topics:
+         - virustotal.raw
+
+cloud.id: "INSERT-YOUR-CLOUD-ID"
+cloud.auth: "INSERT-YOUR-CLOUD-AUTH"
+```
+
+## Setup
+
+Run filebeat setup to establish ILM policies, Elasticsearch mapping patterns, Kibana index patterns. This needs to happen every time you change a `fields.yml`. This configuration will overwrite existing items, except maybe Kibana index pattern. I just manually delete that before I run setup. I also delete the existing `filebeat-*` index in Elasticsearch too, to avoid mapping conflicts. See notes on `kafka` and `elastidump` below.
+
+```shell
+./filebeat -c filebeat.dev.yml -e setup -E setup.template.overwrite=true -E setup.ilm.overwrite=true
+```
+
+## Kafka
+
+I added Kafka as an input type because I think some users will find this useful and it's incredibly useful to replay events from VT for development purposes. I used `docker-compose` to standup a local Kafka cluster for development purposes.
+
+```shell
+# Download my compose file
+curl -O https://gist.githubusercontent.com/dcode/a79d24624aee11ca713250cc5ba02a22/raw/e519b85bad45b3a2f757fbdc2f9808c94969cf13/docker-compose.yml
+
+# Bring up cluster
+docker-compose up -d
+```
+
+Configure filebeat to use this as an input for the module in your `filebeat.dev.yml`, where `virustotal.raw` is the topic name of the unmodified LiveHunt notification file objects.
+
+```yaml
+filebeat.modules:
+  - module: virustotal
+    livehunt:
+      enabled: True
+      var.input: kafka
+      var.kafka_brokers:
+        - 127.0.0.1:9093
+      var.kafka_topics:
+         - virustotal.raw
+```
+
+## Replay Events using Kafka
+
+First, save off existing events from the cluster. Do this before you delete the index in the **setup** step above.
+
+```shell
+# Install elasticdump, uses npm/node.js
+npm install elasticdump -g
+
+# Install kafkacat and jq
+brew install kafkacat jq
+
+# Dump the filebeat index
+elasticdump --input=https://elastic:[email protected]:9243/filebeat-* \
+  --output=$ \
+  | gzip > data/filebeat-virustotal.json.gz
+
+# Replay filebeat data into kafka topic (if setup using compose file above)
+gzcat data/filebeat-virustotal.json.gz | jq -cr '._source.event.original' | kafkacat -b 127.0.0.1:9093 -P -t virustotal.raw
+```
+
+NOTES:
+
+- `https://elastic:[email protected]:9243`: is the HTTPS endpoint as retreived from the Elastic Cloud panel, with the `elastic` username and password prefixed to the server. You can optionally use an HTTP auth ini file with `elasticdump`. See the `--help` output for specifics.
+- Elasticdump can do a lot of things. In this scenario, I'm merely compressing it and writing it to a disk. These are the raw JSON documents as stored in Elasticsearch.
+- I'm dumping only indices that match the pattern `filebeat-*`. You can make this more or less specific, as desired.
+- I'm using `jq` here to output compact (single line) JSON documents as raw strings, which unquotes the field value.
+- kafkacat here is connecting to the local broker `-b`, in producer mode `-P`, and writing to the topic `-t` using data from stdin.
+
+If you wanted to validate or otherwise manipulate the raw data, you can use `kafkacat` in consumer mode `-C`. This example shows the first 10 records. You could pipe these to `jq` to format them.
+
+```shell
+kafkacat -b 127.0.0.1:9093 -C -t virustotal.raw | head | jq
+```
+
+Configure filebeat to use the kafka input as show above, and run it until all events are replayed. After which, you can switch back to `httpjson` as the input type and stream new data.
+
+```shell
+./filebeat -c filebeat.dev.yml -e
+```

x-pack/filebeat/module/virustotal/_meta/config.yml

@@ -0,0 +1,8 @@
+- module: virustotal
+  # All logs
+  livehunt:
+    enabled: true
+    # Set the VirusTotal private API key
+    var.apikey: ""
+    # Set retrieval limit, maximum 40, default 10
+    var.limit: 10

x-pack/filebeat/module/virustotal/_meta/fields.yml

@@ -0,0 +1,7 @@
+#spellchecker: disable
+- key: virustotal
+  title: VirusTotal
+  description: >
+    Module for handling the VirusTotal API notifications
+  fields:
+