Add Recorded Future support to threatintel module (#26481)

This adds a new fileset, `recordedfuture`, to the treatintel module. It
ingests indicators via the Recorded Future Connect API.

(cherry picked from commit 6d89566)

CHANGELOG.next.asciidoc

@@ -595,6 +595,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 - Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
 - Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
+- Added dataset `recordedfuture` to the `threatintel` module to ingest indicators from Recorded Future Connect API {pull}26481[26481]
 
 *Heartbeat*
 

filebeat/docs/fields.asciidoc

@@ -145918,6 +145918,17 @@ example: Montreal
 
 --
 
+*`threatintel.indicator.geo.continent_name`*::
++
+--
+Name of the continent.
+
+type: keyword
+
+example: North America
+
+--
+
 *`threatintel.indicator.geo.country_iso_code`*::
 +
 --
@@ -147353,6 +147364,166 @@ type: keyword
 
 --
 
+[float]
+=== recordedfuture
+
+Fields for Recorded Future Threat Intel
+
+
+
+[float]
+=== entity
+
+Entity that represents a threat.
+
+
+
+*`threatintel.recordedfuture.entity.id`*::
++
+--
+Entity ID.
+
+
+type: keyword
+
+example: ip:192.0.2.13
+
+--
+
+*`threatintel.recordedfuture.entity.name`*::
++
+--
+Entity name. Value for the entity.
+
+
+type: keyword
+
+example: 192.0.2.13
+
+--
+
+*`threatintel.recordedfuture.entity.type`*::
++
+--
+Entity type.
+
+
+type: keyword
+
+example: IpAddress
+
+--
+
+*`threatintel.recordedfuture.intelCard`*::
++
+--
+Link to the Recorded Future Intelligence Card for to this indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.recordedfuture.ip_range`*::
++
+--
+Range of IPs for this indicator.
+
+
+type: ip_range
+
+example: 192.0.2.0/16
+
+--
+
+[float]
+=== risk
+
+Risk fields.
+
+
+
+*`threatintel.recordedfuture.risk.criticality`*::
++
+--
+Risk criticality (0-4).
+
+
+type: byte
+
+--
+
+*`threatintel.recordedfuture.risk.criticalityLabel`*::
++
+--
+Risk criticality label. One of None, Unusual, Suspicious, Malicious, Very Malicious.
+
+
+type: keyword
+
+--
+
+*`threatintel.recordedfuture.risk.evidenceDetails`*::
++
+--
+Risk's evidence details.
+
+
+type: flattened
+
+--
+
+*`threatintel.recordedfuture.risk.score`*::
++
+--
+Risk score (0-99).
+
+
+type: short
+
+--
+
+*`threatintel.recordedfuture.risk.riskString`*::
++
+--
+Number of Risk Rules observed as a factor of total number of rules.
+
+
+type: keyword
+
+example: 1/54
+
+--
+
+*`threatintel.recordedfuture.risk.riskSummary`*::
++
+--
+Risk summary.
+
+
+type: keyword
+
+example: 1 of 54 Risk Rules currently observed.
+
+--
+
+*`threatintel.recordedfuture.risk.riskSummary.text`*::
++
+--
+type: text
+
+--
+
+*`threatintel.recordedfuture.risk.rules`*::
++
+--
+Number of rules observed.
+
+
+type: long
+
+--
+
 [[exported-fields-tomcat]]
 == Apache Tomcat fields
 

filebeat/docs/modules/threatintel.asciidoc

@@ -30,6 +30,7 @@ fields.
 * <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
 * <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
 * <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
+* <<recordedfuture,recordedfuture>>: Supports gathering threat intel attributes from Recorded Future.
 
 include::../include/gs-link.asciidoc[]
 
@@ -224,7 +225,7 @@ How often the API is polled for updated information.
 
 *`var.first_interval`*::
 
-How far back to search when retrieving events the first time the beat starts up.
+How far back to search when retrieving events the first time {beatname_uc} starts up.
 After the first interval has passed the module itself will use the timestamp
 from the last response as the filter when retrieving new events.
 
@@ -298,7 +299,7 @@ How often the API is polled for updated information.
 
 *`var.first_interval`*::
 
-How far back to search when retrieving events the first time the beat starts up.
+How far back to search when retrieving events the first time the {beatname_uc} starts up.
 After the first interval has passed the module itself will use the timestamp
 from the last response as the filter when retrieving new events.
 
@@ -410,7 +411,7 @@ Anomali Threat Intel is mapped to the following ECS fields.
 
 To configure the ThreatStream integration you first need to define an output
 in the Anomali ThreatStream Integrator using the Elastic SDK provided by Anomali.
-It will deliver indicators via HTTP or HTTPS to a Filebeat instance running as
+It will deliver indicators via HTTP or HTTPS to a {beatname_uc} instance running as
 a server.
 
 Configure an Integrator output with the following settings:
@@ -420,12 +421,12 @@ Configure an Integrator output with the following settings:
   Adjust the paths to the python executable and the directory where the Elastic SDK
   has been unpacked.
 * Metadata in JSON Format: `{"url": "https://filebeat:8080/", "server_certificate": "/path/to/cert.pem", "secret": "my secret"}`.
-    - `url`: Use the host and port where Filebeat will be running, and `http` or `https` accordingly.
+    - `url`: Use the host and port where {beatname_uc} will be running, and `http` or `https` accordingly.
     - `server_certificate`: If using HTTPS, absolute path to the server certificate. Otherwise don't set
         this field.
-    - `secret`: A shared secret string to authenticate messages between the SDK and Filebeat.
+    - `secret`: A shared secret string to authenticate messages between the SDK and {beatname_uc}.
 
-Then configure the `anomalithreatstream` fileset in Filebeat accordingly:
+Then configure the `anomalithreatstream` fileset in {beatname_uc} accordingly:
 [source,yaml]
 ----
 - module: threatintel
@@ -450,11 +451,11 @@ Port number to use for the HTTP server.
 
 *`var.secret`*::
 
-Shared secret between the SDK and Filebeat, used to authenticate messages.
+Shared secret between the SDK and {beatname_uc}, used to authenticate messages.
 
 *`var.ssl_certificate`*::
 
-Path to the public SSL certificate for the HTTPS server. If unset, Filebeat
+Path to the public SSL certificate for the HTTPS server. If unset, {beatname_uc}
 will use unsecure HTTP connections.
 
 *`var.ssl_key`*::
@@ -489,6 +490,94 @@ Anomali ThreatStream fields are mapped to the following ECS fields:
 [[a]]
 [small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomalithreatstream`.#
 
+[[recordedfuture]]
+[float]
+==== `recordedfuture` fileset settings
+
+The `recordedfuture` fileset fetches intelligence from the Recorded Future Connect API.
+It supports `domain`, `hash`, `ip` and `url` data types.
+
+To enable it you need to define the URL to fetch data from. You can construct this URL
+using the https://api.recordedfuture.com/index.html[Recorded Future API Explorer.] The URL
+must point to the `/search` endpoint and contain a suitable `limit`
+(how many records to return from a single request) and `fields` parameters.
+The `entity` and `timestamps` fields are required.
+
+Sample configuration:
+[source,yaml]
+----
+- module: threatintel
+  recordedfuture:
+    enabled: true
+    var.input: httpjson
+    var.interval: 5m
+    var.first_interval: 168h
+    var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
+    var.api_token: "<RF_TOKEN>"
+----
+
+To fetch threat intelligence from multiple data types, you must define more than
+one instance of the module:
+[source,yaml]
+----
+- module: threatintel
+  recordedfuture:
+    enabled: true
+    var.input: httpjson
+    var.interval: 5m
+    var.first_interval: 168h
+    var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
+    var.api_token: "<RF_TOKEN>"
+- module: threatintel
+  recordedfuture:
+    enabled: true
+    var.input: httpjson
+    var.interval: 1m
+    var.first_interval: 168h
+    var.url: "https://api.recordedfuture.com/v2/hash/search?limit=200&fields=entity,fileHashes,timestamps,risk,intelCard,location&metadata=false"
+    var.api_token: "<RF_TOKEN>"
+----
+
+*`var.url`*::
+
+The URL of the API endpoint to connect with.
+
+*`var.api_token`*::
+
+The API token used to access Recorded Future API.
+
+*`var.interval`*::
+
+How often the API is polled for updated information.
+
+*`var.first_interval`*::
+
+How far back to search when retrieving events the first time {beatname_uc} starts up.
+After the first interval has passed the module itself will use the timestamp
+from the last response as the filter when retrieving new events.
+
+*`var.proxy_url`*::
+
+Optional URL to use as HTTP proxy.
+
+
+Recorded Future fields are mapped to the following ECS fields:
+
+[options="header"]
+|=============================================================
+| Recorded Future fields | ECS Fields
+| entity.name            | threatintel.indicator.{url,ip,domain,file.hash}
+| entity.type            | threatintel.indicator.type
+| fileHashes             | threatintel.indicator.file.hash
+| intelCard              | event.reference
+| location.asn           | threatintel.indicator.as.number
+| location.location      | threatintel.indicator.geo
+| location.organization  | threatintel.indicator.as.organization.name
+| risk.score             | event.risk_score
+| timestamps.firstSeen   | threatintel.indicator.first_seen
+| timestamps.lastSeen    | threatintel.indicator.last_seen
+|=============================================================
+
 :has-dashboards!:
 
 [float]

filebeat/tests/system/test_modules.py

@@ -287,6 +287,7 @@ def clean_keys(obj):
         "threatintel.anomali",
         "threatintel.anomalithreatstream",
         "threatintel.malwarebazaar",
+        "threatintel.recordedfuture",
         "snyk.vulnerabilities",
         "snyk.audit",
         "awsfargate.log",

x-pack/filebeat/filebeat.reference.yml

@@ -2274,6 +2274,38 @@ filebeat.modules:
     # var.ssl_certificate: path/to/server_ssl_cert.pem
     # var.ssl_key: path/to/ssl_key.pem
 
+  recordedfuture:
+    enabled: true
+
+    # Input used for ingesting threat intel data
+    var.input: httpjson
+
+    # The interval to poll the API for updates
+    var.interval: 5m
+
+    # How far back in time to start fetching intelligence when run for the
+    # first time. Value must be in hours. Default: 168h (1 week).
+    var.first_interval: 168h
+
+    # The URL used for Threat Intel API calls.
+    # Must include the `limit` parameter and at least `entity` and `timestamps` fields.
+    # See the Connect API Explorer for a list of possible parameters.
+    #
+    # For `ip` entities:
+    var.url: "https://api.recordedfuture.com/v2/ip/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
+
+    # For `domain` entities:
+    # var.url: "https://api.recordedfuture.com/v2/domain/search?limit=200&fields=entity,timestamps,risk,intelCard,location&metadata=false"
+
+    # For `hash` entities:
+    # var.url: "https://api.recordedfuture.com/v2/hash/search?limit=200&fields=entity,fileHashes,timestamps,risk,intelCard,location&metadata=false"
+
+    # For `url` entities:
+    # var.url: "https://api.recordedfuture.com/v2/url/search?limit=200&fields=entity,timestamps,risk&metadata=false"
+
+    # Set your API Token.
+    var.api_token: "<RF_TOKEN>"
+
 #---------------------------- Apache Tomcat Module ----------------------------
 - module: tomcat
   log: