[Shadowserver] Import Shadowserver Connector

external-import/shadowserver/.env.sample

@@ -0,0 +1,21 @@
+OPENCTI_URL=http://opencti:8080
+# Use one as provided by the platform
+OPENCTI_TOKEN=
+# Generate one with uuidgen
+CONNECTOR_ID=
+# From 0 (Unknown) to 100 (Fully trusted)
+CONNECTOR_CONFIDENCE_LEVEL=100 
+# One of the following: debug, info, warning, error
+CONNECTOR_LOG_LEVEL=info
+CONNECTOR_RUN_EVERY=1d
+CONNECTOR_UPDATE_EXISTING_DATA=false
+CONNECTOR_TYPE=EXTERNAL_IMPORT
+CONNECTOR_SCOPE=stix2
+CONNECTOR_NAME=Shadowserver
+# Custom connector parameters.
+SHADOWSERVER_API_KEY=CHANGEME
+SHADOWSERVER_API_SECRET=CHANGEME
+SHADOWSERVER_MARKING=TLP:CLEAR
+SHADOWSERVER_CREATE_INCIDENT=true
+SHADOWSERVER_INCIDENT_SEVERITY=low
+SHADOWSERVER_INCIDENT_PRIORITY=P4

external-import/shadowserver/Dockerfile

@@ -0,0 +1,15 @@
+FROM python:3.11-alpine
+
+# Install Python modules
+RUN apk --no-cache add git build-base libmagic libffi-dev libxml2-dev libxslt-dev
+COPY requirements.txt /tmp/requirements.txt
+RUN pip3 install --no-cache-dir -r /tmp/requirements.txt
+
+# Copy the connector
+COPY src /opt/connector
+WORKDIR /opt/connector
+
+# Expose and entrypoint
+COPY entrypoint.sh /
+RUN chmod +x /entrypoint.sh
+ENTRYPOINT ["/entrypoint.sh"]

external-import/shadowserver/README.md

@@ -0,0 +1,93 @@
+# Shadowserver Connector
+
+The integration uses Shadowservers reports API to query the available Shadowserver reports and transform them into Stix objects making them available within OpenCTI. All available reports are downloaded and an `Artifact` object is created with the original file. Stix `Note` objects are added to both the `Report` and the `CustomObjectCaseIncident` with a mark-down rendition of each finding from the report.
+
+API and report references from The Shadowserver Foundation
+ - https://github.com/The-Shadowserver-Foundation/api_utils/wiki/API:-Reports-Query
+ - https://interchange.shadowserver.org/schema/reports.json
+
+ The integration creates the following types of Stix objects and relationships between them.
+ - Artifact
+ - AutonomousSystem
+ - CustomObjectCaseIncident (optional)
+ - DomainName
+ - Identity
+ - IPv4Address
+ - IPv6Address
+ - MACAddress
+ - MarkingDefinition
+ - NetworkTraffic
+ - Note
+ - ObservedData
+ - Report
+ - Vulnerability
+ - X509Certificate
+
+On the initial run, the integration defaults to the last 30-days of reports. Every run after that, it provides an update for the last 3-days. 
+
+## Installation
+
+### Requirements
+
+- OpenCTI Platform >= 6.1.12
+
+### Configuration
+
+Configuration parameters are provided using environment variables as described below.
+Some of them are placed directly in the `docker-compose.yml` since they are not expected to be modified by final users once that they have been defined by the developer of the connector.
+
+Note that the values that follow can be grabbed within Python code using `self.helper.{PARAMETER}`, i. e., `self.helper.connector_nane`.
+
+Expected environment variables to be set in the  `docker-compose.yml` that describe the connector itself.
+Most of the times, these values are NOT expected to be changed.
+
+| Parameter                            | Docker envvar                       | Mandatory    | Description                                                                                                                                                |
+| ------------------------------------ | ----------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `connector_type`                     | `CONNECTOR_TYPE`                    | Yes          | Must be `EXTERNAL_IMPORT` (this is the connector type).                                                                                                    |
+| `connector_name`                     | `CONNECTOR_NAME`                    | Yes          | A connector name to be shown in OpenCTI.                                                                                                                   |
+| `connector_scope`                    | `CONNECTOR_SCOPE`                   | Yes          | Supported scope. E. g., `text/html`.                                                                                                                       |
+
+However, there are other values which are expected to be configured by end users.
+The following values are expected to be defined in the `.env` file.
+This file is included in the `.gitignore` to avoid leaking sensitive date). 
+Note tha the `.env.sample` file can be used as a reference.
+
+The ones that follow are connector's generic execution parameters expected to be added for export connectors.
+
+| Parameter                            | Docker envvar                       | Mandatory    | Description                                                                                                                                                |
+| ------------------------------------ | ----------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `opencti_url`                        | `OPENCTI_URL`                       | Yes          | The URL of the OpenCTI platform. Note that final `/` should be avoided. Example value: `http://opencti:8080`                                               |
+| `opencti_token`                      | `OPENCTI_TOKEN`                     | Yes          | The default admin token configured in the OpenCTI platform parameters file.                                                                                |
+| `connector_id`                       | `CONNECTOR_ID`                      | Yes          | A valid arbitrary `UUIDv4` that must be unique for this connector.                                                                                         |
+| `connector_confidence_level`         | `CONNECTOR_CONFIDENCE_LEVEL`        | Yes          | The default confidence level for created sightings (a number between 1 and 4).                                                                             |
+| `connector_log_level`                | `CONNECTOR_LOG_LEVEL`               | Yes          | The log level for this connector, could be `debug`, `info`, `warn` or `error` (less verbose).                                                              |
+| `interval`                           | `CONNECTOR_RUN_EVERY`               | Yes          | The time unit is represented by a single character at the end of the string: d for days, h for hours, m for minutes, and s for seconds. e.g., 30s is 30 seconds. 1d is 1 day.                                                                                  |
+| `update_existing_data`               | `CONNECTOR_UPDATE_EXISTING_DATA`    | Yes          | Whether to update known existing data.                                                                                                                     |
+
+
+Finally, the ones that follow are connector's specific execution parameters expected to be used by this connector.
+
+| Parameter                            | Docker envvar                       | Mandatory    | Description                                                                                                                                                |
+| ------------------------------------ | ----------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `shadowserver_api_key`               | `SHADOWSERVER_API_KEY`              | Yes          | The API key for Shadowserver.                                                                                                                              |
+| `shadowserver_api_secret`            | `SHADOWSERVER_API_SECRET`           | Yes          | The API secret for Shadowserver.                                                                                                                           |
+| `shadowserver_marking`               | `SHADOWSERVER_MARKING`              | Yes          | The marking for the data, e.g., `TLP:CLEAR`, `TLP:GREEN`, `TLP:AMBER`, `TLP:RED`.                                                                                                               |
+| `shadowserver_create_incident`       | `SHADOWSERVER_CREATE_INCIDENT`      | Yes          | Whether to create an incident (`true` or `false`).                                                                                                         |
+| `shadowserver_incident_severity`     | `SHADOWSERVER_INCIDENT_SEVERITY`    | Yes          | The severity of the incident, e.g., `low` (Default: `low`).                                                                                                                 |
+| `shadowserver_incident_priority`     | `SHADOWSERVER_INCIDENT_PRIORITY`    | Yes          | The priority of the incident, e.g., `P4` (Default: `P4`).   
+
+### Debugging ###
+
+The connector can be debugged by setting the appropiate log level.
+Note that logging messages can be added using `self.helper.log_{LOG_LEVEL}("Sample message")`, i. e., `self.helper.log_error("An error message")`.
+
+<!-- Any additional information to help future users debug and report detailed issues concerning this connector -->
+
+### Additional information
+
+<!--
+Any additional information about this connector
+* What information is ingested/updated/changed
+* What should the user take into account when using this connector
+* ...
+-->

external-import/shadowserver/docker-compose.yml

@@ -0,0 +1,30 @@
+version: '3'
+services:
+  connector:
+    build: .
+    container_name: ${CONTAINER_NAME}
+    environment:
+      # Connector's definition parameters:
+      - CONNECTOR_TYPE=${CONNECTOR_TYPE}
+      - CONNECTOR_NAME=${CONNECTOR_NAME}
+      - CONNECTOR_SCOPE=${CONNECTOR_SCOPE}
+      # Connector's generic execution parameters:
+      - OPENCTI_URL=${OPENCTI_URL}
+      - OPENCTI_TOKEN=${OPENCTI_TOKEN}
+      - CONNECTOR_ID=${CONNECTOR_ID}
+      - CONNECTOR_CONFIDENCE_LEVEL=${CONNECTOR_CONFIDENCE_LEVEL} # From 0 (Unknown) to 100 (Fully trusted).
+      - CONNECTOR_LOG_LEVEL=${CONNECTOR_LOG_LEVEL}
+      - CONNECTOR_RUN_EVERY=${CONNECTOR_RUN_EVERY}
+      # Connector's custom execution parameters:
+      - SHADOWSERVER_API_KEY=${SHADOWSERVER_API_KEY}
+      - SHADOWSERVER_API_SECRET=${SHADOWSERVER_API_SECRET}
+      - SHADOWSERVER_MARKING=${SHADOWSERVER_MARKING}
+      - SHADOWSERVER_CREATE_INCIDENT=${SHADOWSERVER_CREATE_INCIDENT}
+      - SHADOWSERVER_INCIDENT_SEVERITY=${SHADOWSERVER_INCIDENT_SEVERITY}
+      - SHADOWSERVER_INCIDENT_PRIORITY=${SHADOWSERVER_INCIDENT_PRIORITY}
+    restart: always
+
+networks:
+  default:
+    external: true
+    name: docker_default

external-import/shadowserver/entrypoint.sh

@@ -0,0 +1,4 @@
+#!/bin/sh
+
+# Start the connector (WORKDIR is /opt/connector as set in the Dockerfile)
+python3 main.py

external-import/shadowserver/requirements.txt

@@ -0,0 +1,4 @@
+pycti==6.1.12
+requests
+pandas
+tabulate # Required for Pandas to Markdown

external-import/shadowserver/src/lib/external_import.py

@@ -0,0 +1,178 @@
+import os
+import sys
+import time
+from datetime import UTC, datetime
+
+import stix2
+from pycti import OpenCTIConnectorHelper
+
+
+class ExternalImportConnector:
+    """Specific external-import connector
+
+    This class encapsulates the main actions, expected to be run by the
+    any external-import connector. Note that the attributes defined below
+    will be complemented per each connector type.
+
+    Attributes:
+        helper (OpenCTIConnectorHelper): The helper to use.
+        interval (str): The interval to use. It SHOULD be a string in the format '7d', '12h', '10m', '30s' where the final letter SHOULD be one of 'd', 'h', 'm', 's' standing for day, hour, minute, second respectively.
+        update_existing_data (str): Whether to update existing data or not in OpenCTI.
+    """
+
+    def __init__(self):
+        self.helper = OpenCTIConnectorHelper({})
+
+        # Specific connector attributes for external import connectors
+        try:
+            self.interval = os.environ.get("CONNECTOR_RUN_EVERY", None).lower()
+            self.helper.log_info(
+                f"Verifying integrity of the CONNECTOR_RUN_EVERY value: '{self.interval}'"
+            )
+            unit = self.interval[-1]
+            if unit not in ["d", "h", "m", "s"]:
+                raise TypeError
+            int(self.interval[:-1])
+        except TypeError as _:
+            msg = f"Error ({_}) when grabbing CONNECTOR_RUN_EVERY environment variable: '{self.interval}'. It SHOULD be a string in the format '7d', '12h', '10m', '30s' where the final letter SHOULD be one of 'd', 'h', 'm', 's' standing for day, hour, minute, second respectively. "
+            self.helper.log_error(msg)
+            raise ValueError(msg)
+
+        update_existing_data = os.environ.get("CONNECTOR_UPDATE_EXISTING_DATA", "false")
+        if isinstance(update_existing_data, str) and update_existing_data.lower() in [
+            "true",
+            "false",
+        ]:
+            self.update_existing_data = (
+                True if update_existing_data.lower() == "true" else False
+            )
+        elif isinstance(update_existing_data, bool) and update_existing_data.lower in [
+            True,
+            False,
+        ]:
+            self.update_existing_data = update_existing_data
+        else:
+            msg = f"Error when grabbing CONNECTOR_UPDATE_EXISTING_DATA environment variable: '{update_existing_data}'. It SHOULD be either `true` or `false`. `false` is assumed. "
+            self.helper.log_warning(msg)
+            self.update_existing_data = "false"
+
+    def _collect_intelligence(self) -> list:
+        """Collect intelligence from the source"""
+        raise NotImplementedError
+
+    def _get_interval(self) -> int:
+        """Returns the interval to use for the connector
+
+        This SHOULD return always the interval in seconds. If the connector is execting that the parameter is received as hoursUncomment as necessary.
+        """
+        unit = self.interval[-1:]
+        value = self.interval[:-1]
+
+        try:
+            if unit == "d":
+                # In days:
+                return int(value) * 60 * 60 * 24
+            elif unit == "h":
+                # In hours:
+                return int(value) * 60 * 60
+            elif unit == "m":
+                # In minutes:
+                return int(value) * 60
+            elif unit == "s":
+                # In seconds:
+                return int(value)
+        except Exception as e:
+            self.helper.log_error(
+                f"Error when converting CONNECTOR_RUN_EVERY environment variable: '{self.interval}'. {str(e)}"
+            )
+            raise ValueError(
+                f"Error when converting CONNECTOR_RUN_EVERY environment variable: '{self.interval}'. {str(e)}"
+            )
+
+    def run(self) -> None:
+        # Main procedure
+        self.helper.log_info(f"Starting {self.helper.connect_name} connector...")
+        while True:
+            try:
+                # Get the current timestamp and check
+                timestamp = int(time.time())
+                current_state = self.helper.get_state()
+                if current_state is not None and "last_run" in current_state:
+                    last_run = current_state["last_run"]
+                    self.helper.log_info(
+                        f"{self.helper.connect_name} connector last run @ {datetime.fromtimestamp(last_run, tz=UTC).isoformat()}"
+                    )
+                else:
+                    last_run = None
+                    self.helper.log_info(
+                        f"{self.helper.connect_name} connector has never run"
+                    )
+
+                # If the last_run is more than interval-1 day
+                if last_run is None or ((timestamp - last_run) >= self._get_interval()):
+                    self.helper.log_info(f"{self.helper.connect_name} will run!")
+                    friendly_name = f"{self.helper.connect_name} run @ {datetime.fromtimestamp(timestamp, tz=UTC).isoformat()}"
+                    work_id = self.helper.api.work.initiate_work(
+                        self.helper.connect_id, friendly_name
+                    )
+
+                    try:
+                        # Performing the collection of intelligence
+                        bundle_objects = self._collect_intelligence()
+                        bundle = stix2.Bundle(
+                            objects=bundle_objects, allow_custom=True
+                        ).serialize()
+
+                        self.helper.log_info(
+                            f"Sending {len(bundle_objects)} STIX objects to OpenCTI..."
+                        )
+                        self.helper.send_stix2_bundle(
+                            bundle,
+                            update=self.update_existing_data,
+                            work_id=work_id,
+                        )
+                    except Exception as e:
+                        self.helper.log_error(str(e))
+
+                    # Store the current timestamp as a last run
+                    message = (
+                        f"{self.helper.connect_name} connector successfully run, storing last_run as "
+                        + str(timestamp)
+                    )
+                    self.helper.log_info(message)
+
+                    self.helper.log_debug(
+                        f"Grabbing current state and update it with last_run: {timestamp}"
+                    )
+                    current_state = self.helper.get_state()
+                    if current_state:
+                        current_state["last_run"] = timestamp
+                    else:
+                        current_state = {"last_run": timestamp}
+                    self.helper.set_state(current_state)
+
+                    self.helper.api.work.to_processed(work_id, message)
+                    self.helper.log_info(
+                        "Last_run stored, next run in: "
+                        + str(round(self._get_interval() / 60 / 60, 2))
+                        + " hours"
+                    )
+                else:
+                    new_interval = self._get_interval() - (timestamp - last_run)
+                    self.helper.log_info(
+                        f"{self.helper.connect_name} connector will not run, next run in: "
+                        + str(round(new_interval / 60 / 60, 2))
+                        + " hours"
+                    )
+
+            except (KeyboardInterrupt, SystemExit):
+                self.helper.log_info(f"{self.helper.connect_name} connector stopped")
+                sys.exit(0)
+            except Exception as e:
+                self.helper.log_error(str(e))
+
+            if self.helper.connect_run_and_terminate:
+                self.helper.log_info(f"{self.helper.connect_name} connector ended")
+                sys.exit(0)
+
+            time.sleep(60)