[FR] Re-factor Build Integrations Manifest

[terrancedejesus]

Issues

• [FR] Update Build Integrations Logic to Reference Metadata Tags #2273
• [Enhancement] Review "integrations-pr" CLI Command for Enhancements and Package Storage v2 #2244

Overview

We should look to adjust the way we determine which integrations to reference for building the integrations manifest.

Problem

At this time, we reference the rules/integrations folder names to determine which integrations we use to build the integrations manifest. This becomes an issue, for instance, if a core OS is using the endpoint security integration or a network rule is using the NPC integration.

Solution

We should load all rules and reference the metadata tags to create a set list of integrations so the building process is resilient to the location of a rule. In addition to this we need to account for elastic/kibana#139440 where Kibana would identify a ^ in the field value and know this is the least compatible. Additionally, we need to filter for the latest major for this comparison logic.

• Pull integration list from rule.contents.metadata.integration
• Filter integrations_manifests in integrations.find_least_compatible_version to only return latest major
• Adjust integrations.find_least_compatible_version to return version with a ^

View Rule Example:

{
  "author": [
    "Elastic"
  ],
  "description": "Assigning the administrative role to a user will grant them access to the Google Admin console and grant them administrator privileges which allow them to access and manage various resources and applications. An adversary may create a new administrator account for persistence or apply the admin role to an existing user to carry out further intrusion efforts. Users with super-admin privileges can bypass single-sign on if enabled in Google Workspace.",
  "false_positives": [
    "Google Workspace admin role assignments may be modified by system administrators. Verify that the configuration change was expected. Exceptions can be added to this rule to filter expected behavior."
  ],
  "from": "now-130m",
  "index": [
    "filebeat-*",
    "logs-google_workspace*"
  ],
  "interval": "10m",
  "language": "kuery",
  "license": "Elastic License v2",
  "name": "Google Workspace Admin Role Assigned to a User",
  "note": "### Important Information Regarding Google Workspace Event Lag Times\n- As per Google's documentation, Google Workspace administrators may observe lag times ranging from minutes up to 3 days between the time of an event's occurrence and the event being visible in the Google Workspace admin/audit logs.\n- This rule is configured to run every 10 minutes with a lookback time of 130 minutes.\n- To reduce the risk of false negatives, consider reducing the interval that the Google Workspace (formerly G Suite) Filebeat module polls Google's reporting API for new events.\n- By default, `var.interval` is set to 2 hours (2h). Consider changing this interval to a lower value, such as 10 minutes (10m).\n- See the following references for further information:\n  - https://support.google.com/a/answer/7061566\n  - https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-google_workspace.html",
  "query": "event.dataset:\"google_workspace.admin\" and event.category:\"iam\" and event.action:\"ASSIGN_ROLE\"\n  and google_workspace.event.type:\"DELEGATED_ADMIN_SETTINGS\" and google_workspace.admin.role.name : *_ADMIN_ROLE\n",
  "references": [
    "https://support.google.com/a/answer/172176?hl=en"
  ],
  "related_integrations": [
    {
      "package": "google_workspace",
      "version": "^1.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.category",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.dataset",
      "type": "keyword"
    },
    {
      "ecs": false,
      "name": "google_workspace.admin.role.name",
      "type": "keyword"
    },
    {
      "ecs": false,
      "name": "google_workspace.event.type",
      "type": "keyword"
    }
  ],
  "risk_score": 73,
  "rule_id": "68994a6c-c7ba-4e82-b476-26a26877adf6",
  "setup": "The Google Workspace Fleet integration, Filebeat module, or similarly structured data is required to be compatible with this rule.",
  "severity": "high",
  "tags": [
    "Elastic",
    "Cloud",
    "Google Workspace",
    "Continuous Monitoring",
    "SecOps",
    "Identity and Access",
    "Persistence"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0003",
        "name": "Persistence",
        "reference": "https://attack.mitre.org/tactics/TA0003/"
      },
      "technique": [
        {
          "id": "T1098",
          "name": "Account Manipulation",
          "reference": "https://attack.mitre.org/techniques/T1098/",
          "subtechnique": [
            {
              "id": "T1098.003",
              "name": "Additional Cloud Roles",
              "reference": "https://attack.mitre.org/techniques/T1098/003/"
            }
          ]
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "query",
  "version": 101
}

integration_manifests vs latest_major_integration_manifests value:

Update - September 8, 2022

While we are adjusting the build integrations manifest process, we should look to pro-actively refactor it's current process as well to pull package manifest information from the EPR CDN (JSON format) instead of from the GH repository itself. This is possible due to the searching capability of the package registry.

• change manifest pull from GH to EPR CDN with specific URL search parameters

The following was captured from the console output of running the dev integrations build-manifests command:

loading rules to determine all integration tags
integration tags identified: ['google_workspace', 'azure', 'okta', 'gcp', 'cyberarkpas', 'o365', 'kubernetes', 'aws', 'endpoint']
loaded google_workspace manifests from the following package versions: ['1.1.2', '1.6.1', '1.7.0', '1.1.3', '1.2.0', '1.2.2', '1.3.0', '1.3.1', '1.3.2', '1.3.3', '1.4.0', '1.5.0', '1.5.1', '1.6.0']
loaded azure manifests from the following package versions: ['0.0.2', '0.1.2', '0.1.5', '0.1.7', '0.4.0', '0.11.0', '0.12.0', '0.8.2', '0.8.5', '1.0.0', '1.0.1', '1.1.10', '1.1.2', '1.1.6', '1.1.7', '1.1.8', '1.3.0']
loaded okta manifests from the following package versions: ['1.0.0', '1.1.0', '1.10.0', '1.2.0', '1.9.1', '1.9.2', '1.2.3', '1.3.0', '1.3.2', '1.4.0', '1.5.0', '1.5.1', '1.5.2', '1.6.0', '1.7.0', '1.8.0', '1.9.0']
loaded gcp manifests from the following package versions: ['1.1.2', '1.10.0', '1.2.0', '1.2.2', '1.3.0', '1.3.1', '1.4.1', '1.4.2', '1.5.0', '1.5.1', '1.6.1', '1.8.0', '1.9.0', '1.9.1', '1.9.2', '2.2.1', '2.3.0', '2.7.0', '1.1.0']
loaded cyberarkpas manifests from the following package versions: ['1.0.0', '1.1.0', '1.2.3', '2.1.2', '2.5.1', '2.6.0', '2.1.4', '2.2.0', '2.2.2', '2.3.0', '2.3.2', '2.4.0', '2.4.2', '2.5.0']
loaded o365 manifests from the following package versions: ['1.7.1', '1.8.1', '1.0.0', '1.1.4', '1.2.3', '1.2.4', '1.3.0', '1.3.2', '1.4.0', '1.4.1', '1.4.2', '1.4.3', '1.5.1', '1.6.0', '1.7.0']
loaded kubernetes manifests from the following package versions: ['1.1.0', '1.1.2', '1.1.3', '1.17.2', '1.17.3', '1.18.0', '1.18.1', '1.19.0', '1.19.1', '1.20.0', '1.21.0', '1.21.1', '1.21.2', '1.22.0', '1.4.1', '1.5.0', '1.6.0', '1.7.0', '1.8.0', '1.8.1', '1.9.0']
loaded aws manifests from the following package versions: ['0.10.4', '0.10.7', '0.5.0', '0.5.6', '0.6.1', '0.8.0', '1.0.0', '1.1.0', '1.10.2', '1.11.0', '1.12.1', '1.13.0', '1.13.1', '1.14.0', '1.14.5', '1.14.7', '1.14.8', '1.16.0', '1.16.4', '1.16.6', '1.17.0', '1.17.1', '1.17.4', '1.18.2', '1.23.2', '1.23.3', '1.23.4', '1.4.1', '1.5.0', '1.5.1', '1.6.1', '1.9.0']
loaded endpoint manifests from the following package versions: ['0.13.0', '0.13.1', '0.16.0', '0.16.1', '0.16.2', '0.17.0', '0.17.1', '0.18.0', '0.19.1', '0.20.2', '1.0.0', '1.1.0', '1.1.1', '1.2.0', '1.2.1', '1.2.2', '1.3.0', '1.4.0', '1.4.1', '1.5.0', '8.2.0', '8.3.0', '8.4.0', '8.4.1']
final integrations manifests dumped: /Users/tdejesus/code/src/detection-rules/detection_rules/etc/integration-manifests.json.gz

Testing

For testing, I used the view-rule command to see what a finalized rule built with the new related integrations field to ensure the value is the same or expected. As seen below, the ^ is added to allow Kibana to identify this is the least compatible version and the related integration field is populated.

{
  "author": [
    "Elastic"
  ],
  "description": "Assigning the administrative role to a user will grant them access to the Google Admin console and grant them administrator privileges which allow them to access and manage various resources and applications. An adversary may create a new administrator account for persistence or apply the admin role to an existing user to carry out further intrusion efforts. Users with super-admin privileges can bypass single-sign on if enabled in Google Workspace.",
  "false_positives": [
    "Google Workspace admin role assignments may be modified by system administrators. Verify that the configuration change was expected. Exceptions can be added to this rule to filter expected behavior."
  ],
  "from": "now-130m",
  "index": [
    "filebeat-*",
    "logs-google_workspace*"
  ],
  "interval": "10m",
  "language": "kuery",
  "license": "Elastic License v2",
  "name": "Google Workspace Admin Role Assigned to a User",
  "note": "### Important Information Regarding Google Workspace Event Lag Times\n- As per Google's documentation, Google Workspace administrators may observe lag times ranging from minutes up to 3 days between the time of an event's occurrence and the event being visible in the Google Workspace admin/audit logs.\n- This rule is configured to run every 10 minutes with a lookback time of 130 minutes.\n- To reduce the risk of false negatives, consider reducing the interval that the Google Workspace (formerly G Suite) Filebeat module polls Google's reporting API for new events.\n- By default, `var.interval` is set to 2 hours (2h). Consider changing this interval to a lower value, such as 10 minutes (10m).\n- See the following references for further information:\n  - https://support.google.com/a/answer/7061566\n  - https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-google_workspace.html",
  "query": "event.dataset:\"google_workspace.admin\" and event.category:\"iam\" and event.action:\"ASSIGN_ROLE\"\n  and google_workspace.event.type:\"DELEGATED_ADMIN_SETTINGS\" and google_workspace.admin.role.name : *_ADMIN_ROLE\n",
  "references": [
    "https://support.google.com/a/answer/172176?hl=en"
  ],
  "related_integrations": [
    {
      "package": "google_workspace",
      "version": "^1.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.category",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.dataset",
      "type": "keyword"
    },
    {
      "ecs": false,
      "name": "google_workspace.admin.role.name",
      "type": "keyword"
    },
    {
      "ecs": false,
      "name": "google_workspace.event.type",
      "type": "keyword"
    }
  ],
  "risk_score": 73,
  "rule_id": "68994a6c-c7ba-4e82-b476-26a26877adf6",
  "setup": "The Google Workspace Fleet integration, Filebeat module, or similarly structured data is required to be compatible with this rule.",
  "severity": "high",
  "tags": [
    "Elastic",
    "Cloud",
    "Google Workspace",
    "Continuous Monitoring",
    "SecOps",
    "Identity and Access",
    "Persistence"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0003",
        "name": "Persistence",
        "reference": "https://attack.mitre.org/tactics/TA0003/"
      },
      "technique": [
        {
          "id": "T1098",
          "name": "Account Manipulation",
          "reference": "https://attack.mitre.org/techniques/T1098/",
          "subtechnique": [
            {
              "id": "T1098.003",
              "name": "Additional Cloud Roles",
              "reference": "https://attack.mitre.org/techniques/T1098/003/"
            }
          ]
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "query",
  "version": 101
}

Additionally, the build-release command was used to test this on all rules to ensure this does not have any exceptions or errors for all rules:

Output:

[+] Building package 8.5
 - 10 rules excluded from package
Rule changes detected!
 - 185 changed rules
 - 5 new rules
 - 0 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
Package saved to: /Users/tdejesus/code/src/detection-rules/releases/8.5
- sha256: 29ffc33b40793df8bedee9b1a51f7b82c7294638cfe9af36c2e2878e5539837e
- 684 rules included

example integration rule from build package:

{
  "author": [
    "Elastic"
  ],
  "description": "Identifies the deletion of an Amazon Elastic Compute Cloud (EC2) network access control list (ACL) or one of its ingress/egress entries.",
  "false_positives": [
    "Network ACL's may be deleted by a network administrator. Verify whether the user identity, user agent, and/or hostname should be making changes in your environment. Network ACL deletions by unfamiliar users or hosts should be investigated. If known behavior is causing false positives, it can be exempted from the rule."
  ],
  "from": "now-60m",
  "index": [
    "filebeat-*",
    "logs-aws*"
  ],
  "interval": "10m",
  "language": "kuery",
  "license": "Elastic License v2",
  "name": "AWS EC2 Network Access Control List Deletion",
  "note": "",
  "query": "event.dataset:aws.cloudtrail and event.provider:ec2.amazonaws.com and event.action:(DeleteNetworkAcl or DeleteNetworkAclEntry) and event.outcome:success\n",
  "references": [
    "https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/delete-network-acl.html",
    "https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DeleteNetworkAcl.html",
    "https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/delete-network-acl-entry.html",
    "https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DeleteNetworkAclEntry.html"
  ],
  "related_integrations": [
    {
      "integration": "cloudtrail",
      "package": "aws",
      "version": "^1.10.2"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.dataset",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.outcome",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.provider",
      "type": "keyword"
    }
  ],
  "risk_score": 47,
  "rule_id": "8623535c-1e17-44e1-aa97-7a0699c3037d",
  "setup": "The AWS Fleet integration, Filebeat module, or similarly structured data is required to be compatible with this rule.",
  "severity": "medium",
  "tags": [
    "Elastic",
    "Cloud",
    "AWS",
    "Continuous Monitoring",
    "SecOps",
    "Network Security"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0005",
        "name": "Defense Evasion",
        "reference": "https://attack.mitre.org/tactics/TA0005/"
      },
      "technique": [
        {
          "id": "T1562",
          "name": "Impair Defenses",
          "reference": "https://attack.mitre.org/techniques/T1562/",
          "subtechnique": [
            {
              "id": "T1562.001",
              "name": "Disable or Modify Tools",
              "reference": "https://attack.mitre.org/techniques/T1562/001/"
            }
          ]
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "query",
  "version": 101
}

[terrancedejesus]

Outcome from changes below. Note we need to pull in endpoint as well to account for any endpoint security integration based rules. These rules should an integration tag as well, but this will be addressed in another issue/PR.

['okta', 'endpoint', 'gcp', 'kubernetes', 'azure', 'o365', 'google_workspace', 'cyberarkpas', 'aws']

[terrancedejesus]

Unit Testing

I adjusted a unit test as well which used the folders to determine if a tag was valid or necessary. Rather this, we can rely on the EPR URL to grab a list of all valid integration names and if the tag value does not exist in this list, it is invalid.

Additionally, similar to how we do the related_integrations build time field, every rule with an ast object should have it's query parsed and if event.dataset exists, an integration tag should be given.

Example testing screenshot:

[Mikaayenson]

LGTM.! Great work 🔥

[Mikaayenson]

LGTM

[terrancedejesus]

Thanks @w0rk3r !