Add third-party REST API

[leehinman]

What does this PR do?

Adds third-party REST API to several packages. This allows pulling data from Splunk REST API.

• Apache
• nginx
• AWS Cloudtrail
• Zeek

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all datasets collect metrics or logs.

Screenshots

Add Apache Integration Screen

Third-Party REST API Configuration

Example ingested document

{
  "_index": "filebeat-8.0.0-2021.01.11-000001",
  "_type": "_doc",
  "_id": "d9a88109c64ba572d8319e3325901415ff78f6b6fae8346d38c28c98010ee0b0",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "name": "Lees-MacBook-Pro.local",
      "id": "7b45f55c-a54b-4f81-a8e5-8b4033d14305",
      "type": "filebeat",
      "ephemeral_id": "3170de0e-eb99-4784-b17e-eb4292443378",
      "version": "8.0.0"
    },
    "source": {
      "address": "127.0.0.1",
      "ip": "127.0.0.1"
    },
    "fileset": {
      "name": "access"
    },
    "url": {
      "original": "/"
    },
    "tags": [
      "forwarded"
    ],
    "input": {
      "type": "httpjson"
    },
    "file": {
      "path": "/var/log/apache2/access.log"
    },
    "apache": {
      "access": {}
    },
    "@timestamp": "2021-01-12T17:45:52.000Z",
    "ecs": {
      "version": "1.7.0"
    },
    "service": {
      "type": "apache"
    },
    "host": {
      "name": "ubuntu-bionic"
    },
    "http": {
      "request": {
        "referrer": "-",
        "method": "GET"
      },
      "response": {
        "status_code": 200,
        "body": {
          "bytes": 11173
        }
      },
      "version": "1.1"
    },
    "event": {
      "ingested": "2021-01-12T17:46:01.271439318Z",
      "kind": "event",
      "created": "2021-01-12T17:46:00.241Z",
      "module": "apache",
      "category": "web",
      "dataset": "apache.access",
      "outcome": "success"
    },
    "user": {
      "name": "-"
    },
    "user_agent": {
      "original": "curl/7.58.0",
      "name": "curl",
      "device": {
        "name": "Other"
      },
      "version": "7.58.0"
    }
  },
  "fields": {
    "event.ingested": [
      "2021-01-12T17:46:01.271Z"
    ],
    "@timestamp": [
      "2021-01-12T17:45:52.000Z"
    ],
    "event.created": [
      "2021-01-12T17:46:00.241Z"
    ]
  },
  "sort": [
    1610473552000
  ]
}

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #545 updated

• Start Time: 2021-03-10T21:13:49.730+0000

• Duration: 32 min 23 sec

• Commit: ce66995

Test stats 🧪

Test	Results
Failed	0
Passed	445
Skipped	0
Total	445

Trends 🧪

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

I would really like to see some tests that exercise the httpjson part of the config. This is the idea I had when I was thinking about how to test some other packages that use httpjson (like google workspace and okta): https://github.com/andrewkroh/stream/issues/9

[andrewkroh]

This looks misspelled.

[andrewkroh]

There are quite a few possible SSL options available. I suggest hiding the SSL options under an advanced setting and letting users configure it as YAML. Passing PEM encoded CA certs too is a little easier IMO with the textbox it gives you.

integrations/packages/zoom/data_stream/webhook/manifest.yml

Lines 40 to 50 in 8770467

	- name: ssl
	type: yaml
	title: TLS
	description: Options for enabling TLS for the listening webhook endpoint. Zoom requires webhooks listen on HTTPS. You must either provide a valid TLS certificate or use a reverse proxy in front of the integration. See the [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html) for a list of all options.
	multi: false
	required: false
	show_user: false
	default: |
	enabled: false
	certificate: "/etc/pki/client/cert.pem"
	key: "/etc/pki/client/cert.key"

[leehinman]

Done, see if the screenshot is what you were thinking

[andrewkroh]

We should probably mark this with "(experimental)". Once we have a few more necessary package/fleet features we'll be able to implement this an independent package and remove these.