Elastic connector integration #10898

jedrazb · 2024-08-27T16:50:59Z

Proposed commit message

Initial implementation of the package for the connectors service, running exclusively in agentless mode. The package supports ~ 30 connector types (e.g., Google Drive, SharePoint, MongoDB), each syncing data to an ES index. We have a policy_template per connector type, so that we have a unique tiles displayed the integrations UI. This PR just adds support for Google Drive and Github Connectors.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
- This component doesn't produce any datastreams (the logs are collected via "System" integration) do we need to document it here?
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Verified policy locally with WIP connectors running in agent (custom docker image)

How to test this PR locally

Reach out to @jedrazb (@jedr.blaszyk on Slack) I have doc with all the steps to get this running locally, not linking here since it's public repo

Related issues

Screenshots

.github/CODEOWNERS

packages/elastic_connectors/manifest.yml

seanstory

LGTM from the Search side. Two small comments.

packages/elastic_connectors/docs/github.md

seanstory · 2024-09-05T17:32:01Z

packages/elastic_connectors/manifest.yml

+  kibana:
+    version: "^9.0.0"
+  elastic:
+    subscription: "basic"


CC @danajuratoni - I expect we actually want this platinum+, but since it'll only be accessible through Agentless (cloud and serverless) maybe this doesn't matter?

@seanstory This should be standard (I confirmed with Dana). Looking at integration manifest only 4 options are supported in enum: basic, gold, platinum and enterprise. So I think in such a case: basic == standard?

I expect we actually want this platinum+, but since it'll only be accessible through Agentless (cloud and serverless) maybe this doesn't matter?

This is a great place to get our wires crossed, so I'd like to double check:

licensing for Elastic managed connectors (i.e. Agentless connectors) should remain unchanged. This means Standard on Elastic Cloud hosted.

licensing for connectors on Agent (i.e. self-managed connectors via Agent) is not part of 9.0, but should be equivalent to self-managed connector clients. This would mean likely Platinum+ for self-managed connectors via Agent, final decision to be made in the future / if and when we'll add support.

The approach we took makes it hard if not impossible for users to run this on agent so I wouldn't worry about the self-managed agent case. But I'm following up on this thread.

"standard" == "platinum" IIRC. Standard is our lowest tier in cloud, but it's still a paid tier, to it's higher than basic.

🤯 Good point, Sean. We need to verify that the cloud standard license is included if we enforce platinum in the integration definition.

…l previews, please remove prerelease tag

jedrazb · 2024-09-10T12:10:58Z

Setting the Kibana version restriction to 9+ is causing a Buildkite error. I’m thinking we need to point it to an existing version instead

Error response from daemon: manifest for docker.elastic.co/kibana/kibana:9.0.0 not found: manifest unknown: manifest unknown
--
  | Error: booting up the stack failed: running docker-compose failed: running command failed: running Docker Compose up command failed: exit status 18: manifest for docker.elastic.co/kibana/kibana:9.0.0 not found: manifest

packages/elastic_connectors/manifest.yml

kfirpeled · 2024-09-11T15:52:34Z

packages/elastic_connectors/manifest.yml

+        enabled: false
+      agentless:
+        enabled: true
+    multiple: false


pay attention a user will be able to enable more than one input using the high-level tile.

Correct me if I'm wrong @jsoriano but there's no way to validate there's only one input in use, correct?

Is there actually a way for us to hide/disable this high-level tile? Since we really only want one input at a time, only the lower-level tiles really make sense for us.

not that I know of
with CSPM integration the top-level tier doesn't serve our use case either. When we thought of enabling such capability we noticed that in the fleet, the top-level tier is used in multiple flows, so it seemed like a big change at the time to address. We ended up having a custom UI component instead.

We ended up having a custom UI component instead.

Can you elaborate?

kfirpeled · 2024-09-11T15:53:27Z

packages/elastic_connectors/manifest.yml

+        template_path: github.yml.hbs
+  - name: google_drive
+    title: Google Drive Connector
+    icons:


nit: going forward it is recommended to have screenshots per policy_template

eyalkraft · 2024-09-12T14:58:47Z

packages/elastic_connectors/manifest.yml

+    deployment_modes:
+      default:
+        enabled: false
+      agentless:


FYI we'll soon require some owner tags as part of this section.
It's not yet part of the spec but you can follow here to add it when it'll be added to the spec:

[Change Proposal] When the deployment_mode agentless is enabled we want the package to have additional fields populated. package-spec#795

I assume this won't block us from merging, will just be something we need to add to a todo list?

seanstory · 2024-09-13T14:21:50Z

buildkite test this

elasticmachine · 2024-09-13T14:42:42Z

💚 Build Succeeded

Buildkite Build
Commit: 0598b01

History

💔 Build #15777 failed 0598b01
💔 Build #15704 failed 79843ac
💔 Build #15695 failed 4f587d5
💔 Build #15529 failed 3168a8e
💚 Build #15460 succeeded a4510a7
💔 Build #15084 failed 8629309

elastic-sonarqube · 2024-09-13T14:42:45Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

seanstory · 2024-09-13T22:18:57Z

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

Can I ignore this? Looks like the PR would let me merge as is, but I don't want to skip something I shouldn't

seanstory · 2024-09-16T17:52:55Z

Forgiveness over permission 🤞

elasticmachine · 2024-09-16T18:10:21Z

Package elastic_connectors - 0.0.1 containing this change is available at https://epr.elastic.co/search?package=elastic_connectors

[WIP] Elastic connector integration

8629309

andrewkroh added the New Integration label Aug 27, 2024

jedrazb mentioned this pull request Aug 30, 2024

POC package definition elastic/python-elastic-agent-client#6

Closed

jedrazb added 2 commits September 4, 2024 14:05

Add logo, screenshot, test policy locally

67660cb

Add ingestion team to codeowners

b1e2749

jedrazb commented Sep 4, 2024

View reviewed changes

.github/CODEOWNERS Show resolved Hide resolved

jedrazb added 2 commits September 4, 2024 14:26

Update manifest

726c9de

Fix the version to 0.0.1 as this is initial pr

a4510a7

jedrazb changed the title ~~[WIP] Elastic connector integration~~ Elastic connector integration Sep 4, 2024

jedrazb marked this pull request as ready for review September 4, 2024 12:33

jedrazb commented Sep 4, 2024

View reviewed changes

packages/elastic_connectors/manifest.yml Show resolved Hide resolved

Update version + ES stack constraint

3168a8e

seanstory reviewed Sep 5, 2024

View reviewed changes

jlind23 requested a review from a team September 10, 2024 05:51

jedrazb added 2 commits September 10, 2024 10:22

Remove preview suffix as versions below 1.0.0 are considered technica…

4f587d5

…l previews, please remove prerelease tag

Update input component names for policy templates

79843ac

jedrazb commented Sep 10, 2024

View reviewed changes

packages/elastic_connectors/manifest.yml Show resolved Hide resolved

Add per-service configuration

0598b01

kfirpeled reviewed Sep 11, 2024

View reviewed changes

kfirpeled approved these changes Sep 11, 2024

View reviewed changes

eyalkraft reviewed Sep 12, 2024

View reviewed changes

seanstory merged commit 0cb33d6 into elastic:main Sep 16, 2024
4 of 5 checks passed

eyalkraft mentioned this pull request Sep 17, 2024

Add support for integration deployment mode agentless to include identity fields elastic/package-spec#801

Merged

2 tasks

andrewkroh added the Integration:elastic_connectors Elastic Connectors label Sep 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastic connector integration #10898

Elastic connector integration #10898

jedrazb commented Aug 27, 2024 •

edited

Loading

seanstory left a comment

seanstory Sep 5, 2024

jedrazb Sep 10, 2024

danajuratoni Sep 10, 2024

jedrazb Sep 10, 2024

seanstory Sep 10, 2024

jedrazb Sep 11, 2024

jedrazb commented Sep 10, 2024

kfirpeled Sep 11, 2024

seanstory Sep 11, 2024

kfirpeled Sep 12, 2024

seanstory Sep 13, 2024

kfirpeled Sep 11, 2024

eyalkraft Sep 12, 2024 •

edited

Loading

seanstory Sep 13, 2024

seanstory commented Sep 13, 2024

elasticmachine commented Sep 13, 2024

elastic-sonarqube bot commented Sep 13, 2024

seanstory commented Sep 13, 2024

seanstory commented Sep 16, 2024

elasticmachine commented Sep 16, 2024

Elastic connector integration #10898

Elastic connector integration #10898

Conversation

jedrazb commented Aug 27, 2024 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

seanstory left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jedrazb commented Sep 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eyalkraft Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seanstory commented Sep 13, 2024

elasticmachine commented Sep 13, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Sep 13, 2024

Quality Gate failed

seanstory commented Sep 13, 2024

seanstory commented Sep 16, 2024

elasticmachine commented Sep 16, 2024

jedrazb commented Aug 27, 2024 •

edited

Loading

eyalkraft Sep 12, 2024 •

edited

Loading