Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[apache_spark][driver] Add Apache Spark package with Driver data stream #2945

Merged
merged 10 commits into from
Apr 22, 2022
Merged

[apache_spark][driver] Add Apache Spark package with Driver data stream #2945

merged 10 commits into from
Apr 22, 2022

Conversation

yug-rajani
Copy link
Contributor

@yug-rajani yug-rajani commented Mar 30, 2022 •
edited
Loading

What does this PR do?

  • Generated the skeleton of Apache Spark integration package.
  • Added 1 data stream ( Driver )
  • Added data collection logic.
  • Added the ingest pipelines.
  • Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

How to test this PR locally

  • Clone integrations repo.
  • Install elastic-package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/apache_spark directory.
  • Run the following command to run tests.

elastic-package test

@yug-rajani yug-rajani requested a review from a team as a code owner March 30, 2022 18:32
@yug-rajani
Copy link
Contributor Author

This PR is a split of #2811 as discussed over the comment #2811 (comment)

@elasticmachine
Copy link

elasticmachine commented Mar 30, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-04-22T08:08:43.652+0000

  • Duration: 15 min 6 sec

Test stats 🧪

Test Results
Failed 0
Passed 9
Skipped 0
Total 9

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@yug-rajani yug-rajani self-assigned this Mar 30, 2022
@yug-rajani yug-rajani mentioned this pull request Mar 30, 2022
Closed
4 tasks
@yug-rajani yug-rajani requested a review from mtojek March 30, 2022 19:19
@yug-rajani yug-rajani added enhancement New feature or request Team:Integrations Label for the Integrations team New Integration Issue or pull request for creating a new integration package. labels Mar 30, 2022
@elasticmachine
Copy link

Pinging @elastic/integrations (Team:Integrations)

mtojek
mtojek reviewed Mar 31, 2022
View reviewed changes
packages/apache_spark/data_stream/driver/fields/fields.yml Outdated Show resolved Hide resolved
packages/apache_spark/data_stream/driver/fields/fields.yml Outdated Show resolved Hide resolved
packages/apache_spark/data_stream/driver/fields/fields.yml Show resolved Hide resolved
packages/apache_spark/docs/README.md Outdated
@@ -0,0 +1,231 @@
# Apache Spark
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/module/integration/g :)

@yug-rajani yug-rajani requested a review from a team as a code owner April 5, 2022 09:30
@yug-rajani yug-rajani requested a review from mtojek April 5, 2022 10:33
@yug-rajani yug-rajani requested a review from mtojek April 6, 2022 11:50
@yug-rajani yug-rajani requested a review from mtojek April 6, 2022 19:38
@yug-rajani yug-rajani mentioned this pull request Apr 6, 2022
Merged
4 tasks
mtojek
mtojek approved these changes Apr 7, 2022
View reviewed changes
Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yug-rajani yug-rajani merged commit a2e1a3a into elastic:main Apr 22, 2022
v1v added a commit to v1v/integrations that referenced this pull request Apr 25, 2022
@v1v
'feature/labels-cloud-ephemeral-components' of github.com:v1v/integr…
02b4c7f 
…ations into feature/labels-cloud-ephemeral-components

* 'feature/labels-cloud-ephemeral-components' of github.com:v1v/integrations: (23 commits)
  use branch
  standardise labels/tags
  Update .ci/Jenkinsfile
  [ci][terraform][aws] tags with metadata
  Bump github.com/elastic/elastic-package from 0.46.0 to 0.47.0 (elastic#3182)
  Spring boot package [Memory - data stream] (elastic#2979)
  [cisco_ise] Add Cisco ISE package (elastic#2855)
  [apache_spark][executor] Add Apache Spark package with Executor data stream (elastic#2943)
  [apache_spark][driver] Add Apache Spark package with Driver data stream (elastic#2945)
  cisco_duo: simplify grok expression for handling ports (elastic#3170)
  cisco_duo: fix handling of IP addresses with port numbers (elastic#3117)
  [sophos] Various improvements and log samples from 18.5 (elastic#3127)
  [cisco_asa] fix visualizations (elastic#3146)
  [apache_spark][application] Add Apache Spark package with Application data stream (elastic#2941)
  Fix pagination bug that skipped events when more than one page is present. (elastic#3140)
  Cis k8s name migration (elastic#3113)
  remove exported fields; these will be added to Kibana docs instead (elastic#3093)
  Exclude ecosystem as owner of subdirectories of packages (elastic#3132)
  [awsfargate] bump package version (elastic#3130)
  add first csp rule template (elastic#3081)
  ...
@yug-rajani yug-rajani linked an issue Apr 26, 2022 that may be closed by this pull request
Create Apache Spark integration #493
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lalit-satapathy lalit-satapathy lalit-satapathy approved these changes

@mtojek mtojek mtojek approved these changes

Assignees

@yug-rajani yug-rajani

Labels
enhancement New feature or request New Integration Issue or pull request for creating a new integration package. Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create Apache Spark integration
4 participants
@yug-rajani @elasticmachine @mtojek @lalit-satapathy