Support input groups #137
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| README template for integration |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| DynamoDB README template file |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| EC2 README template file |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| - version: 1.0.0 | ||
| changes: | ||
| - description: initial release | ||
| type: enhancement | ||
| link: https://github.com/elastic/package-spec/pull/0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| README for integration |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| DynamoDB README file |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| EC2 README file |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| format_version: 1.0.0 | ||
| name: aws | ||
| title: AWS | ||
| version: 0.3.17 | ||
| license: basic | ||
| description: AWS Integration | ||
| type: integration | ||
| categories: | ||
| - aws | ||
| - cloud | ||
| - network | ||
| - security | ||
| release: beta | ||
| conditions: | ||
| kibana.version: '^7.9.0' | ||
| icons: | ||
| - src: /img/aws-logo.svg | ||
| title: logo aws | ||
| size: 32x32 | ||
| type: image/svg+xml | ||
| screenshots: | ||
| - src: /img/aws-general-dashboard.png | ||
| title: General AWS dashboard | ||
| size: 1702x1063 | ||
| type: image/png | ||
| vars: | ||
| - name: access_key_id | ||
| type: text | ||
| title: Access Key ID | ||
| - name: secret_access_key | ||
| type: text | ||
| title: Secret Access Key | ||
| policy_templates: | ||
|
There was a problem hiding this comment. @mtojek is this the array from which Kibana only shows the first element today? Or is that something different? There was a problem hiding this comment. Or maybe @jen-huang can weigh in on my question since she's already watching this PR. 🙂 There was a problem hiding this comment. yes, this is what Kibana uses the first element of today |
||
| - name: ec2 | ||
| title: AWS EC2 | ||
| description: Collect logs and metrics from EC2 service | ||
| inputs: | ||
| - type: s3 | ||
| title: Collect logs from EC2 service | ||
| description: Collecting EC2 logs using S3 input | ||
| policy_group: logs | ||
| vars: | ||
| - name: visibility_timeout | ||
| type: text | ||
| title: Visibility Timeout | ||
| description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours. | ||
| - name: api_timeout | ||
| type: text | ||
| title: API Timeout | ||
| description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value. | ||
| - type: aws/metrics | ||
| title: Collect metrics from EC2 service | ||
| description: Collecting EC2 metrics using AWS CloudWatch | ||
| policy_group: metrics | ||
| vars: | ||
| - name: endpoint | ||
| type: text | ||
| title: Endpoint | ||
| default: "amazonaws.com" | ||
| description: URL of the entry point for an AWS web service. | ||
| icons: | ||
| - src: /img/aws-ec2.svg | ||
| title: AWS EC2 logo | ||
| size: 32x32 | ||
| type: image/svg+xml | ||
| screenshots: | ||
| - src: /img/aws-ec2-overview.png | ||
| title: AWS EC2 Overview | ||
| size: 1702x1063 | ||
| type: image/png | ||
| vars: | ||
| - name: ec2_foobar_property | ||
| type: bool | ||
| title: EC2 Specific Property | ||
| - name: dynamodb | ||
| title: AWS DynamoDB | ||
| description: Collect logs and metrics from DynamoDB service | ||
| categories: | ||
| - datastore | ||
| inputs: | ||
| - type: s3 | ||
kaiyan-sheng marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| title: Collect logs from the DynamoDB service | ||
mtojek marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| description: Collecting DynamoDB logs using S3 input | ||
| policy_group: logs | ||
| - type: aws/metrics | ||
| title: Collect metrics from DynamoDB service | ||
| description: Collecting DynamoDB metrics using AWS CloudWatch | ||
| policy_group: metrics | ||
| icons: | ||
| - src: /img/aws-dynamodb.svg | ||
| title: AWS DynamoDB logo | ||
| size: 32x32 | ||
| type: image/svg+xml | ||
| screenshots: | ||
| - src: /img/aws-dynamodb-overview.png | ||
| title: AWS DynamoDB Overview | ||
| size: 1702x1063 | ||
| type: image/png | ||
| vars: | ||
| - name: dynamodb_foobar_property | ||
| type: bool | ||
| title: DynamoDB Specific Property | ||
| policy_groups: | ||
mtojek marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Ignoring naming on my end for now ;-) I'm wondering if the groups should be defined as part of each policy_template or global. Having it local would allow to give more descriptive descriptions. At the same time, I don't think we use it today. Global requires much fewer configuration. An alternative option is to just not have it in the package at first. Only 2-3 groups are supported, required by the spec and hardcoded into Kibana. Like this everyone is forced to use There was a problem hiding this comment. Actually your both options are related. I agree that global definitions would require less configuration properties and might be easier to setup. The second option is an interesting solution - we can try to kick off the feature with predefined groups and expose only them to users. We need to remember that we won't escape from these tabs in the future :) Let's collect some other comments on this idea. There was a problem hiding this comment. I'd prefer to start with the most restrictive option we can get away with and then make things less restrictive / more flexible depending on feedback from actual usage. So I'm ++ for hardcoding the group names ( How far do we take this hardcoding? I assume not just the group There was a problem hiding this comment. @sorantis Any thoughts about hardcoding group names (logs, metrics, later on - checks)? |
||
| - name: logs | ||
| title: "Logs" | ||
| description: "Collect logs from AWS service" | ||
| - name: metrics | ||
| title: "Metrics" | ||
| description: "Collect metrics from AWS service" | ||
| owner: | ||
| github: elastic/integrations-platforms | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sorantis Could have a group/set its own categories? Is it always a subset of the package categories?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In some cases it's a bigger set, i.e. DynamoDB is an
awsservice, acloudservice and adatastoreservice.datastorehere is specific to DynamoDB and will not be applicable to other, for example compute related services.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The question is what happens if someone clicks on a category, what will the filter return:
Based on the above we should know if we need categories inside the integration sets / policy templates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah and extending to what @ruflin said, it's more natural to keep DynamoDB, EC2 and others as separate packages ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the user clicks on say "Datastore" then they should be able to see all integrations that cover databases, data stores, etc. which also includes cloud data store services. Since the tiles are (will be) generated from policy_templates then I think it's the option 3 in our list.
This creates another question though. What happens if the user searches for AWS only, do we show AWS as a tile? The answer yes if we create a policy template (integration set) that is called AWS.
This should work automatically for other integrations, for example NGINX. When the user searches for Nginx we show the Nginx tile because there's a policy template for it, we don't have to use package level information for it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Speaking of how to express categories in the manifest file, I suggest we keep the top level categories section required, and introduce an optional categories section to a policy template.
This way each policy template will inherit top level categories and also has the option to add more policy template specific categories if needed.
Thoughts? @mtojek @ycombinator @ruflin
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In other words, categories on the root level (in manifest) would be common buckets valid for all policy templates, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, with the option to extend this list for each policy template. I.e. for AWS common buckets could be
cloud,awsand DynamoDB could optionally extend this list withdatastoresby defining it at the policy template level.