-
Notifications
You must be signed in to change notification settings - Fork 67
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add code to expose datasource configs in API (#216)
In #212 the datasource structure was introduced in the manifests but not exposed through the APIs yet. This PR changes this exposing all the fields. So far it only exposes the fields and no validation is done yet. This needs to be added in the future. A test package was added to see the output of datasource configs and potential changes to it.
- Loading branch information
Showing
14 changed files
with
328 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,7 @@ | |
| { | ||
| "id": "logs", | ||
| "title": "Logs", | ||
| "count": 2 | ||
| "count": 3 | ||
| }, | ||
| { | ||
| "id": "metrics", | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23 changes: 23 additions & 0 deletions
23
testdata/package/datasources-1.0.0/dataset/example/manifest.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| title: Example dataset with inputs | ||
| type: logs | ||
|
|
||
| # List of supported inputs | ||
| inputs: | ||
| - type: log | ||
| vars: | ||
| - name: paths | ||
| required: true | ||
| # Should we define this as array? How will the UI best make sense of it? | ||
| description: Paths to the nginx access log file. | ||
| type: text | ||
| multi: true | ||
| default: | ||
| - /var/log/nginx/access.log* | ||
| # I suggest to use ECS fields for this config options here: https://github.com/elastic/ecs/blob/master/schemas/os.yml | ||
| # This would need to be based on a predefined definition on what can be filtered on | ||
| os.darwin: | ||
| default: | ||
| - /usr/local/var/log/nginx/access.log* | ||
| os.windows: | ||
| default: | ||
| - c:/programdata/nginx/logs/*access.log* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| format_version: 1.0.0 | ||
|
|
||
| name: datasources | ||
| description: Package with data sources | ||
| version: 1.0.0 | ||
| title: Default datasource Integration | ||
| categories: ["logs"] | ||
| type: integration | ||
|
|
||
|
|
||
| datasources: | ||
| - | ||
| # Do we need a name for the data source? | ||
| name: nginx | ||
|
|
||
| # List of inputs this datasource supports | ||
| inputs: | ||
| - | ||
| # An id can be given, in case the type used here is not unique | ||
| # This is for selection in the stream | ||
| # id: nginx | ||
| type: nginx/metrics | ||
| descrition: Collecting metrics for nginx. | ||
|
|
||
| # Common configuration options for this input | ||
| vars: | ||
| - name: hosts | ||
| description: Nginx hosts | ||
| default: | ||
| ["http://127.0.0.1"] | ||
| # All the config options that are required should be shown in the UI | ||
| required: true | ||
| multi: true | ||
| type: text | ||
| - name: period | ||
| description: "Collection period. Valid values: 10s, 5m, 2h" | ||
| default: "10s" | ||
| type: duration | ||
| - name: username | ||
| type: text | ||
| - name: password | ||
| # This is the html input type? | ||
| type: password | ||
|
|
||
| - | ||
| type: logs | ||
| description: Collect nginx logs. | ||
|
|
||
| # Common configuration options for this input | ||
| vars: | ||
|
|
||
| - | ||
| type: syslog | ||
|
|
||
| # Common configuration options for this input | ||
| vars: |
23 changes: 23 additions & 0 deletions
23
testdata/public/package/datasources-1.0.0/dataset/example/manifest.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| title: Example dataset with inputs | ||
| type: logs | ||
|
|
||
| # List of supported inputs | ||
| inputs: | ||
| - type: log | ||
| vars: | ||
| - name: paths | ||
| required: true | ||
| # Should we define this as array? How will the UI best make sense of it? | ||
| description: Paths to the nginx access log file. | ||
| type: text | ||
| multi: true | ||
| default: | ||
| - /var/log/nginx/access.log* | ||
| # I suggest to use ECS fields for this config options here: https://github.com/elastic/ecs/blob/master/schemas/os.yml | ||
| # This would need to be based on a predefined definition on what can be filtered on | ||
| os.darwin: | ||
| default: | ||
| - /usr/local/var/log/nginx/access.log* | ||
| os.windows: | ||
| default: | ||
| - c:/programdata/nginx/logs/*access.log* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| { | ||
| "name": "datasources", | ||
| "title": "Default datasource Integration", | ||
| "version": "1.0.0", | ||
| "description": "Package with data sources", | ||
| "type": "integration", | ||
| "categories": [ | ||
| "logs" | ||
| ], | ||
| "requirement": { | ||
| "kibana": {} | ||
| }, | ||
| "assets": [ | ||
| "/package/datasources-1.0.0/manifest.yml", | ||
| "/package/datasources-1.0.0/dataset/example/manifest.yml" | ||
| ], | ||
| "format_version": "1.0.0", | ||
| "datasets": [ | ||
| { | ||
| "title": "Example dataset with inputs", | ||
| "name": "example", | ||
| "release": "beta", | ||
| "type": "logs", | ||
| "inputs": [ | ||
| { | ||
| "type": "log", | ||
| "vars": [ | ||
| { | ||
| "default": [ | ||
| "/var/log/nginx/access.log*" | ||
| ], | ||
| "description": "Paths to the nginx access log file.", | ||
| "multi": true, | ||
| "name": "paths", | ||
| "os": { | ||
| "darwin": { | ||
| "default": [ | ||
| "/usr/local/var/log/nginx/access.log*" | ||
| ] | ||
| }, | ||
| "windows": { | ||
| "default": [ | ||
| "c:/programdata/nginx/logs/*access.log*" | ||
| ] | ||
| } | ||
| }, | ||
| "required": true, | ||
| "type": "text" | ||
| } | ||
| ] | ||
| } | ||
| ], | ||
| "package": "datasources" | ||
| } | ||
| ], | ||
| "datasources": [ | ||
| { | ||
| "name": "nginx", | ||
| "inputs": [ | ||
| { | ||
| "descrition": "Collecting metrics for nginx.", | ||
| "type": "nginx/metrics", | ||
| "vars": [ | ||
| { | ||
| "default": [ | ||
| "http://127.0.0.1" | ||
| ], | ||
| "description": "Nginx hosts", | ||
| "multi": true, | ||
| "name": "hosts", | ||
| "required": true, | ||
| "type": "text" | ||
| }, | ||
| { | ||
| "default": "10s", | ||
| "description": "Collection period. Valid values: 10s, 5m, 2h", | ||
| "name": "period", | ||
| "type": "duration" | ||
| }, | ||
| { | ||
| "name": "username", | ||
| "type": "text" | ||
| }, | ||
| { | ||
| "name": "password", | ||
| "type": "password" | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "description": "Collect nginx logs.", | ||
| "type": "logs", | ||
| "vars": null | ||
| }, | ||
| { | ||
| "type": "syslog", | ||
| "vars": null | ||
| } | ||
| ] | ||
| } | ||
| ], | ||
| "download": "/epr/datasources/datasources-1.0.0.tar.gz", | ||
| "path": "/package/datasources-1.0.0" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| format_version: 1.0.0 | ||
|
|
||
| name: datasources | ||
| description: Package with data sources | ||
| version: 1.0.0 | ||
| title: Default datasource Integration | ||
| categories: ["logs"] | ||
| type: integration | ||
|
|
||
|
|
||
| datasources: | ||
| - | ||
| # Do we need a name for the data source? | ||
| name: nginx | ||
|
|
||
| # List of inputs this datasource supports | ||
| inputs: | ||
| - | ||
| # An id can be given, in case the type used here is not unique | ||
| # This is for selection in the stream | ||
| # id: nginx | ||
| type: nginx/metrics | ||
| descrition: Collecting metrics for nginx. | ||
|
|
||
| # Common configuration options for this input | ||
| vars: | ||
| - name: hosts | ||
| description: Nginx hosts | ||
| default: | ||
| ["http://127.0.0.1"] | ||
| # All the config options that are required should be shown in the UI | ||
| required: true | ||
| multi: true | ||
| type: text | ||
| - name: period | ||
| description: "Collection period. Valid values: 10s, 5m, 2h" | ||
| default: "10s" | ||
| type: duration | ||
| - name: username | ||
| type: text | ||
| - name: password | ||
| # This is the html input type? | ||
| type: password | ||
|
|
||
| - | ||
| type: logs | ||
| description: Collect nginx logs. | ||
|
|
||
| # Common configuration options for this input | ||
| vars: | ||
|
|
||
| - | ||
| type: syslog | ||
|
|
||
| # Common configuration options for this input | ||
| vars: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters