forked from elastic/beats
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Initial support for Nomad (elastic#14954)
Initial features to support logs collection from applications deployed in Nomad. Add a new `nomad` autodiscover provider (based on the Kubernetes provider). With this new provider, it is possible to start new harvesters by looking at the jobs allocated on each node. With this, filebeat can be run as a system job on each node and each filebeat instance is responsible for enriching and shipping the local logs. This autodiscover provider supports hints-based autodiscover. Add a new `add_nomad_metadata` processor that matches events to specific allocations and adds the metadata. Co-authored-by: Jaime Soriano Pastor <[email protected]>
- Loading branch information
Showing
30 changed files
with
4,676 additions
and
740 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| Filebeat supports templates for inputs and modules. | ||
|
|
||
| ["source","yaml",subs="attributes"] | ||
| ------------------------------------------------------------------------------------- | ||
| filebeat.autodiscover: | ||
| providers: | ||
| - type: nomad | ||
| node: nomad1 | ||
| scope: local | ||
| hints.enabled: true | ||
| allow_stale: true | ||
| templates: | ||
| - condition: | ||
| equals: | ||
| nomad.namespace: web | ||
| config: | ||
| - type: log | ||
| paths: | ||
| - /var/lib/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.stderr.[0-9]* | ||
| exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines | ||
| ------------------------------------------------------------------------------------- | ||
| This configuration launches a `log` input for all jobs under the `web` Nomad namespace. | ||
| If you are using modules, you can override the default input and customize it to read from the | ||
| `${data.nomad.task.name}.stdout` and/or `${data.nomad.task.name}.stderr` files. | ||
| ["source","yaml",subs="attributes"] | ||
| ------------------------------------------------------------------------------------- | ||
| filebeat.autodiscover: | ||
| providers: | ||
| - type: nomad | ||
| templates: | ||
| - condition: | ||
| equals: | ||
| nomad.task.service.tags: "redis" | ||
| config: | ||
| - module: redis | ||
| log: | ||
| input: | ||
| type: log | ||
| paths: | ||
| - /var/lib/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.* | ||
| ------------------------------------------------------------------------------------- | ||
| WARNING: The `docker` input is currently not supported. Nomad doesn't expose the container id | ||
| associated with the allocation. Without the container id, there is no way of generating the proper | ||
| path for reading the container's logs. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.