Add Promtail Nomad Pack

packs/promtail/README.md

@@ -0,0 +1,37 @@
+# Promtail
+
+[Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) is an agent which ships the contents of local logs to a private Loki instance or [Grafana Cloud](https://grafana.com/oss/loki). It is usually deployed to every machine that has applications needed to be monitored.
+
+This pack deploys Promtail as a Nomad [System Job](https://www.nomadproject.io/docs/schedulers#system) using the `grafana/promtail` Docker image and Consul Service named "promtail".
+
+## Dependencies
+
+This pack requires Linux clients to run properly.
+
+
+## Configuration
+
+This pack allows passing a pre-made Promtail configuration file by setting the `config_file` variable to a filepath relative to the directory the `nomad-pack` command is being called from. If the custom config file being used will require the promtail container to rune as a privileged container, you must set the `privileged_container` variable to `true`.
+
+If no custom configuration file is provided, a default template will be used which is configured to scrape systemd-journal logs by default. You will need to set the `client_urls` variable with a list of URL's in order for promtail to ship the logs. **Using the default configuration sets the container to be run as privileged**.
+
+## Variables
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_job_name"></a> [job\_name](#input\_job\_name) | The name to use as the job name which overrides using the pack name. | `string` | `""` | no |
+| <a name="input_service_name"></a> [service\_name](#input\_service\_name) | Name used to register the Consul Service | `string` | `"promtail"` | no |
+| <a name="input_service_check_name"></a> [service\_check\_name](#input\_service\_check\_name) | Name of the service check registered with the Consul Service | `string` | `"Readiness"` | no |
+| <a name="input_datacenters"></a> [datacenters](#input\_datacenters) | A list of datacenters in the region which are eligible for task placement. | `list(string)` | <pre>[<br>  "dc1"<br>]</pre> | no |
+| <a name="input_region"></a> [region](#input\_region) | The region where the job should be placed. | `string` | `"global"` | no |
+| <a name="input_version_tag"></a> [version\_tag](#input\_version\_tag) | The docker image version. For options, see https://hub.docker.com/grafana/promtail | `string` | `"latest"` | no |
+| <a name="input_http_port"></a> [http\_port](#input\_http\_port) | The Nomad client port that routes to the Promtail. | `number` | `9080` | no |
+| <a name="input_resources"></a> [resources](#input\_resources) | The resource to assign to the promtail service task. | <pre>object({<br>    cpu    = number<br>    memory = number<br>  })</pre> | <pre>{<br>  "cpu": 200,<br>  "memory": 256<br>}</pre> | no |
+| <a name="input_config_file"></a> [config\_file](#input\_config\_file) | Path to custom Promtail configuration file. | `string` | `""` | no |
+| <a name="input_mount_journal"></a> [mount\_journal](#input\_mount\_journal) | Controls whether /var/log/journal is mounted in the container. If true, container will be run privileged. | `bool` | `true` | no |
+| <a name="input_mount_machine_id"></a> [mount\_machine\_id](#input\_mount\_machine\_id) | Controls whether /etc/machine-id is mounted in the container. If true, container will be run privileged. | `bool` | `true` | no |
+| <a name="input_privileged_container"></a> [privileged\_container](#input\_privileged\_container) | Run as a privileged container. Setting mount\_journal or mount\_machine\_id to true will override this. | `bool` | `false` | no |
+| <a name="input_client_urls"></a> [client\_urls](#input\_client\_urls) | A list of client url's for promtail to send it's data to. | `list(string)` | `[]` | no |
+| <a name="input_journal_max_age"></a> [journal\_max\_age](#input\_journal\_max\_age) | Maximum age of journald entries to scrape. | `string` | `"12h"` | no |
+| <a name="input_log_level"></a> [log\_level](#input\_log\_level) | Promtail log level configuration. | `string` | `"info"` | no |
+| <a name="input_upstreams"></a> [upstreams](#input\_upstreams) | Define Connect Upstreams used by Promtail. | <pre>list(object({<br>    name = string<br>    port = number<br>  }))</pre> | `[]` | no |

packs/promtail/metadata.hcl

@@ -0,0 +1,11 @@
+app {
+  url    = "https://grafana.com/docs/promtail/latest/clients/promtail"
+  author = "Ricky Grassmuck"
+}
+
+pack {
+  name        = "promtail"
+  description = "Promtail is an agent which ships the contents of local logs to a private Loki instance or Grafana Cloud."
+  url         = "https://github.com/hashicorp/nomad-pack-community-registry/promtail"
+  version     = "0.0.1"
+}

packs/promtail/outputs.tpl

@@ -0,0 +1 @@
+Promtail successfully deployed.

packs/promtail/templates/_helpers.tpl

@@ -0,0 +1,51 @@
+// allow nomad-pack to set the job name
+
+[[- define "job_name" -]]
+[[- if eq .promtail.job_name "" -]]
+[[- .nomad_pack.pack.name | quote -]]
+[[- else -]]
+[[- .promtail.job_name | quote -]]
+[[- end -]]
+[[- end -]]
+
+// only deploys to a region if specified
+
+[[- define "region" -]]
+[[- if not (eq .promtail.region "") -]]
+region = [[ .promtail.region | quote]]
+[[- end -]]
+[[- end -]]
+
+// Use provided config file if defined, else render a default
+[[- define "promtail_config" -]]
+[[- if (eq .promtail.config_file "") -]]
+server:
+  http_listen_port: 9080
+  log_level: [[ .promtail.log_level ]]
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  [[ range $idx, $url := .promtail.client_urls ]][[if $idx]]  [[end]][[ $url | printf "- url: %s\n" ]][[ end ]]
+scrape_configs:
+- job_name: journal
+  journal:
+    max_age: [[ .promtail.journal_max_age ]]
+    json: false
+    labels:
+      job: systemd-journal
+  relabel_configs:
+  - source_labels:
+    - __journal__systemd_unit
+    target_label: systemd_unit
+  - source_labels:
+    - __journal__hostname
+    target_label: nodename
+  - source_labels:
+    - __journal_syslog_identifier
+    target_label: syslog_identifier
+[[- else -]]
+[[ $config := .promtail.config_file ]][[ fileContents $config ]]
+[[- end -]]
+[[- end -]]

packs/promtail/templates/promtail.nomad.tpl

@@ -0,0 +1,102 @@
+job [[ template "job_name" . ]] {
+
+  [[ template "region" . ]]
+
+  datacenters = [ [[ range $idx, $dc := .promtail.datacenters ]][[if $idx]],[[end]][[ $dc | quote ]][[ end ]] ]
+  type = "system"
+
+  constraint {
+    attribute = "${attr.kernel.name}"
+    value     = "linux"
+  }
+
+  group "promtail" {
+    count = 1
+
+    network {
+      mode = "bridge"
+      port "http" {
+        to = [[ .promtail.http_port ]]
+      }
+    }
+
+    service {
+      name = [[ .promtail.service_name | quote ]]
+      port = "http"
+      [[- if not (eq (len .promtail.upstreams) 0) ]] // Render Connect upstreams if defined
+      connect {
+        sidecar_service {
+          proxy {
+            [[- range $upstream := .promtail.upstreams ]]
+            upstreams {
+              destination_name = [[ $upstream.name | quote ]]
+              local_bind_port  = [[ $upstream.port ]]
+            }
+            [[- end ]]
+          }
+        }
+      }
+      [[- end ]]
+      check {
+        name = [[ .promtail.service_check_name | quote ]]
+        port = "http"
+        type = "http"
+        path = "/ready"
+        timeout = "2s"
+        interval = "10s"
+      }
+    }
+
+    task "promtail" {
+      driver = "docker"
+      privileged = true
+
+      template {
+        destination = "local/promtail-config.yaml"
+        data = <<EOT
+[[ template "promtail_config" . ]]
+EOT
+      }
+
+      config {
+        image = "grafana/promtail:[[ .promtail.version_tag ]]"
+        args = [
+          "-config.file=/etc/promtail/promtail-config.yaml",
+          "-log.level=[[ .promtail.log_level ]]"
+        ]
+
+        privileged = [[ if or (.promtail.mount_journal) (.promtail.mount_machine_id) (.promtail.privileged_container) ]]true[[ else ]]false[[ end ]]
+
+        mount {
+          type = "bind"
+          target = "/etc/promtail/promtail-config.yaml"
+          source = "local/promtail-config.yaml"
+          readonly = false
+          bind_options { propagation = "rshared" }
+        }
+        [[ if .promtail.mount_journal ]]
+        mount {
+          type = "bind"
+          target = "/var/log/journal"
+          source = "/var/log/journal"
+          readonly = true
+          bind_options { propagation = "rshared" }
+        }
+        [[ end ]]
+        [[ if .promtail.mount_machine_id ]]
+        mount {
+          type = "bind"
+          target = "/etc/machine-id"
+          source = "/etc/machine-id"
+          readonly = false
+          bind_options { propagation = "rshared" }
+        }
+        [[ end ]]
+      }
+      resources {
+        cpu    = [[ .promtail.resources.cpu ]]
+        memory = [[ .promtail.resources.memory ]]
+      }
+    }
+  }
+}

packs/promtail/variables.hcl

@@ -0,0 +1,108 @@
+variable "job_name" {
+  description = "The name to use as the job name which overrides using the pack name."
+  type        = string
+  // If "", the pack name will be used
+  default     = ""
+}
+
+variable "service_name" {
+  description = "Name used to register the Consul Service"
+  type		    = string
+  default     = "promtail"
+}
+
+variable "service_check_name" {
+  description = "Name of the service check registered with the Consul Service"
+  type		    = string
+  default     = "Readiness"
+}
+
+variable "datacenters" {
+  description = "A list of datacenters in the region which are eligible for task placement."
+  type        = list(string)
+  default     = ["dc1"]
+}
+
+variable "region" {
+  description = "The region where the job should be placed."
+  type        = string
+  default     = "global"
+}
+
+variable "version_tag" {
+  description = "The docker image version. For options, see https://hub.docker.com/grafana/promtail"
+  type        = string
+  default     = "latest"
+}
+
+variable "http_port" {
+  description = "The Nomad client port that routes to the Promtail."
+  type        = number
+  default     = 9080
+}
+
+variable "resources" {
+  description = "The resource to assign to the promtail service task."
+  type = object({
+    cpu    = number
+    memory = number
+  })
+  default = {
+    cpu    = 200,
+    memory = 256
+  }
+}
+
+variable "config_file" {
+  description = "Path to custom Promtail configuration file."
+  type		= string
+  default = ""
+}
+
+variable "mount_journal" {
+  description = "Controls whether /var/log/journal is mounted in the container. If true, container will be run privileged."
+  type		= bool
+  default = true
+}
+
+variable "mount_machine_id" {
+  description = "Controls whether /etc/machine-id is mounted in the container. If true, container will be run privileged."
+  type		= bool
+  default = true
+}
+
+// Default configuration mounts paths /var/log/journal and /etc/machine-id from 
+//   the nomad client. This requires a privileged container.
+variable "privileged_container" {
+  description = "Run as a privileged container. Setting mount_journal or mount_machine_id to true will override this."
+  type		= bool
+  default = false
+}
+
+// Default config options used when no config file is specified
+variable "client_urls" {
+  description = "A list of client url's for promtail to send it's data to."
+  type		= list(string)
+  default = []
+}
+
+variable "journal_max_age" {
+  description = "Maximum age of journald entries to scrape."
+  type		= string
+  default = "12h"
+}
+
+variable "log_level" {
+  description = "Promtail log level configuration."
+  type		= string
+  default = "info"
+}
+
+variable "upstreams" {
+  description = "Define Connect Upstreams used by Promtail."
+  type = list(object({
+    name = string
+    port = number
+  }))
+  default = []
+}