Skip to content

Commit

Permalink
Add no-file and keep-file log option for docker driver. (#1906)
Browse files Browse the repository at this point in the history 
* Add no-file and keep-file option.

By default no-file is false, this means the driver will create json log files that can be used to run `docker logs` command.
If no-file is true, no files are created and so `docker log` won't work. This allow the ability to run the driver without caring of space left on the host.

keep-file tells the driver to keep file even after the container has been stopped, by default this options is false. This means files are cleaned up once containers are stopped and not history is left to be used for `docker logs`.

Signed-off-by: Cyril Tovena <[email protected]>

* Review feedback !

Signed-off-by: Cyril Tovena <[email protected]>
  • Loading branch information
@cyriltovena
cyriltovena authored Apr 13, 2020
1 parent 2c8251f commit 25aed2b
Show file tree
Hide file tree
Showing 3 changed files with 91 additions and 40 deletions.
49 changes: 26 additions & 23 deletions cmd/docker-driver/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ docker run --log-driver=loki \

> **Note**: The Loki logging driver still uses the json-log driver in combination with sending logs to Loki, this is mainly useful to keep the `docker logs` command working.
> You can adjust file size and rotation using the respective log option `max-size` and `max-file`.
> You can deactivate this behavior by setting the log option `no-file` to true.
### Configure the default logging driver

Expand Down Expand Up @@ -115,29 +116,31 @@ You can add more labels by using `loki-external-labels`,`loki-pipeline-stage-fil

To specify additional logging driver options, you can use the --log-opt NAME=VALUE flag.

| Option | Required? | Default Value | Description
| ------------------------- | :-------: | :------------------: | -------------------------------------- |
| `loki-url` | Yes | | Loki HTTP push endpoint.
| `loki-external-labels` | No | `container_name={{.Name}}` | Additional label value pair separated by `,` to send with logs. The value is expanded with the [Docker tag template format](https://docs.docker.com/config/containers/logging/log_tags/). (eg: `container_name={{.ID}}.{{.Name}},cluster=prod`)
| `loki-timeout` | No | `10s` | The timeout to use when sending logs to the Loki instance. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
| `loki-batch-wait` | No | `1s` | The amount of time to wait before sending a log batch complete or not. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
| `loki-batch-size` | No | `102400` | The maximum size of a log batch to send.
| `loki-min-backoff` | No | `100ms` | The minimum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
| `loki-max-backoff` | No | `10s` | The maximum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
| `loki-retries` | No | `10` | The maximum amount of retries for a log batch.
| `loki-pipeline-stage-file` | No | | The location of a pipeline stage configuration file ([example](./pipeline-example.yaml)). Pipeline stages allows to parse log lines to extract more labels. [see documentation](../../docs/logentry/processing-log-lines.md)
| `loki-tenant-id` | No | | Set the tenant id (http header`X-Scope-OrgID`) when sending logs to Loki. It can be overrides by a pipeline stage.
| `loki-tls-ca-file` | No | | Set the path to a custom certificate authority.
| `loki-tls-cert-file` | No | | Set the path to a client certificate file.
| `loki-tls-key-file` | No | | Set the path to a client key.
| `loki-tls-server-name` | No | | Name used to validate the server certificate.
| `loki-tls-insecure-skip-verify` | No | `false` | Allow to skip tls verification.
| `loki-proxy-url` | No | | Proxy URL use to connect to Loki.
| `max-size` | No | -1 | The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to -1 (unlimited). This is used by json-log required to keep the `docker log` command working.
| `max-file` | No | 1 | The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1.
| `labels` | No | | Comma-separated list of keys of labels, which should be included in message, if these labels are specified for container.
| `env` | No | | Comma-separated list of keys of environment variables to be included in message if they specified for a container.
| `env-regex` | No | | A regular expression to match logging-related environment variables. Used for advanced log label options. If there is collision between the label and env keys, the value of the env takes precedence. Both options add additional fields to the labels of a logging message.
| Option | Required? | Default Value | Description |
|---------------------------------|:---------:|:--------------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `loki-url` | Yes | | Loki HTTP push endpoint. |
| `loki-external-labels` | No | `container_name={{.Name}}` | Additional label value pair separated by `,` to send with logs. The value is expanded with the [Docker tag template format](https://docs.docker.com/config/containers/logging/log_tags/). (eg: `container_name={{.ID}}.{{.Name}},cluster=prod`) |
| `loki-timeout` | No | `10s` | The timeout to use when sending logs to the Loki instance. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
| `loki-batch-wait` | No | `1s` | The amount of time to wait before sending a log batch complete or not. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
| `loki-batch-size` | No | `102400` | The maximum size of a log batch to send. |
| `loki-min-backoff` | No | `100ms` | The minimum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
| `loki-max-backoff` | No | `10s` | The maximum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
| `loki-retries` | No | `10` | The maximum amount of retries for a log batch. |
| `loki-pipeline-stage-file` | No | | The location of a pipeline stage configuration file ([example](./pipeline-example.yaml)). Pipeline stages allows to parse log lines to extract more labels. [see documentation](../../docs/logentry/processing-log-lines.md) |
| `loki-tenant-id` | No | | Set the tenant id (http header`X-Scope-OrgID`) when sending logs to Loki. It can be overrides by a pipeline stage. |
| `loki-tls-ca-file` | No | | Set the path to a custom certificate authority. |
| `loki-tls-cert-file` | No | | Set the path to a client certificate file. |
| `loki-tls-key-file` | No | | Set the path to a client key. |
| `loki-tls-server-name` | No | | Name used to validate the server certificate. |
| `loki-tls-insecure-skip-verify` | No | `false` | Allow to skip tls verification. |
| `loki-proxy-url` | No | | Proxy URL use to connect to Loki. |
| `no-file` | No | `false` | This indicates the driver to not create log files on disk, however this means you won't be able to use `docker logs` on the container anymore. You can use this if you don't need to use `docker logs` and you run with limited disk space. (By default files are created) |
| `keep-file` | No | `false` | This indicates the driver to keep json log files once the container is stopped. By default files are removed, this means you won't be able to use `docker logs` once the container is stopped. |
| `max-size` | No | -1 | The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to -1 (unlimited). This is used by json-log required to keep the `docker log` command working. |
| `max-file` | No | 1 | The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1. |
| `labels` | No | | Comma-separated list of keys of labels, which should be included in message, if these labels are specified for container. |
| `env` | No | | Comma-separated list of keys of environment variables to be included in message if they specified for a container. |
| `env-regex` | No | | A regular expression to match logging-related environment variables. Used for advanced log label options. If there is collision between the label and env keys, the value of the env takes precedence. Both options add additional fields to the labels of a logging message. |

## Uninstall the plugin

Expand Down
16 changes: 16 additions & 0 deletions cmd/docker-driver/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ const (
cfgMaxRetriesKey = "loki-retries"
cfgPipelineStagesKey = "loki-pipeline-stage-file"
cfgTenantIDKey = "loki-tenant-id"
cfgNofile = "no-file"
cfgKeepFile = "keep-file"

swarmServiceLabelKey = "com.docker.swarm.service.name"
swarmStackLabelKey = "com.docker.stack.namespace"
Expand Down Expand Up @@ -101,6 +103,8 @@ func validateDriverOpt(loggerInfo logger.Info) error {
case cfgMaxRetriesKey:
case cfgPipelineStagesKey:
case cfgTenantIDKey:
case cfgNofile:
case cfgKeepFile:
case "labels":
case "env":
case "env-regex":
Expand Down Expand Up @@ -321,3 +325,15 @@ func parseInt(key string, logCtx logger.Info, set func(i int)) error {
}
return nil
}

func parseBoolean(key string, logCtx logger.Info, defaultValue bool) (bool, error) {
value, ok := logCtx.Config[key]
if !ok || value == "" {
return defaultValue, nil
}
b, err := strconv.ParseBool(value)
if err != nil {
return false, err
}
return b, nil
}
66 changes: 49 additions & 17 deletions cmd/docker-driver/driver.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ type logPair struct {
stream io.ReadCloser
info logger.Info
logger log.Logger
// folder where json log files will be created.
folder string
// keep created files after stopping the container.
keepFile bool
}

func (l *logPair) Close() {
Expand All @@ -45,6 +49,9 @@ func (l *logPair) Close() {
if err := l.lokil.Close(); err != nil {
level.Error(l.logger).Log("msg", "error while closing loki logger", "err", err)
}
if l.jsonl == nil {
return
}
if err := l.jsonl.Close(); err != nil {
level.Error(l.logger).Log("msg", "error while closing json logger", "err", err)
}
Expand All @@ -65,16 +72,29 @@ func (d *driver) StartLogging(file string, logCtx logger.Info) error {
return fmt.Errorf("logger for %q already exists", file)
}
d.mu.Unlock()
folder := fmt.Sprintf("/var/log/docker/%s/", logCtx.ContainerID)
logCtx.LogPath = filepath.Join(folder, "json.log")

if logCtx.LogPath == "" {
logCtx.LogPath = filepath.Join("/var/log/docker", logCtx.ContainerID)
}
if err := os.MkdirAll(filepath.Dir(logCtx.LogPath), 0755); err != nil {
return errors.Wrap(err, "error setting up logger dir")
noFile, err := parseBoolean(cfgNofile, logCtx, false)
if err != nil {
return err
}
jsonl, err := jsonfilelog.New(logCtx)

keepFile, err := parseBoolean(cfgKeepFile, logCtx, false)
if err != nil {
return errors.Wrap(err, "error creating jsonfile logger")
return err
}

var jsonl logger.Logger
if !noFile {
if err := os.MkdirAll(folder, 0755); err != nil {
return errors.Wrap(err, "error setting up logger dir")
}

jsonl, err = jsonfilelog.New(logCtx)
if err != nil {
return errors.Wrap(err, "error creating jsonfile logger")
}
}

lokil, err := New(logCtx, d.logger)
Expand All @@ -88,7 +108,7 @@ func (d *driver) StartLogging(file string, logCtx logger.Info) error {
}

d.mu.Lock()
lf := &logPair{jsonl, lokil, f, logCtx, d.logger}
lf := &logPair{jsonl, lokil, f, logCtx, d.logger, folder, keepFile}
d.logs[file] = lf
d.idx[logCtx.ContainerID] = lf
d.mu.Unlock()
Expand All @@ -100,12 +120,19 @@ func (d *driver) StartLogging(file string, logCtx logger.Info) error {
func (d *driver) StopLogging(file string) {
level.Debug(d.logger).Log("msg", "Stop logging", "file", file)
d.mu.Lock()
defer d.mu.Unlock()
lf, ok := d.logs[file]
if ok {
lf.Close()
delete(d.logs, file)
if !ok {
return
}
lf.Close()
delete(d.logs, file)
if !lf.keepFile && lf.jsonl != nil {
// delete the folder where all log files were created.
if err := os.RemoveAll(lf.folder); err != nil {
level.Debug(d.logger).Log("msg", "error deleting folder", "folder", lf.folder)
}
}
d.mu.Unlock()
}

func consumeLog(lf *logPair) {
Expand Down Expand Up @@ -138,10 +165,11 @@ func consumeLog(lf *logPair) {
if err := lf.lokil.Log(&msg); err != nil {
level.Error(lf.logger).Log("msg", "error pushing message to loki", "id", lf.info.ContainerID, "err", err, "message", msg)
}

if err := lf.jsonl.Log(&msg); err != nil {
level.Error(lf.logger).Log("msg", "error writing log message", "id", lf.info.ContainerID, "err", err, "message", msg)
continue
if lf.jsonl != nil {
if err := lf.jsonl.Log(&msg); err != nil {
level.Error(lf.logger).Log("msg", "error writing log message", "id", lf.info.ContainerID, "err", err, "message", msg)
continue
}
}

buf.Reset()
Expand All @@ -156,10 +184,14 @@ func (d *driver) ReadLogs(info logger.Info, config logger.ReadConfig) (io.ReadCl
return nil, fmt.Errorf("logger does not exist for %s", info.ContainerID)
}

if lf.jsonl == nil {
return nil, fmt.Errorf("%s option set to true, no reading capability", cfgNofile)
}

r, w := io.Pipe()
lr, ok := lf.jsonl.(logger.LogReader)
if !ok {
return nil, fmt.Errorf("logger does not support reading")
return nil, errors.New("logger does not support reading")
}

go func() {
Expand Down

0 comments on commit 25aed2b

Please sign in to comment.