Unable to Convert a Static Mode Logs Config to Flow Mode Automatically

[SeamusGrafana]

What's wrong?

I am trying to convert some of my Static Mode configs to Flow Configs, using the convert command.

In this instance, I found that one of my Scrape Jobs was missing from the converted configuration.

In the Static Mode Config, I have a Logs Scrape Job of job_name: integrations/mysql but this is not included in the output.

I tried commenting out a few lines (like the expression), incase this was an issue, but it still did not generate and I cant see why.

While I can manually convert the config easily, the purpose was to test the convert tool (so I could compare to my manual config to see what differences there was).

Steps to reproduce

Create a Static Mode Config with the YAML below.
Convert it to Flow Mode with the convert option, and also including the -b option to bypass the wal_directory issue for Metrics
Check output River file to see if the Logs Scrape Job is included.

System information

Linux

Software version

v0.37.1

Configuration

Static Config;

server:
  log_level: ${GRAFANA_AGENT_LOG_LEVEL:-info}

integrations:

metrics:
  wal_directory: /tmp/grafana-agent-wal
  global:
    external_labels:
      cluster: docker-compose
      source: agent
    scrape_interval: 30s
  configs:
  - name: default
    remote_write:
      - url: http://mimir:8080/api/v1/push
        send_exemplars: true
    scrape_configs:

logs:
  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: integrations/mysql
        static_configs:
          - labels:
              instance: grafana-mysql-db
              job: integrations/mysql
              __path__: /var/log/mysql/error-log.log #/var/log/mysql/*.log
        pipeline_stages:
          - regex:
              expression: '(?P<timestamp>.+) (?P<thread>[\d]+) \[(?P<label>.+?)\]( \[(?P<err_code>.+?)\] \[(?P<subsystem>.+?)\])? (?P<msg>.+)'
          - labels:
              label:
              err_code:
              subsystem:
          - drop:
              expression: "^ *$"
              drop_counter_reason: "drop empty lines"
    clients:
      - url: http://loki:3100/loki/api/v1/push
        external_labels:
          cluster: docker-compose
          source: agent

Flow Mode Converted Config Output;

prometheus.remote_write "metrics_default" {
	external_labels = {
		cluster = "docker-compose",
		source  = "agent",
	}

	endpoint {
		name = "default-de1a95"
		url  = "http://mimir:8080/api/v1/push"

		queue_config { }

		metadata_config { }
	}
}

loki.write "logs_default" {
	endpoint {
		url = "http://loki:3100/loki/api/v1/push"
	}
	external_labels = {
		cluster = "docker-compose",
		source  = "agent",
	}
}

Logs

AGENT_MODE=flow /home/srt/Downloads/grafana-agent-linux-amd64 convert -f static /home/srt/dev/docker-compose/common/grafana-agent/agent_with_logs_sql.yaml -o /home/srt/dev/docker-compose/common/grafana-agent/converted/agent_with_logs_sql.river
Error: (Info) Converted 1 remote_write[s] "default-de1a95" into...
        A prometheus.remote_write.metrics_default component
(Warning) global positions configuration is not supported - each Flow Mode's loki.source.file component has its own positions file in the component's data directory
(Warning) server.log_level is not supported - Flow mode components may produce different logs
(Warning) Please review your agent command line flags and ensure they are set in your Flow mode config file where necessary.
(Error) unsupported wal_directory metrics config was provided. use the run command flag --storage.path for Flow mode instead.

[hainenber]

This is kinda odd but if you add target option in your static_configs, it'd work!

Here's the output after I did that

prometheus.remote_write "metrics_default" {
        external_labels = {
                cluster = "docker-compose",
                source  = "agent",
        }

        endpoint {
                name = "default-de1a95"
                url  = "http://mimir:8080/api/v1/push"

                queue_config { }

                metadata_config { }
        }
}

local.file_match "logs_default_integrations_mysql" {
        path_targets = [{
                __address__ = "localhost",
                __path__    = "/var/log/mysql/error-log.log",
                instance    = "grafana-mysql-db",
                job         = "integrations/mysql",
        }]
}

loki.process "logs_default_integrations_mysql" {
        forward_to = [loki.write.logs_default.receiver]

        stage.regex {
                expression = "(?P<timestamp>.+) (?P<thread>[\\d]+) \\[(?P<label>.+?)\\]( \\[(?P<err_code>.+?)\\] \\[(?P<subsystem>.+?)\\])? (?P<msg>.+)"
        }

        stage.labels {
                values = {
                        err_code  = null,
                        label     = null,
                        subsystem = null,
                }
        }

        stage.drop {
                drop_counter_reason = "drop empty lines"
                expression          = "^ *$"
        }
}

loki.source.file "logs_default_integrations_mysql" {
        targets    = local.file_match.logs_default_integrations_mysql.targets
        forward_to = [loki.process.logs_default_integrations_mysql.receiver]
}

loki.write "logs_default" {
        endpoint {
                url = "http://loki:3100/loki/api/v1/push"
        }
        external_labels = {
                cluster = "docker-compose",
                source  = "agent",
        }
}

Hope this help you

[rfratto]

@erikbaranowski I'm not sure this should be an error:

(Error) unsupported wal_directory metrics config was provided. use the run command flag --storage.path for Flow mode instead.

IMO it's not necessarily a problem that the WAL directory a user specified before won't carry over, and that they'll need to follow up with a flag.

While I'm here, these feel off to me too:

(Warning) server.log_level is not supported - Flow mode components may produce different logs

server.log_level is supported. It's equivalent to log_level in the logging block.

(Warning) global positions configuration is not supported - each Flow Mode's loki.source.file component has its own positions file in the component's data directory

I'm not sure this should be a warning, since there's no action the user might need to take. Maybe this should be info level.

[erikbaranowski]

@rfratto I addressed some of the diagnostic concerns in a PR that was merged to main #5505.

@SeamusGrafana I am looking at the provided config and am tracking down a couple things.

I know this wasn't in the original ask but I want to call this out. The metrics section has no scrape_configs so it resulted in only a prometheus.remote_write component. This doesn't make a lot of sense practically speaking but I'm not sure what that static mode config is accomplishing for metrics as written either.

The logs section is not converting as you expected because it has no targets in the static_configs as @hainenber observed. I'm trying to track down if this behavior is incorrect and this promtail doc leads me to believe a default target of localhost should be used if it isn't set in the static mode config.

• https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#static_configs.
• We do have some tests explicitly showing this:

  agent/converter/internal/promtailconvert/testdata/static_pipeline_example.yaml

  Line 41 in 06c6792

  - labels:

[thampiotr]

I've opened #5546 to fix the main issue with missing localhost default target.

Added a test case that illustrates how the config in the repro instructions will look like after the fix: https://github.com/grafana/agent/pull/5546/files#diff-c188124e35b3f04832c940570a36b7a3a4499c9ac88a9d042072ff537813b117