caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"

[fluxcap1]

Greetings.

I've setup promtail for forwarding logs to Loki, relevant bits below -

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:

- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/messages

- job_name: zypp-history
  static_configs:
  - targets:
      - localhost
    labels:
      job: zypper-history
      __path__: /var/log/zypp/history
      
- job_name: zypper
  static_configs:
  - targets:
      - localhost
    labels:
      job: zypper
      __path__: /var/log/zypper.log      
      
- job_name: system-wrn
  static_configs:
  - targets:
      - localhost
    labels:
      job: system-warn
      __path__: /var/log/warn

- job_name: alternatives
  static_configs:
  - targets:
      - localhost
    labels:
      job: alternatives
      __path__: /var/log/alternatives.log

- job_name: "suricata-eve" 
  pipeline_stages:
      - json:
          expressions:
            timestamp: timestamp
            message: message
            thread_name: thread_name
            level: level
            trace_id: traceId
            span_id: spanId
            stack_trace: stack_trace
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: RFC3339Nano
      - template:
          source: trace_id
          template: '{{if .Value }} {{- .Value -}} {{else}} {{- "-" -}} {{end}}'
      - template:
          source: span_id
          template: '{{if .Value }} {{- .Value -}} {{else}} {{- "-" -}} {{end}}'
      - template:
          source: stack_trace
          template: '{{if .Value }} {{- " | Stack Trace: " -}}{{- .Value -}} {{else}} {{- " " -}} {{end}}'
      - template:
          source: output
          template: '{{ ToUpper .level }} | Trace[{{ .trace_id }}] Span[{{ .span_id }}] Thread[{{ .thread_name }}] | {{ .message }}{{- .stack_trace -}}'
      - output:
          source: output
  static_configs:
      - targets:
          - localhost
        labels:
          job: "suricata-eve"
          instance: "flux.localdomain" 
          environment: "prod"
          __path__: /var/log/suricata/eve.json

- job_name: "suricata-flowbits" 
  pipeline_stages:
      - json:
          expressions:
            timestamp: timestamp
            message: message
            thread_name: thread_name
            level: level
            trace_id: traceId
            span_id: spanId
            stack_trace: stack_trace
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: RFC3339Nano
      - template:
          source: trace_id
          template: '{{if .Value }} {{- .Value -}} {{else}} {{- "-" -}} {{end}}'
      - template:
          source: span_id
          template: '{{if .Value }} {{- .Value -}} {{else}} {{- "-" -}} {{end}}'
      - template:
          source: stack_trace
          template: '{{if .Value }} {{- " | Stack Trace: " -}}{{- .Value -}} {{else}} {{- " " -}} {{end}}'
      - template:
          source: output
          template: '{{ ToUpper .level }} | Trace[{{ .trace_id }}] Span[{{ .span_id }}] Thread[{{ .thread_name }}] | {{ .message }}{{- .stack_trace -}}'
      - output:
          source: output
  static_configs:
      - targets:
          - localhost
        labels:
          job: "suricata-eve"
          instance: "flux.localdomain" 
          environment: "prod"
          __path__: /var/log/suricata/flowbits.log
      
- job_name: suricata-fastlog
  static_configs:
  - targets:
      - localhost
    labels:
      job: suricata-fast
      __path__: /var/log/suricata/fast.log
      
- job_name: ossec
  static_configs:
  - targets:
      - localhost
    labels:
      job: ossec-alerts
      __path__: /var/ossec/logs/alerts/alerts.log
      
- job_name: zeek
  static_configs:
  - targets:
      - localhost
    labels:
      job: zeek
      __path__: /var/log/zeek/current/*.log
      
- job_name: audit
  static_configs:
  - targets:
      - localhost
    labels:
      job: audit
      __path__: /var/log/audit/audit.log
      
- job_name: postfix-info
  static_configs:
  - targets:
      - localhost
    labels:
      job: postfix-info
      __path__: /var/log/mail.info
      
- job_name: postfix
  static_configs:
  - targets:
      - localhost
    labels:
      job: postfix-mail
      __path__: /var/log/mail
      
- job_name: redis-suricata
  static_configs:
  - targets:
      - localhost
    labels:
      job: redis-suricata   
      __PATH__: /var/log/redis/suricata.log
      
- job_name: redis
  static_configs:
  - targets:
      - localhost
    labels:
      job: redis
      __PATH__: /var/log/redis/default.log
      
- job_name: redis-ntop
  static_configs:
  - targets:
      - localhost
    labels: 
      job: redis-ntop
      __PATH__: /var/log/redis/redis-ntopng.log

- job_name: squid-access
  static_configs:
  - targets:
      - localhost
    labels: 
      job: squid-access
      __PATH__: /var/log/squid/access.log
      
- job_name: squidguard
  static_configs:
  - targets:
      - localhost
    labels: 
      job: squidGuard
      __PATH__: /var/log/squidGuard/squidGuard.log

When I start promtail service I see these warnings -

Aug 31 09:54:15 flux promtail[1299]: level=warn ts=2020-08-31T16:54:15.530589929Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Aug 31 09:54:15 flux promtail[1299]: level=warn ts=2020-08-31T16:54:15.530745717Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Aug 31 09:54:15 flux promtail[1299]: level=warn ts=2020-08-31T16:54:15.530899339Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Aug 31 09:54:15 flux promtail[1299]: level=warn ts=2020-08-31T16:54:15.531049521Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Aug 31 09:54:15 flux promtail[1299]: level=warn ts=2020-08-31T16:54:15.531190367Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Aug 31 09:54:15 flux promtail[1299]: level=warn ts=2020-08-31T16:54:15.53134105Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Aug 31 09:54:15 flux promtail[1299]: level=warn ts=2020-08-31T16:54:15.53148731Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Aug 31 09:54:15 flux promtail[1299]: level=warn ts=2020-08-31T16:54:15.531672872Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Aug 31 09:54:15 flux promtail[1299]: level=info ts=2020-08-31T16:54:15.533223673Z caller=server.go:179 http=[::]:9080 grpc=[::]:24139 msg="server listening on addresses"
Aug 31 09:54:15 flux promtail[1299]: level=info ts=2020-08-31T16:54:15.533507373Z caller=main.go:67 msg="Starting Promtail" version="(version=1.5.0, branch=NA, revision=NA)"

Please let me what I should change in promtail.yaml and fix these warnings.

[slim-bean]

There are a couple things happening here, one of which i'm quite sure is a regression, as I've also seen these warnings.

However, the proper syntax should be to include an empty pipeline_stages: element even if you don't define any stages.

This should silence that warning however I suspect it might not as I mentioned I believe a bug has snuck in here.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[kavirajk]

from what I see here https://github.com/grafana/loki/blob/master/pkg/promtail/targets/file/filetargetmanager.go#L91-L112

I think the issue is treating cfg.EntryParser has default value even though its not present in the config.

This happens in the case where both cfg.PipelineStages and cfg.EntryParser are empty and the log output this warning which is confusing.

Expected behavior would be output this warning only if pipeline.Size() == 0 && cfg.EntryParser != nil. Of course this needs change in the Config to make *cfg.EntryParser (as pointer) and setting it as omitempty.

Not sure if it breaks any compatibility. WDYT? @slim-bean @cyriltovena

also related to #2631

[cyriltovena]

I don't think this will cause compatibility issue. Alternatively if you can't make it a pointer you could also test if it's the default value. Either way both solution works for me.

[slim-bean]

I think it's actually time to just remove support entirely for EntryParser.

It's been deprecated for more than a year now.

[slim-bean]

@kavirajk can you open a PR to remove it?

[kavirajk]

@slim-bean sure! 👍

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[fluxcap1]

Thanks for the hints, I was away and couldn't respond.