Set measurement name based on matched pattern in logparser

[botzill]

Hi.

Seems to not be possible to define smth like this? Can we somehow define it without redefining the hole [[inputs.logparser]] again?

[[inputs.logparser]]
  files = ["/var/log/supervisor/app.log"]
  from_beginning = false
  watch_method = "poll"
  [inputs.logparser.grok]
    measurement = "measurement_1"
    custom_pattern_files = ["..."]
    patterns = ["..."]
  [inputs.logparser.grok]
    measurement = "measurement_2"
    custom_pattern_files = ["..."]
    patterns = ["..."]
  [inputs.logparser.tags]
    app = "tag1"

[danielnelson]

The best you can do is to have a multiple patterns:

[[inputs.logparser]]
  files = ["/var/log/supervisor/app.log"]
  from_beginning = false
  watch_method = "poll"
  [inputs.logparser.grok]
    patterns = [
      "pattern1",
      "pattern2"
    ]
    custom_pattern_files = [
	  "CP1 ...",
	  "CP2 ..."
    ]

Once you have that working you may also want to combine your patterns for increased performance:

[[inputs.logparser]]
  files = ["/var/log/supervisor/app.log"]
  from_beginning = false
  watch_method = "poll"
  [inputs.logparser.grok]
    patterns = [
      "%{PATTERN1}|%{PATTERN2}"
    ]
    custom_pattern_files = [
      "PATTERN1 ...",
      "PATTERN2 ...",
	  "CP1 ...",
	  "CP2 ..."
    ]

Unfortunately you won't be able to name the measurements differently.

[botzill]

I see. Don't you think such option with be useful ?

[danielnelson]

I think it will be useful, though I will have to think about how this can be expressed in the configuration. Perhaps we could have parameter like patterns that is a table so that the individual patterns can be named.

[maxunt]

We could add a new modifier for grok patterns, such as "name", that when matched would change the measurement name to the parsed grok field

[danielnelson]

I think this would be a good thing to do, but it wouldn't allow you to set the measurement to a value not in the parsed data. @botzill would this meet your requirements?

[danielnelson]

@botzill with the change in #4433 you can set the measurement name from a matched pattern. This isn't exactly what is described above, but I think it will be a good solution for setting a dynamic measurement name, and it fits in well with the existing grok pattern language. Please let us know if it doesn't work for your use-case and we can consider additional changes.