Add a field mapping transformer

[xcaptain]

A raw log may contain many fields but at last we may only care some of them and want them to have a different name, For example, a transformer produces log like

{
    "a": 1,
    "c": "some non-sence data"
}

we may want to rename field a to b and remove field c and get a result like

{
    "b": 1
}

currently the add_fields and remove_fields transformer can not achieve this goal.

[xcaptain]

I made a simple attempt to implement this idea, not hard to implement

[xcaptain]

I added config file parser for this map_fields transformer, unit test and function test passed. Below is a simple example:

vector_test.toml

[transforms.map_fields]
  type = "fieldmapper"
  inputs = ["transform_timestamp"]
[transforms.map_fields.fields]
  status = "http_status"

[sinks.my_sink1]
  type = "console"
  inputs = ["map_fields"]
  target = "stdout"
  encoding = "json"

[sinks.my_sink2]
  type = "console"
  inputs = ["transform_timestamp"]
  target = "stdout"
  encoding = "json"

Using this transformer I filtered status field and then renamed it to http_status, the other fields are ignored.

[lukesteensen]

Thanks for opening this @xcaptain! You're right that there is a gap in those transforms around renaming fields.

This is a good example of a situation where I'd recommend using the lua transform. Here's the equivalent of your example config:

[transforms.map_fields]
  type = "lua"
  inputs = ["transform_timestamp"]
  source = """
event["http_status"] = event["status"]
"""

[sinks.my_sink1]
  type = "console"
  inputs = ["map_fields"]
  target = "stdout"
  encoding = "json"

[sinks.my_sink2]
  type = "console"
  inputs = ["transform_timestamp"]
  target = "stdout"
  encoding = "json"

If you want to remove the old status field, you can add event["status"] = nil.

We are also in the process of planning a more general transform that will consolidate adding, removing, renaming, coercing, etc into a single more powerful transform (see #750).

[lukesteensen]

Closing in favor of #750

[xcaptain]

The lua transform is modify event inplace, if I want to remove other fields I have to use remove_fields, not very convenient but it's ok to use. looking forward to your new transform,and wish this project get better.