loki_out: add stuctured_metadata_map_keys

[0x006EA1E5]

Resolves #9463

• Adds stuctured_metadata_map_keys config to dynamically populate stuctured_metadata from a map

The below is the results for running this with /docker_compose/loki-grafana-structured_metadata_map/, which has input

        {
          "message": "simple log generated",
          "logger": "my.logger",
          "level": "INFO",
          "hostname": "localhost",
          "my_map_of_attributes_1": {
            "key_1": "hello, world!",
            "key_2": "goodbye, world!"
          },
          "my_map_of_maps_1": {
            "root_key": {
              "sub_key_1": "hello, world!",
              "sub_key_2": "goodbye, world!"
            }
          }
        }

I used the following to query Loki: curl http://localhost:3100/loki/api/v1/query_range --data-urlencode 'query={service_name="test"}' --data-urlencode 'limit=1' --data-urlencode 'step=5' | jq '.data.result'

This is the output with the option enabled. As you can see, the stream section contains the keys for the structured metadata:

[
  {
    "stream": {
      "hostname": "localhost",
      "key_1": "hello, world!",
      "key_2": "goodbye, world!",
      "level": "INFO",
      "logger": "my.logger",
      "service_name": "test",
      "sub_key_1": "hello, world!",
      "sub_key_2": "goodbye, world!"
    },
    "values": [
      [
        "1740489069234881956",
        "simple log generated"
      ]
    ]
  }
]

And here is the output with the feature not enabled. This also removes my_map_of_attributes_1 and my_map_of_maps_1 from remove_keys. As you can see, key_1, key_2, sub_key_1andsub_key_2are no longer in thestream` section:

[
  {
    "stream": {
      "hostname": "localhost",
      "level": "INFO",
      "logger": "my.logger",
      "service_name": "test"
    },
    "values": [
      [
        "1740489174234932650",
        "message=\"simple log generated\" my_map_of_attributes_1=\"map[key_1:\"hello, world!\" key_2:\"goodbye, world!\"]\" my_map_of_maps_1=\"map[root_key:\"map[sub_key_1:\"hello, world!\" sub_key_2:\"goodbye, world!\"]\"]\""
      ]
    ]
  }
]

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

fluent/fluent-bit-docs#1527

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

@0x006EA1E5 would you please add a JSON example that shows the before the option is set and the after so we can review the behavior intended to be implemented ?

thank you

[0x006EA1E5]

@0x006EA1E5 would you please add a JSON example that shows the before the option is set and the after so we can review the behavior intended to be implemented ?

thank you

Hey @edsiper, what kind of JSON do you need?

This only touches the Loki output, so I could include:

• A JSON log input
• The JSON output from querying Loki against the ingested log
  • Both before and after the option is set

Would this be okay?
FYI, this can be tested via the included docker-compose in /docker_compose/loki-grafana-structured_metadata_map/

[0x006EA1E5]

@0x006EA1E5 would you please add a JSON example that shows the before the option is set and the after so we can review the behavior intended to be implemented ?
thank you

Hey @edsiper, what kind of JSON do you need?

This only touches the Loki output, so I could include:

• A JSON log input

• The JSON output from querying Loki against the ingested log

  • Both before and after the option is set

Would this be okay? FYI, this can be tested via the included docker-compose in /docker_compose/loki-grafana-structured_metadata_map/

The below is the results for running this with /docker_compose/loki-grafana-structured_metadata_map/, which has input

        {
          "message": "simple log generated",
          "logger": "my.logger",
          "level": "INFO",
          "hostname": "localhost",
          "my_map_of_attributes_1": {
            "key_1": "hello, world!",
            "key_2": "goodbye, world!"
          },
          "my_map_of_maps_1": {
            "root_key": {
              "sub_key_1": "hello, world!",
              "sub_key_2": "goodbye, world!"
            }
          }
        }

I used the following to query Loki: curl http://localhost:3100/loki/api/v1/query_range --data-urlencode 'query={service_name="test"}' --data-urlencode 'limit=1' --data-urlencode 'step=5' | jq '.data.result'

This is the output with the option enabled. As you can see, the stream section contains the keys for the structured metadata:

[
  {
    "stream": {
      "hostname": "localhost",
      "key_1": "hello, world!",
      "key_2": "goodbye, world!",
      "level": "INFO",
      "logger": "my.logger",
      "service_name": "test",
      "sub_key_1": "hello, world!",
      "sub_key_2": "goodbye, world!"
    },
    "values": [
      [
        "1740489069234881956",
        "simple log generated"
      ]
    ]
  }
]

And here is the output with the feature not enabled. This also removes my_map_of_attributes_1 and my_map_of_maps_1 from remove_keys. As you can see, key_1, key_2, sub_key_1andsub_key_2are no longer in thestream` section:

[
  {
    "stream": {
      "hostname": "localhost",
      "level": "INFO",
      "logger": "my.logger",
      "service_name": "test"
    },
    "values": [
      [
        "1740489174234932650",
        "message=\"simple log generated\" my_map_of_attributes_1=\"map[key_1:\"hello, world!\" key_2:\"goodbye, world!\"]\" my_map_of_maps_1=\"map[root_key:\"map[sub_key_1:\"hello, world!\" sub_key_2:\"goodbye, world!\"]\"]\""
      ]
    ]
  }
]

[patrick-stephens]

Ignore the container failures - it is a permissions problem I think for forks, they build ok but fail to push.

[patrick-stephens]

Build and pushed manually: ghcr.io/fluent/fluent-bit:pr-9530

docker buildx build -t ghcr.io/fluent/fluent-bit:pr-9530 --platform=linux/amd64 --target=production --push=true .

[0x006EA1E5]

Build and pushed manually: ghcr.io/fluent/fluent-bit:pr-9530

docker buildx build -t ghcr.io/fluent/fluent-bit:pr-9530 --platform=linux/amd64 --target=production --push=true .

This image is now available for us, and my testing confirms it is working as expected 👍