Tag based upon namespace

[stevesloka]

Is there a way to have fluentd tag the logs based on the namespace? I have a need to put different application (which are defined by namespace) into different destinations.

[chancez]

Yep. Use the record_transformer with the rewrite_tag_filter plugins like so:

# Query the API for extra metadata.
<filter kubernetes.**>
  type kubernetes_metadata
  # If the logs begin with '{' and end with '}' then it's JSON so merge
  # the JSON log field into the log event
  merge_json_log true
  preserve_json_log true
</filter>

# rewrite_tag_filter does not support nested fields like
# kubernetes.container_name, so this exists to flatten the fields
# so we can use them in our rewrite_tag_filter
<filter kubernetes.**>
  @type record_transformer
  enable_ruby true
  <record>
    kubernetes_namespace_container_name ${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["container_name"]}
  </record>
</filter>

# retag based on the namespace and container name of the log message
<match kubernetes.**>
  @type rewrite_tag_filter
  # Update the tag have a structure of kube.<namespace>.<containername>
  rewriterule1 kubernetes_namespace_container_name  ^(.+)$ kube.$1
</match>

# Remove the unnecessary field as the information is already available on
# other fields.
<filter kube.**>
  @type record_transformer
  remove_keys kubernetes_namespace_container_name
</filter>

# Parse logs in the kube-system namespace using the kubernetes formatter.
<filter kube.kube-system.**>
  type parser
  format kubernetes
  reserve_data true
  key_name log
  suppress_parse_error_log true
</filter>

The filter at the bottom is an example of matching by namespace, you would match the same way with your output plugin.

[stevesloka]

BOOM! Worked perfectly! Thanks so much for the quick and complete response!

[jimmidyson]

Thanks @chancez!

[jvassev]

@stevesloka @chancez

I liked your approach and added some Go code to automate the boring stuff. You can have a look at it here: https://github.com/vmware/kube-fluentd-operator

I've taken the tag-rewriting to the extreme and at the namespace level you can now target a container in a pod based on container labels:

<match $labels(app=nginx)>
  @type ...
</match>

I'd be happy to get your feedback.

Thanks,
Julian

[richm]

<match $labels(app=nginx)>

does this syntax work with fluentd 0.12? We still have to support that version of fluentd.

[jvassev]

$labels is actually a macro: it gets translated to a couple of tag-rewriting <match> directives internally.

As such, it will work with older versions of Fluentd but only in the context of kube-fluentd-operator.

[richm]

$labels is actually a macro: it gets translated to a couple of tag-rewriting directives internally.

Translated by whom? By fluentd? Do you run this through some sort of pre-processor?

[chancez]

@richm If you read his first comment and most recent one he's specifically referring to the kube-fluentd-operator doing the preprocessing.

[shanmukha511]

is there any ways to restrict kube-system namespace logs in fluentd conf?

[mohankrishnavanga]

Hi @chancez,

Our scenario does not have a fluentd interface for logs and we would like to create these in Cloud watch. What changes needs to be the done to the code mentioned above? Please advice. Thanks

[jcantrill]

Hi @chancez,

Our scenario does not have a fluentd interface for logs and we would like to create these in Cloud watch. What changes needs to be the done to the code mentioned above? Please advice. Thanks

What does this mean? " fluentd interface for logs"