[BUG] [parser /container] Error with Reduced max_log_size in OpenTelemetry Collector's Filelog Receiver

[Masmiiadm]

Component(s)

pkg/stanza

What happened?

Description

I based the new log parsing setup for the OpenTelemetry Collector on this documentation, and everything worked perfectly with the default configuration.

However, when I tried reducing the max_log_size parameter in the filelogreceiver configuration from its default value of 1 MiB to 40 KiB, I encountered the following error
"data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/kiali-operator_kiali-kiali-operator-5f5f969b89-7vqcx_bd93551f-ff0e-413e-95b1-4fd8420f2690/operator/0.log", "error": "failed to detect a valid container log format: entry cannot be parsed as container logs" "deployment\": {\".\": {}, \"f:instanceName\": {}, \"f:namespace\": {}}, \"f:environment\": {\".\": {}, \"f:isKubernetes\": {}}, \"f....

Expected Result

the fileconsumer parses these logs successfully like the 1MiB default configuration

Actual Result

the filelog receiver and th container parser failed to detect a valid container log format when configure max_log_size 40 KiB

Collector version

0.111.0

Environment information

No response

OpenTelemetry Collector configuration

apiVersion: v1
data:
  relay: |
    exporters:
      debug: {}
    extensions:
      health_check:
        endpoint: ${env:MY_POD_IP}:13133
    processors:
      batch: {}
      k8sattributes:
        extract:
          metadata:
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.statefulset.name
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
        filter:
          node_from_env_var: K8S_NODE_NAME
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
    receivers:
      filelog:
        exclude:
        - /var/log/pods/*/opentelemetry-collector/*.log
        include:
        - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
        max_log_size: 40KiB
        operators:
        - id: container-parser
          type: container
        retry_on_failure:
          enabled: true
        start_at: end
    service:
      extensions:
      - health_check
      pipelines:
        logs:
          exporters:
          - debug
          processors:
          - transform
          - k8sattributes
          - memory_limiter
          - resource
          - batch
          receivers:
          - filelog
      telemetry:
        logs:
          level: info
        metrics:
          address: ${env:MY_POD_IP}:8888

Log output

2024-10-28T10:46:19.065Z  error reader/reader.go:148  process: %w {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/kiali-operator_kiali-kiali-operator-5f5f969b89-7vqcx_bd93551f-ff0e-413e-95b1-4fd8420f2690/operator/0.log", "error": "failed to detect a valid container log format: entry cannot be parsed as container logs: [\\\"openid\\\", \\\"profile\\\", \\\"email\\\"], \\\"username_claim\\\": \\\"sub\\\"}, \\\"openshift\\\": {\\\"client_id_prefix\\\": \\\"kiali\\\"}, \\\"strategy\\\": \\\"anonymous\\\"}, \\\"custom_dashboards\\\": [], \\\"deployment\\\": {\\\"accessible_namespaces\\\": [\\\"**\\\"], \\\"affinity\\\": {\\\"node\\\": {}, \\\"pod\\\": {}, \\\"pod_anti\\\": {}}, \\\"hpa\\\":

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

cc @ChrsMark

[ChrsMark]

Hey @Masmiiadm, could you share a sample log line that causes that failure? From what I can see in the Log Output section of the issue, the entry does not really look like a valid format? Or what I see is the trimmed log line?

I assume that when the max_log_size is set the log lines can be trimmed. In that case the container format might have been corrupted and seeing the parser failing actually makes sense.

[Masmiiadm]

Hi @ChrsMark . you will find the file 0.log attached
0.log

[ChrsMark]

Hi @ChrsMark . you will find the file 0.log attached 0.log

The link does not work for me. Just a single log line in a comment will be fine, no need to share the whole log file :).

[Masmiiadm]

@ChrsMark , bellow the log line , thx
' {"level":"error","ts":1730728178.0375009,"logger":"runner","msg":"Using /etc/ansible/ansible.cfg as config file\r\n\r\nPLAY [localhost] ******************************************************************************************************\n\r\nTASK [debug] *******************************************************************\nok: [localhost] => {\r\n "msg": "KIALI RECONCILIATION START"\r\n}\n\r\nTASK [debug] *******************************************************************\nok: [localhost] => {\r\n "msg": "CR: name=kiali, namespace=kiali-operator"\r\n}\n\r\nTASK [Playbook start time] *****************************************************\nok: [localhost] => {"ansible_facts": {"playbook_time_start": "2024-11-04 13:49:13"}, "changed": false}\n\r\nTASK [Determine the default playbook] ******************************************\nok: [localhost] => {"ansible_facts": {"default_playbook": {"playbook": "default"}}, "ansible_included_var_files": ["/opt/ansible/playbooks/default-playbook.yml"], "changed": false}\n\r\nTASK [Determine the version that is to be installed] ***************************\nok: [localhost] => {"ansible_facts": {"version": "default"}, "changed": false}\n\r\nTASK [Determine the default supported images for all known versions] ***********\nok: [localhost] => {"ansible_facts": {"supported_kiali_images": {"default": {"image_name": "quay.io/kiali/kiali", "image_version": "operator_version"}, "v1.12": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.12"}, "v1.24": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.24"}, "v1.36": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.36"}}}, "ansible_included_var_files": ["/opt/ansible/playbooks/default-supported-images.yml"], "changed": false}\n\r\nTASK [Override the supported images if found in the environment] ***************\nskipping: [localhost] => (item={'key': 'default', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'operator_version'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "default", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "operator_version"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.12', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.12'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.12", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.12"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.24', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.24'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.24", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.24"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.36', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.36'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.36", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.36"}}, "skip_reason": "Conditional result was False"}\n\r\nTASK [Examine environment and determine if supported image for the requested version is overridden even if no default is known] ***\nskipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}\n\r\nTASK [Make sure a default supported image or an override image is known] *******\nskipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}\n\r\nTASK [Run the version-specific deploy role] ************************************\n\r\nTASK [default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":'

[ChrsMark]

Thank's @Masmiiadm! One more question: what container runtime your k8s cluster runs with?

Actually what we need to check here is what is stored in the log file at /var/log/pods/kiali-operator_kiali-kiali-operator-5f5f969b89-7vqcx_bd93551f-ff0e-413e-95b1-4fd8420f2690/operator/0.log (sorry for not mentioning that earlier). Could you tail that file and share a complete log line?

I think that for cri (containerd and cri-o) formats even with max_log_size set the parser should be able to detect the format since the format information is only on the "head" of the line. For Docker I don't think we can do much since it's actually json and if the json is incomplete the unmarshaling will fail anyways.

[Masmiiadm]

Hi @ChrsMark , the log format is containerd ,
2024-11-05T06:38:48.568172379+01:00 stderr F {"debug":"info","ts":1730785128.5658307,"logger","msg":"Using /etc/ansible/ansible.cfg as config file\r\n\r\nPLAY [localhost] ******************************************************************************************************\n\r\nTASK [debug] *******************************************************************\nok: [localhost] => {\r\n "msg": "KIALI RECONCILIATION START"\r\n}\n\r\nTASK [debug] *******************************************************************\nok: [localhost] => {\r\n "msg": "CR: name=kiali, namespace=kiali-operator"\r\n}\n\r\nTASK [Playbook start time] *****************************************************\nok: [localhost] => {"ansible_facts": {"playbook_time_start": "2024-11-04 13:49:13"}, "changed": false}\n\r\nTASK [Determine the default playbook] ******************************************\nok: [localhost] => {"ansible_facts": {"default_playbook": {"playbook": "default"}}, "ansible_included_var_files": ["/opt/ansible/playbooks/default-playbook.yml"], "changed": false}\n\r\nTASK [Determine the version that is to be installed] ***************************\nok: [localhost] => {"ansible_facts": {"version": "default"}, "changed": false}\n\r\nTASK [Determine the default supported images for all known versions] ***********\nok: [localhost] => {"ansible_facts": {"supported_kiali_images": {"default": {"image_name": "quay.io/kiali/kiali", "image_version": "operator_version"}, "v1.12": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.12"}, "v1.24": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.24"}, "v1.36": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.36"}}}, "ansible_included_var_files": ["/opt/ansible/playbooks/default-supported-images.yml"], "changed": false}\n\r\nTASK [Override the supported images if found in the environment] ***************\nskipping: [localhost] => (item={'key': 'default', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'operator_version'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "default", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "operator_version"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.12', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.12'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.12", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.12"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.24', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.24'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.24", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.24"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.36', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.36'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.36", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.36"}}, "skip_reason": "Conditional result was False"}\n\r\nTASK [Examine environment and determine if supported image for the requested version is overridden even if no default is known] ***\nskipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}\n\r\nTASK [Make sure a default supported image or an override image is known] *******\nskipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}\n\r\nTASK [Run the version-specific deploy role] ************************************\n\r\nTASK [default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsTy"namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsTy"FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":'

[ChrsMark]

Ok, I was able to reproduce this by setting max_log_size: 40.

What happens here is that the log line is split based on the max_log_size. So in that case the first 40 bytes would be a valid containerd log but then the rest of the chunks would only be random strings that will fail to get parsed as container logs.

In that case the failure makes sense. Not sure what the container parser could do differently here. Maybe we could just introduce a setting to ignore such failures in case they are coming from chunked log lines?
send_quiet should be able to handle this but right now it's not working as expected: #35726
@djaglowski any thoughts here?