0.13.2 crash at msgpack_sbuffer_write #611

solsson · 2018-06-01T12:57:43Z

Looks a bit similar to #557 (comment) but that one is reportedly fixed. I have a crashlooping pod with

[2018/06/01 12:01:55] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[engine] caught signal (SIGSEGV)
Fluent-Bit v0.13.2
Copyright (C) Treasure Data

#0  0x7f238c5480d4      in  ???() at ???:0
#1  0x55e2f84f6ede      in  msgpack_sbuffer_write() at lib/msgpack-2.1.3/include/msgpack/sbuffer.h:84
#2  0x55e2f870aca5      in  msgpack_pack_ext_body() at lib/msgpack-2.1.3/include/msgpack/pack_template.h:890
#3  0x55e2f870aca5      in  msgpack_pack_object() at lib/msgpack-2.1.3/src/objectc.c:72
#4  0x55e2f84f78c0      in  pack_map_content() at plugins/filter_kubernetes/kubernetes.c:321
#5  0x55e2f84f8129      in  cb_kube_filter() at plugins/filter_kubernetes/kubernetes.c:493
#6  0x55e2f8497aea      in  flb_filter_do() at src/flb_filter.c:86
#7  0x55e2f849553c      in  flb_input_dbuf_write_end() at include/fluent-bit/flb_input.h:642
#8  0x55e2f849709c      in  flb_input_dyntag_append_raw() at src/flb_input.c:894
#9  0x55e2f84bbb1d      in  process_content() at plugins/in_tail/tail_file.c:290
#10 0x55e2f84bc968      in  flb_tail_file_chunk() at plugins/in_tail/tail_file.c:651
#11 0x55e2f84ba357      in  in_tail_collect_static() at plugins/in_tail/tail.c:129
#12 0x55e2f84975db      in  flb_input_collector_fd() at src/flb_input.c:995
#13 0x55e2f849e370      in  flb_engine_handle_event() at src/flb_engine.c:296
#14 0x55e2f849e370      in  flb_engine_start() at src/flb_engine.c:515
#15 0x55e2f843e606      in  main() at src/fluent-bit.c:824
#16 0x7f238c4e42e0      in  ???() at ???:0
#17 0x55e2f843ca89      in  ???() at ???:0
#18 0xffffffffffffffff  in  ???() at ???:0

I'll try to get back with more details as comments.

The text was updated successfully, but these errors were encountered:

solsson · 2018-06-01T17:10:11Z

With Log_Level debug:

[2018/06/01 16:59:13] [debug] [in_tail] file=/var/log/containers/metrics-server-v0.2.1-7f8dd98c8f-f625c_kube-system_metrics-server-35823808ebf2ec5d704b1f9a32218304d3da9c6d05ce0811b15657e7bd71e34c.log promote to TAIL_EVENT
[2018/06/01 16:59:13] [debug] [in_tail] file=/var/log/containers/metrics-server-v0.2.1-7f8dd98c8f-f625c_kube-system_metrics-server-6a72cbc9ad98c33a417347dcb326f102fac3cba300df17016894847fc8fe4845.log promote to TAIL_EVENT
[2018/06/01 16:59:13] [debug] [in_tail] file=/var/log/containers/metrics-server-v0.2.1-7f8dd98c8f-f625c_kube-system_metrics-server-nanny-d6c433290505d923b3967809656a9e2542146d82b29f146cc1275613002f5b10.log promote to TAIL_EVENT
[2018/06/01 16:59:13] [debug] [in_tail] file=/var/log/containers/nginx-ingress-controller-8dcfb95b9-ndxck_ingress-nginx_nginx-ingress-controller-24c5184a4558e6050ee444a17563c7142c1d8cfd80b6703d199650e39016487e.log promote to TAIL_EVENT
[2018/06/01 16:59:13] [debug] [in_tail] file=/var/log/containers/node-exporter-htn6m_monitoring_kube-rbac-proxy-311e2be04eb5e4557e47464c107fe865943d777f08c4b911448be175a5005d3c.log promote to TAIL_EVENT
[2018/06/01 16:59:13] [debug] [in_tail] file=/var/log/containers/node-exporter-htn6m_monitoring_node-exporter-57c1f3ca81d9d152887edc04150cfb441fc7c1a969d7870dc396e2df7737d23c.log promote to TAIL_EVENT
[2018/06/01 16:59:14] [debug] [filter_kube] API Server (ns=site, pod=notifications-8685b9d8b9-jx248) http_do=0, HTTP Status: 200
[2018/06/01 16:59:14] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/01 16:59:14] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/01 16:59:14] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/01 16:59:14] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/01 16:59:14] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/01 16:59:14] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/01 16:59:14] [debug] [filter_kube] could not merge JSON log as requested
[engine] caught signal (SIGSEGV)
Fluent-Bit v0.13.2
Copyright (C) Treasure Data

#0  0x7f206238b0d4      in  ???() at ???:0
#1  0x5557a67abede      in  msgpack_sbuffer_write() at lib/msgpack-2.1.3/include/msgpack/sbuffer.h:84
#2  0x5557a69bfca5      in  msgpack_pack_ext_body() at lib/msgpack-2.1.3/include/msgpack/pack_template.h:890
#3  0x5557a69bfca5      in  msgpack_pack_object() at lib/msgpack-2.1.3/src/objectc.c:72
#4  0x5557a67ac8c0      in  pack_map_content() at plugins/filter_kubernetes/kubernetes.c:321
#5  0x5557a67ad129      in  cb_kube_filter() at plugins/filter_kubernetes/kubernetes.c:493
#6  0x5557a674caea      in  flb_filter_do() at src/flb_filter.c:86
#7  0x5557a674a53c      in  flb_input_dbuf_write_end() at include/fluent-bit/flb_input.h:642
#8  0x5557a674c09c      in  flb_input_dyntag_append_raw() at src/flb_input.c:894
#9  0x5557a6770b1d      in  process_content() at plugins/in_tail/tail_file.c:290
#10 0x5557a6771968      in  flb_tail_file_chunk() at plugins/in_tail/tail_file.c:651
#11 0x5557a676f357      in  in_tail_collect_static() at plugins/in_tail/tail.c:129
#12 0x5557a674c5db      in  flb_input_collector_fd() at src/flb_input.c:995
#13 0x5557a6753370      in  flb_engine_handle_event() at src/flb_engine.c:296
#14 0x5557a6753370      in  flb_engine_start() at src/flb_engine.c:515
#15 0x5557a66f3606      in  main() at src/fluent-bit.c:824
#16 0x7f20623272e0      in  ???() at ???:0
#17 0x5557a66f1a89      in  ???() at ???:0
#18 0xffffffffffffffff  in  ???() at ???:0

How can I figure out which file is being processed when the error occurs?

edsiper · 2018-06-04T22:40:13Z

would you please check in the log (the content above the one provided) which log file was being processed ?

solsson · 2018-06-05T07:04:28Z

Those logs are gone now. I will reopen this issue if I see it again.

solsson · 2018-06-05T07:18:19Z

Happened again now :)

What is the message I'm looking for that identifies the file being processed? Below I've copied everything form the second before the crash, but omitted lots of "Could not merge JSON as requested" (we do have some not so strict json formatted logs).

[2018/06/05 07:08:08] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:08] [debug] [input tail.0] [mem buf] size = 152689
[2018/06/05 07:08:08] [debug] [in_tail] file=/var/log/containers/boards-6694f97b8d-sjh8k_nryw_yolean-boards-eb58e0894e273224b1e03085663409244e5a762e4fbb41421940c6983674a12a.log read=32767 lines=73
[2018/06/05 07:08:08] [debug] [filter_kube] API Server (ns=yolean, pod=boards-6bf9ffb678-9hcsj) http_do=0, HTTP Status: 200
[2018/06/05 07:08:08] [debug] [filter_kube] could not merge JSON log as requested
...
[2018/06/05 07:08:08] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:08] [debug] [input tail.0] [mem buf] size = 233374
[2018/06/05 07:08:08] [debug] [in_tail] file=/var/log/containers/boards-6bf9ffb678-9hcsj_yolean_yolean-boards-46b201d230fee124c54bdc6d188a4f18aadfea279b90db27d0d43801f8e7656a.log read=32767 lines=78
[2018/06/05 07:08:08] [debug] [in_tail] file=/var/log/containers/boards-7844bbdcf7-z2p8w_xyz_yolean-boards-86d2625a7ab30155481c055470052639cb0f53a599983255dab30ed9196b4e7c.log promote to TAIL_EVENT
[2018/06/05 07:08:08] [debug] [filter_kube] API Server (ns=leanconstruction, pod=boards-785d9d746c-8286c) http_do=0, HTTP Status: 200
[2018/06/05 07:08:08] [debug] [filter_kube] could not merge JSON log as requested
...
[2018/06/05 07:08:08] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:08] [debug] [input tail.0] [mem buf] size = 324249
[2018/06/05 07:08:08] [debug] [in_tail] file=/var/log/containers/boards-785d9d746c-8286c_leanconstruction_yolean-boards-ba0917c7fe94f537e8ee0890070abc56f0f3bf742c0d653d061dbbfd6bd4ed4d.log read=32767 lines=114
[2018/06/05 07:08:08] [debug] [in_tail] file=/var/log/containers/cp-55b4b9f8bd-2f4ws_ch_exporter-a2319826edd23c74ba439542467af263606aa40ba95780262337eceeb1daf83a.log promote to TAIL_EVENT
[2018/06/05 07:08:08] [debug] [filter_kube] API Server (ns=ch, pod=cp-55b4b9f8bd-2f4ws) http_do=0, HTTP Status: 200
[2018/06/05 07:08:09] [debug] [filter_kube] could not merge JSON log as requested
...
[2018/06/05 07:08:09] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:09] [debug] [input tail.0] [mem buf] size = 508520
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/fluent-bit-swk9k_logging_fluent-bit-3a26b35c3eedca7b663feb94a4eb444108d06c5af7abb706c9cb5be4a5662f06.log read=32767 lines=177
[2018/06/05 07:08:09] [debug] [filter_kube] could not merge JSON log as requested
...
[2018/06/05 07:08:09] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:09] [debug] [input tail.0] [mem buf] size = 528013
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/fluent-bit-swk9k_logging_fluent-bit-441daa067b95726c6433d3244e2375192bb44627ca3e8c783a49efad6fde5178.log read=4228 lines=30
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/integrations-56f745bd46-fbc9z_rrrr_integrations-6990c59822e611529a1a5a9c04f9ed53acbafa2358f2696674a18b7a54046c29.log promote to TAIL_EVENT
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/integrations-5cfdfcf657-dmbn4_xyz_integrations-27f1fa1b3231d853c18ccafde3ffe707734c3b21cbfe7a76a004291259d5ac52.log promote to TAIL_EVENT
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/integrations-5cfdfcf657-dmbn4_xyz_integrations-5fba2c3e7dadd43cf5826febb1d9d3324b488f85ede8661cbc9fc2cc54df7079.log promote to TAIL_EVENT
[2018/06/05 07:08:09] [debug] [filter_kube] API Server (ns=keycloak, pod=kc-757959b446-mrdmk) http_do=0, HTTP Status: 200
[2018/06/05 07:08:09] [debug] [filter_kube] could not merge JSON log as requested
..
[2018/06/05 07:08:09] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:09] [debug] [input tail.0] [mem buf] size = 537068
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/kc-757959b446-mrdmk_keycloak_keycloak-d8d7abc8ba15c87b001c2e2ea7c9c348d145c8af703d1e430c73d0d5f49a4b6f.log read=5874 lines=9
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/kube-dns-7fcdddb4c5-m55pk_kube-system_dnsmasq-a109a517660c4577dfa0394a9cad024e795dcc3c5e5c1b4fa7f58411c882441c.log promote to TAIL_EVENT
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/kube-dns-7fcdddb4c5-m55pk_kube-system_kubedns-ceffa42898da7d4b7adc915413a31f05e7218070f434532c06a046ef460af953.log promote to TAIL_EVENT
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/kube-dns-7fcdddb4c5-m55pk_kube-system_prometheus-to-sd-9b93e6a02ebe9260b38eddeaa67cb5348a85d9e2642e923020feb09c6f9f232a.log promote to TAIL_EVENT
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/kube-dns-7fcdddb4c5-m55pk_kube-system_sidecar-30eaf57cd537f05a2aa630858b687578b0ac02d689dcd439aab2710116da797e.log promote to TAIL_EVENT
[2018/06/05 07:08:09] [debug] [in_tail] file=/var/log/containers/kube-proxy-gke-eu-west-3-b1-highmem-2-pool-d275075a-c9c4_kube-system_kube-proxy-7821096b718bb66f419cef20a177102f4b6dcfc4b98d6208af459651f76efd10.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [filter_kube] API Server (ns=xyz, pod=live-5db689db6b-d5rvf) http_do=0, HTTP Status: 200
[2018/06/05 07:08:10] [debug] [input tail.0] [mem buf] size = 630864
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/live-5db689db6b-d5rvf_xyz_live-b73c7f1c5107b03b585435a50727130bad74ae341182563e309183b7babd7c35.log read=32767 lines=106
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/logs-mv1-6975744b7d-8cr97_sites_logs-mv1-9299893ccc4c282c4f3c07fb96c898169544043f904d1a3cf750a07fbda05e13.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/lop-59d65d6cbc-28ckj_vxyz_yolean-lop-8f5b35dee53efa78082c154c79ca786214f07b21a6c90464a04e3b48ef1c1499.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/lop-b6d4c6555-ncn5z_nryw_yolean-lop-43309690b10c88471223be048cd38f6211b195558e46d75129c5946df243e64c.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/metrics-server-v0.2.1-7f8dd98c8f-f625c_kube-system_metrics-server-35823808ebf2ec5d704b1f9a32218304d3da9c6d05ce0811b15657e7bd71e34c.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/metrics-server-v0.2.1-7f8dd98c8f-f625c_kube-system_metrics-server-6a72cbc9ad98c33a417347dcb326f102fac3cba300df17016894847fc8fe4845.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/metrics-server-v0.2.1-7f8dd98c8f-f625c_kube-system_metrics-server-nanny-d6c433290505d923b3967809656a9e2542146d82b29f146cc1275613002f5b10.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/nginx-ingress-controller-8dcfb95b9-ndxck_ingress-nginx_nginx-ingress-controller-24c5184a4558e6050ee444a17563c7142c1d8cfd80b6703d199650e39016487e.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/node-exporter-htn6m_monitoring_kube-rbac-proxy-311e2be04eb5e4557e47464c107fe865943d777f08c4b911448be175a5005d3c.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [in_tail] file=/var/log/containers/node-exporter-htn6m_monitoring_node-exporter-57c1f3ca81d9d152887edc04150cfb441fc7c1a969d7870dc396e2df7737d23c.log promote to TAIL_EVENT
[2018/06/05 07:08:10] [debug] [filter_kube] API Server (ns=zyx-qa, pod=notifications-8685b9d8b9-jx248) http_do=0, HTTP Status: 200
[2018/06/05 07:08:10] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:10] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:10] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:10] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:10] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:10] [debug] [filter_kube] could not merge JSON log as requested
[2018/06/05 07:08:10] [debug] [filter_kube] could not merge JSON log as requested
[engine] caught signal (SIGSEGV)
Fluent-Bit v0.13.2
Copyright (C) Treasure Data

#0  0x7fcecb9110d4      in  ???() at ???:0
#1  0x564e00f75ede      in  msgpack_sbuffer_write() at lib/msgpack-2.1.3/include/msgpack/sbuffer.h:84
#2  0x564e01189ca5      in  msgpack_pack_ext_body() at lib/msgpack-2.1.3/include/msgpack/pack_template.h:890
#3  0x564e01189ca5      in  msgpack_pack_object() at lib/msgpack-2.1.3/src/objectc.c:72
#4  0x564e00f768c0      in  pack_map_content() at plugins/filter_kubernetes/kubernetes.c:321
#5  0x564e00f77129      in  cb_kube_filter() at plugins/filter_kubernetes/kubernetes.c:493
#6  0x564e00f16aea      in  flb_filter_do() at src/flb_filter.c:86
#7  0x564e00f1453c      in  flb_input_dbuf_write_end() at include/fluent-bit/flb_input.h:642
#8  0x564e00f1609c      in  flb_input_dyntag_append_raw() at src/flb_input.c:894
#9  0x564e00f3ab1d      in  process_content() at plugins/in_tail/tail_file.c:290
#10 0x564e00f3b968      in  flb_tail_file_chunk() at plugins/in_tail/tail_file.c:651
#11 0x564e00f39357      in  in_tail_collect_static() at plugins/in_tail/tail.c:129
#12 0x564e00f165db      in  flb_input_collector_fd() at src/flb_input.c:995
#13 0x564e00f1d370      in  flb_engine_handle_event() at src/flb_engine.c:296
#14 0x564e00f1d370      in  flb_engine_start() at src/flb_engine.c:515
#15 0x564e00ebd606      in  main() at src/fluent-bit.c:824
#16 0x7fcecb8ad2e0      in  ???() at ???:0
#17 0x564e00ebba89      in  ???() at ???:0
#18 0xffffffffffffffff  in  ???() at ???:0

solsson · 2018-06-05T07:25:07Z

@edsiper I should rephrase the question actually: If fluent-bit crashes, how do I find out which log file it was processing? If that can be done without switching to debug level, for example through the state db, it would be great.

jlpettersson · 2018-06-05T11:49:13Z

We have the same problem.

chlunde · 2018-06-05T13:23:02Z

reproducer:

/tmp/x/filename.log

{"log":"    fie {\"a\":\"b\"}\n","stream":"stdout","time":"2018-06-04T13:16:03.859189337Z"}

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On
    # Command      |  Decoder | Field | Optional Action
    # =============|==================|=================
    Decode_Field_As   escaped    log

[SERVICE]
   Flush              1
   Log_Level          info
   Daemon             off
   Parsers_File       parsers.conf

[INPUT]
   Name                tail
   Tag                 kube.*
   Path                /tmp/x/*.log
   Parser              docker

[FILTER]
   Name                kubernetes
   Match               kube.*
   dummy_meta On
   Merge_log           On

[OUTPUT]
   Name                NULL
   Match               *

michiel · 2018-06-06T11:44:53Z

@chlunde FYI - I'm on 0.14-dev and I can't reproduce it there. Using your config and the two lines you gave for reproduction and stdout for OUTPUT, my output is,

~/dev/github/fluent-bit/build(*) » ./bin/fluent-bit -v -c /tmp/fluent-crash.conf                                                                
Fluent-Bit v0.14.0
Copyright (C) Treasure Data

[2018/06/06 21:39:20] [ info] [engine] started (pid=4342)
[2018/06/06 21:39:20] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443
[2018/06/06 21:39:20] [ warn] [filter_kube] using Dummy Metadata
[0] kube.tmp.fluent-crash-log.txt: [1528118163.859189337, {"log"=>"    fie {"a":"b"}", "stream"=>"stdout", "time"=>"2018-06-04T13:16:03.859189337Z", "kubernetes"=>{"dummy"=>"Wed Jun  6 21:39:20 2018"}}]
[1] kube.tmp.fluent-crash-log.txt: [1528118163.859189337, {"log"=>"    fields {"k8s_namespace":"@{FILE['/etc/downward-api/namespace']}","k8s_pod.name":"@{FILE['/etc/downward-api/name']}","k8s_labels":"@{FILE['/etc/downward-api/labels']}"}", "stream"=>"stdout", "time"=>"2018-06-04T13:16:03.859189337Z", "kubernetes"=>{"dummy"=>"Wed Jun  6 21:39:20 2018"}}]

chlunde · 2018-06-06T13:14:57Z

@michiel I guess it depends on compilation options, OS version and more. This was on RHEL 7.5. When I compiled it with AFL it did not crash on that exact output. Try running it with valgrind, or print out the type after the call to flb_pack_json in plugins/filter_kubernetes/kubernetes.c. I get a boolean, and the code assumes it is a map without checking. f is parsed as false, I guess because it's assumed to be a valid JSON keyword.

solsson · 2018-06-07T09:39:28Z

@edsiper Will you be pushing 0.14-dev builds to fluent/fluent-bit?

michiel · 2018-06-07T12:12:23Z

@solsson I'm pushing master builds + my PRs to https://hub.docker.com/r/sddmelb/fluent-bit/tags/

The tag travis-106 is 0.14-dev + #618 - it's pushed from travis and the build history is here https://travis-ci.org/michiel/fluent-bit/builds/388727243

edsiper · 2018-06-07T16:05:03Z

thanks everyone, the fix provided by @chlunde should fix the problem. I've merged the changes into GIT master and I will release a new 0.13.3 today.

solsson · 2018-06-09T04:12:11Z

Unfortunately I get such a crash still

[2018/06/09 03:58:21] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[engine] caught signal (SIGSEGV)
Fluent-Bit v0.13.3
Copyright (C) Treasure Data

#0  0x7febf51e00d4      in  ???() at ???:0
#1  0x563216e30f9e      in  msgpack_sbuffer_write() at lib/msgpack-2.1.3/include/msgpack/sbuffer.h:84
#2  0x563217044d75      in  msgpack_pack_ext_body() at lib/msgpack-2.1.3/include/msgpack/pack_template.h:890

I've tried to switch to the NULL output as in @chlunde's repro case but I don't get any lines at all before the crash.

the build from fluent/fluent-bit#611 (comment)

ae-dg · 2018-06-12T09:20:18Z

I'm having the same issue. I have narrowed it a bit. I have two lines in a log file that causes a SIGSEGV:

{"log":"+ echo 'The database is ready.'\n","stream":"stderr","time":"2018-06-11T14:37:30.661464705Z"}
{"log":"now \n","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}

If I remove the space between the "now" and "\n" on the second line I don't get the crash. If I remove the first line (leaving in the space in the other line) it doesn't crash but there is no output from the first line. Lines added to the log afterwards are output though.

Also if I rename the log file I get a warning 'invalid pattern for given tag kube.var.log.containers....' but it also doesn't crash with the space in it.

Hope this helps.

Signed-off-by: Eduardo Silva <[email protected]>

edsiper · 2018-06-13T00:54:46Z

thanks all for the information and test cases.

I've pushed a fix into GIT master which avoids the problem described by @ae-dg . If anyone of you can build from master and test it would be awesome.

solsson · 2018-06-13T05:19:40Z

@michiel can you push a new build from master with your travis setup?

michiel · 2018-06-13T13:03:22Z

@solsson I've rebased against master and pushed out the changes. As above, this is master + open PR #618 (which only affects filter_modify).

Travis build is https://travis-ci.org/michiel/fluent-bit/builds/391740777 - that's now up as tag travis-107 at https://hub.docker.com/r/sddmelb/fluent-bit/tags/

solsson · 2018-06-14T05:04:41Z

I've gone from ~300 crashes per day to zero with the latest fix. Thanks a lot everyone for the effort.

Signed-off-by: Eduardo Silva <[email protected]>

edsiper · 2018-06-15T23:24:31Z

thanks again, I will do a new release shortly.

edsiper · 2018-06-15T23:37:31Z

https://fluentbit.io/announcements/v0.13.4/

the doc has a duplicated options tag_prefix Signed-off-by: Evgeniy Shubin <[email protected]>

solsson closed this as completed Jun 5, 2018

solsson reopened this Jun 5, 2018

chlunde mentioned this issue Jun 7, 2018

flb_pack: fix crash with stricter JSON parsing #620

Closed

edsiper self-assigned this Jun 7, 2018

edsiper added bug fixed labels Jun 7, 2018

solsson mentioned this issue Jun 9, 2018

Release: v0.13.3 fluent/fluent-bit-kubernetes-logging#25

Closed

solsson added a commit to Yolean/fluent-bit-kubernetes-logging that referenced this issue Jun 10, 2018

Getting the msgpack crash with this build too,

7c231c1

the build from fluent/fluent-bit#611 (comment)

edsiper added a commit that referenced this issue Jun 13, 2018

filter_kubernetes: fix index adjustment and validate object type (#611)

175bc1a

Signed-off-by: Eduardo Silva <[email protected]>

edsiper added work-in-process and removed fixed labels Jun 13, 2018

michiel mentioned this issue Jun 15, 2018

Feature request for using tag varialbe to route output messages #629

Closed

edsiper added a commit that referenced this issue Jun 15, 2018

filter_kubernetes: fix index adjustment and validate object type (#611)

85a5c6e

Signed-off-by: Eduardo Silva <[email protected]>

edsiper added fixed and removed work-in-process labels Jun 15, 2018

edsiper closed this as completed Jun 15, 2018

rawahars pushed a commit to rawahars/fluent-bit that referenced this issue Oct 24, 2022

Delete duplicated record (fluent#611)

d0b5935

the doc has a duplicated options tag_prefix Signed-off-by: Evgeniy Shubin <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.13.2 crash at msgpack_sbuffer_write #611

0.13.2 crash at msgpack_sbuffer_write #611

solsson commented Jun 1, 2018 •

edited

Loading

solsson commented Jun 1, 2018

edsiper commented Jun 4, 2018

solsson commented Jun 5, 2018 •

edited

Loading

solsson commented Jun 5, 2018

solsson commented Jun 5, 2018

jlpettersson commented Jun 5, 2018

chlunde commented Jun 5, 2018

michiel commented Jun 6, 2018

chlunde commented Jun 6, 2018

solsson commented Jun 7, 2018

michiel commented Jun 7, 2018

edsiper commented Jun 7, 2018

solsson commented Jun 9, 2018

ae-dg commented Jun 12, 2018 •

edited

Loading

edsiper commented Jun 13, 2018

solsson commented Jun 13, 2018

michiel commented Jun 13, 2018 •

edited

Loading

solsson commented Jun 14, 2018 •

edited

Loading

edsiper commented Jun 15, 2018

edsiper commented Jun 15, 2018

0.13.2 crash at msgpack_sbuffer_write #611

0.13.2 crash at msgpack_sbuffer_write #611

Comments

solsson commented Jun 1, 2018 • edited Loading

solsson commented Jun 1, 2018

edsiper commented Jun 4, 2018

solsson commented Jun 5, 2018 • edited Loading

solsson commented Jun 5, 2018

solsson commented Jun 5, 2018

jlpettersson commented Jun 5, 2018

chlunde commented Jun 5, 2018

michiel commented Jun 6, 2018

chlunde commented Jun 6, 2018

solsson commented Jun 7, 2018

michiel commented Jun 7, 2018

edsiper commented Jun 7, 2018

solsson commented Jun 9, 2018

ae-dg commented Jun 12, 2018 • edited Loading

edsiper commented Jun 13, 2018

solsson commented Jun 13, 2018

michiel commented Jun 13, 2018 • edited Loading

solsson commented Jun 14, 2018 • edited Loading

edsiper commented Jun 15, 2018

edsiper commented Jun 15, 2018

solsson commented Jun 1, 2018 •

edited

Loading

solsson commented Jun 5, 2018 •

edited

Loading

ae-dg commented Jun 12, 2018 •

edited

Loading

michiel commented Jun 13, 2018 •

edited

Loading

solsson commented Jun 14, 2018 •

edited

Loading