Crash on startup #1362

rolandjitsu · 2019-05-29T02:36:35Z

Bug Report

Describe the bug

fluent-bit keeps crashing on startup.

To Reproduce

Rubular link if applicable:
Example log message if applicable:
I'm running fluent-bit as a service, so here are the logs for journal:

May 29 02:26:00 kenpi systemd[1]: Started Fluent Bit.
May 29 02:26:00 kenpi fluent-bit[28585]: Fluent Bit v1.1.1
May 29 02:26:00 kenpi fluent-bit[28585]: Copyright (C) Treasure Data
May 29 02:26:00 kenpi fluent-bit[28585]: [engine] caught signal (SIGSEGV)
May 29 02:26:00 kenpi systemd[1]: fluent-bit.service: Main process exited, code=killed, status=6/ABRT
May 29 02:26:00 kenpi systemd[1]: fluent-bit.service: Unit entered failed state.
May 29 02:26:00 kenpi systemd[1]: fluent-bit.service: Failed with result 'signal'.
May 29 02:26:05 kenpi systemd[1]: fluent-bit.service: Service hold-off time over, scheduling restart.
May 29 02:26:05 kenpi systemd[1]: Stopped Fluent Bit.

And the fluent-bit logs:

[2019/05/29 02:27:20] [ info] [storage] initializing...
[2019/05/29 02:27:20] [ info] [storage] root path '/home/pi/config/fluent-bit/buffer/'
[2019/05/29 02:27:20] [ info] [storage] normal synchronization mode, checksum enabled
[2019/05/29 02:27:20] [ info] [storage] backlog input plugin: storage_backlog.19
[2019/05/29 02:27:20] [ info] [engine] started (pid=28895)
[2019/05/29 02:27:20] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=20
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/fbit-metrics.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/fbit-metrics.log, offset=648966
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=27
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/pastri.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/pastri.log, offset=88982407
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=34
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/edfa.log
[2019/05/29 02:27:20] [debug] [in_tail] Cannot read info from: /home/pi/logs/edfa.log
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=40
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/sfp.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/sfp.log, offset=74235800
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=47
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/power-scan.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/power-scan.log, offset=0
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=54
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/aeat6600.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/aeat6600.log, offset=10690843
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=61
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/acm0.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/acm0.log, offset=23867279
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=68
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/acm1.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/acm1.log, offset=192007948
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=75
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/heartbeat.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/heartbeat.log, offset=1520464
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=82
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/reboots.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/reboots.log, offset=0
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=89
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/repo-status.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/repo-status.log, offset=1698
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=90
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/wireguard-keepalive.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/wireguard-keepalive.log, offset=28999
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=103
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/iperf-client.log
[2019/05/29 02:27:20] [debug] [in_tail] Cannot read info from: /home/pi/logs/iperf-client.log
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=109
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/disk-usage.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/disk-usage.log, offset=8060
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=116
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/network.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/network.log, offset=1203112
[2019/05/29 02:27:20] [debug] [in_tail] inotify watch fd=123
[2019/05/29 02:27:20] [debug] [in_tail] scanning path /home/pi/logs/gpu.log
[2019/05/29 02:27:20] [debug] [in_tail] add to scan queue /home/pi/logs/gpu.log, offset=460833
[2019/05/29 02:27:20] [ info] [storage backlog] queue memory limit: 4.8M
[2019/05/29 02:27:20] [debug] [filter_modify] Initialized modify filter with 0 conditions and 6 rules
[2019/05/29 02:27:20] [debug] [filter_modify] Initialized modify filter with 0 conditions and 11 rules
[2019/05/29 02:27:20] [debug] [router] match rule tail.0:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.1:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.2:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.3:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.4:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.5:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.6:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.7:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.8:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.9:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.10:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.11:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.12:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule cpu.13:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.14:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule mem.15:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.16:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule tail.17:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule proc.18:forward.0
[2019/05/29 02:27:20] [debug] [router] match rule storage_backlog.19:forward.0
[2019/05/29 02:27:20] [ info] [http_server] listen iface=127.0.0.1 tcp_port=2020
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/fbit-metrics.log promote to TAIL_EVENT
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/pastri.log read=8347 lines=70
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/sfp.log read=4096 lines=57
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/power-scan.log promote to TAIL_EVENT
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/aeat6600.log read=894 lines=6
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/acm0.log read=4209 lines=25
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/acm1.log read=16628 lines=61
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/heartbeat.log read=106 lines=1
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/reboots.log promote to TAIL_EVENT
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/repo-status.log promote to TAIL_EVENT
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/wireguard-keepalive.log promote to TAIL_EVENT
[2019/05/29 02:27:20] [debug] [in_tail] file=/home/pi/logs/disk-usage.log promote to TAIL_EVENT
[2019/05/29 02:27:25] [debug] [storage] [cio scan] opening path /home/pi/config/fluent-bit/buffer/
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: fluent_bit
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: pastri
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: edfa
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: sfp
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: power_scan
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: encoder
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: beacon
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: imu
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: heartbeat
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: reboots
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: repo_status
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: wireguard
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: iperf
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: cpu_usage
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: disk_usage
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: memory_usage
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: net
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: gpu_info
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: pastri_ps
[2019/05/29 02:27:25] [debug] [storage] [cio stream] new stream registered: storage_backlog.19
[2019/05/29 02:27:25] [ info] [storage] initializing...
[2019/05/29 02:27:25] [ info] [storage] root path '/home/pi/config/fluent-bit/buffer/'
[2019/05/29 02:27:25] [ info] [storage] normal synchronization mode, checksum enabled
[2019/05/29 02:27:25] [ info] [storage] backlog input plugin: storage_backlog.19
[2019/05/29 02:27:25] [ info] [engine] started (pid=28918)
[2019/05/29 02:27:25] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=20
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/fbit-metrics.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/fbit-metrics.log, offset=648966
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=28
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/pastri.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/pastri.log, offset=88990754
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=21
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/edfa.log
[2019/05/29 02:27:25] [debug] [in_tail] Cannot read info from: /home/pi/logs/edfa.log
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=35
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/sfp.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/sfp.log, offset=74239847
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=41
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/power-scan.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/power-scan.log, offset=0
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=54
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/aeat6600.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/aeat6600.log, offset=10691737
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=55
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/acm0.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/acm0.log, offset=23871425
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=68
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/acm1.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/acm1.log, offset=192024355
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=75
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/heartbeat.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/heartbeat.log, offset=1520570
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=82
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/reboots.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/reboots.log, offset=0
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=89
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/repo-status.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/repo-status.log, offset=1698
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=96
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/wireguard-keepalive.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/wireguard-keepalive.log, offset=28999
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=103
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/iperf-client.log
[2019/05/29 02:27:25] [debug] [in_tail] Cannot read info from: /home/pi/logs/iperf-client.log
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=109
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/disk-usage.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/disk-usage.log, offset=8060
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=116
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/network.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/network.log, offset=1203112
[2019/05/29 02:27:25] [debug] [in_tail] inotify watch fd=123
[2019/05/29 02:27:25] [debug] [in_tail] scanning path /home/pi/logs/gpu.log
[2019/05/29 02:27:25] [debug] [in_tail] add to scan queue /home/pi/logs/gpu.log, offset=460833
[2019/05/29 02:27:25] [ info] [storage backlog] queue memory limit: 4.8M
[2019/05/29 02:27:25] [debug] [filter_modify] Initialized modify filter with 0 conditions and 6 rules
[2019/05/29 02:27:25] [debug] [filter_modify] Initialized modify filter with 0 conditions and 11 rules
[2019/05/29 02:27:25] [debug] [router] match rule tail.0:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.1:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.2:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.3:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.4:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.5:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.6:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.7:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.8:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.9:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.10:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.11:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.12:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule cpu.13:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.14:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule mem.15:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.16:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule tail.17:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule proc.18:forward.0
[2019/05/29 02:27:25] [debug] [router] match rule storage_backlog.19:forward.0
[2019/05/29 02:27:25] [ info] [http_server] listen iface=127.0.0.1 tcp_port=2020
[2019/05/29 02:27:25] [debug] [in_tail] file=/home/pi/logs/fbit-metrics.log promote to TAIL_EVENT
[2019/05/29 02:27:25] [debug] [in_tail] file=/home/pi/logs/pastri.log read=8103 lines=68
[2019/05/29 02:27:25] [debug] [in_tail] file=/home/pi/logs/sfp.log read=7347 lines=101
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/power-scan.log promote to TAIL_EVENT
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/aeat6600.log read=894 lines=6
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/acm0.log read=63 lines=0
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/acm1.log read=16605 lines=61
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/heartbeat.log read=106 lines=1
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/reboots.log promote to TAIL_EVENT
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/repo-status.log promote to TAIL_EVENT
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/wireguard-keepalive.log promote to TAIL_EVENT
[2019/05/29 02:27:26] [debug] [in_tail] file=/home/pi/logs/disk-usage.log promote to TAIL_EVENT
[2019/05/29 02:27:31] [debug] [storage] [cio scan] opening path /home/pi/config/fluent-bit/buffer/
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: fluent_bit
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: pastri
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: edfa
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: sfp
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: power_scan
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: encoder
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: beacon
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: imu
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: heartbeat
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: reboots
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: repo_status
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: wireguard
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: iperf
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: cpu_usage
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: disk_usage
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: memory_usage
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: net
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: gpu_info
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: pastri_ps
[2019/05/29 02:27:31] [debug] [storage] [cio stream] new stream registered: storage_backlog.19
[2019/05/29 02:27:31] [ info] [storage] initializing...
[2019/05/29 02:27:31] [ info] [storage] root path '/home/pi/config/fluent-bit/buffer/'
[2019/05/29 02:27:31] [ info] [storage] normal synchronization mode, checksum enabled
[2019/05/29 02:27:31] [ info] [storage] backlog input plugin: storage_backlog.19
[2019/05/29 02:27:31] [ info] [engine] started (pid=28928)
[2019/05/29 02:27:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=20
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/fbit-metrics.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/fbit-metrics.log, offset=648966
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=27
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/pastri.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/pastri.log, offset=88998857
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=34
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/edfa.log
[2019/05/29 02:27:31] [debug] [in_tail] Cannot read info from: /home/pi/logs/edfa.log
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=35
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/sfp.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/sfp.log, offset=74247145
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=46
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/power-scan.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/power-scan.log, offset=0
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=54
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/aeat6600.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/aeat6600.log, offset=10692631
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=61
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/acm0.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/acm0.log, offset=23871425
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=68
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/acm1.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/acm1.log, offset=192040764
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=75
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/heartbeat.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/heartbeat.log, offset=1520676
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=82
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/reboots.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/reboots.log, offset=0
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=89
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/repo-status.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/repo-status.log, offset=1698
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=96
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/wireguard-keepalive.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/wireguard-keepalive.log, offset=28999
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=103
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/iperf-client.log
[2019/05/29 02:27:31] [debug] [in_tail] Cannot read info from: /home/pi/logs/iperf-client.log
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=109
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/disk-usage.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/disk-usage.log, offset=8060
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=116
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/network.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/network.log, offset=1203112
[2019/05/29 02:27:31] [debug] [in_tail] inotify watch fd=123
[2019/05/29 02:27:31] [debug] [in_tail] scanning path /home/pi/logs/gpu.log
[2019/05/29 02:27:31] [debug] [in_tail] add to scan queue /home/pi/logs/gpu.log, offset=460833
[2019/05/29 02:27:31] [ info] [storage backlog] queue memory limit: 4.8M
[2019/05/29 02:27:31] [debug] [filter_modify] Initialized modify filter with 0 conditions and 6 rules
[2019/05/29 02:27:31] [debug] [filter_modify] Initialized modify filter with 0 conditions and 11 rules
[2019/05/29 02:27:31] [debug] [router] match rule tail.0:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.1:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.2:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.3:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.4:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.5:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.6:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.7:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.8:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.9:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.10:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.11:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.12:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule cpu.13:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.14:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule mem.15:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.16:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule tail.17:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule proc.18:forward.0
[2019/05/29 02:27:31] [debug] [router] match rule storage_backlog.19:forward.0
[2019/05/29 02:27:31] [ info] [http_server] listen iface=127.0.0.1 tcp_port=2020
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/fbit-metrics.log promote to TAIL_EVENT
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/pastri.log read=8700 lines=73
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/sfp.log read=7348 lines=101
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/power-scan.log promote to TAIL_EVENT
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/aeat6600.log read=745 lines=5
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/acm0.log read=4159 lines=25
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/acm1.log read=16580 lines=61
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/heartbeat.log read=106 lines=1
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/reboots.log promote to TAIL_EVENT
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/repo-status.log promote to TAIL_EVENT
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/wireguard-keepalive.log promote to TAIL_EVENT
[2019/05/29 02:27:31] [debug] [in_tail] file=/home/pi/logs/disk-usage.log promote to TAIL_EVENT
[2019/05/29 02:27:37] [debug] [storage] [cio scan] opening path /home/pi/config/fluent-bit/buffer/
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: fluent_bit
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: pastri
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: edfa
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: sfp
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: power_scan
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: encoder
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: beacon
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: imu
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: heartbeat
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: reboots
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: repo_status
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: wireguard
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: iperf
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: cpu_usage
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: disk_usage
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: memory_usage
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: net
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: gpu_info
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: pastri_ps
[2019/05/29 02:27:37] [debug] [storage] [cio stream] new stream registered: storage_backlog.19
[2019/05/29 02:27:37] [ info] [storage] initializing...
[2019/05/29 02:27:37] [ info] [storage] root path '/home/pi/config/fluent-bit/buffer/'
[2019/05/29 02:27:37] [ info] [storage] normal synchronization mode, checksum enabled
[2019/05/29 02:27:37] [ info] [storage] backlog input plugin: storage_backlog.19
[2019/05/29 02:27:37] [ info] [engine] started (pid=28956)
[2019/05/29 02:27:37] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=20
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/fbit-metrics.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/fbit-metrics.log, offset=648966
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=27
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/pastri.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/pastri.log, offset=89007557
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=34
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/edfa.log
[2019/05/29 02:27:37] [debug] [in_tail] Cannot read info from: /home/pi/logs/edfa.log
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=40
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/sfp.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/sfp.log, offset=74254444
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=47
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/power-scan.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/power-scan.log, offset=0
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=54
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/aeat6600.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/aeat6600.log, offset=10693376
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=61
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/acm0.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/acm0.log, offset=23875574
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=68
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/acm1.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/acm1.log, offset=192057171
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=75
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/heartbeat.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/heartbeat.log, offset=1520782
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=82
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/reboots.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/reboots.log, offset=0
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=89
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/repo-status.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/repo-status.log, offset=1698
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=96
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/wireguard-keepalive.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/wireguard-keepalive.log, offset=28999
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=103
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/iperf-client.log
[2019/05/29 02:27:37] [debug] [in_tail] Cannot read info from: /home/pi/logs/iperf-client.log
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=109
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/disk-usage.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/disk-usage.log, offset=8060
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=116
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/network.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/network.log, offset=1203112
[2019/05/29 02:27:37] [debug] [in_tail] inotify watch fd=123
[2019/05/29 02:27:37] [debug] [in_tail] scanning path /home/pi/logs/gpu.log
[2019/05/29 02:27:37] [debug] [in_tail] add to scan queue /home/pi/logs/gpu.log, offset=460833
[2019/05/29 02:27:37] [ info] [storage backlog] queue memory limit: 4.8M
[2019/05/29 02:27:37] [debug] [filter_modify] Initialized modify filter with 0 conditions and 6 rules
[2019/05/29 02:27:37] [debug] [filter_modify] Initialized modify filter with 0 conditions and 11 rules
[2019/05/29 02:27:37] [debug] [router] match rule tail.0:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.1:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.2:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.3:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.4:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.5:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.6:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.7:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.8:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.9:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.10:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.11:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.12:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule cpu.13:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.14:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule mem.15:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.16:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule tail.17:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule proc.18:forward.0
[2019/05/29 02:27:37] [debug] [router] match rule storage_backlog.19:forward.0
[2019/05/29 02:27:37] [ info] [http_server] listen iface=127.0.0.1 tcp_port=2020
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/fbit-metrics.log promote to TAIL_EVENT
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/pastri.log read=9139 lines=77
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/sfp.log read=3251 lines=44
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/power-scan.log promote to TAIL_EVENT
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/aeat6600.log read=894 lines=6
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/acm0.log read=10 lines=0
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/acm1.log read=16557 lines=61
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/heartbeat.log read=106 lines=1
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/reboots.log promote to TAIL_EVENT
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/repo-status.log promote to TAIL_EVENT
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/wireguard-keepalive.log promote to TAIL_EVENT
[2019/05/29 02:27:37] [debug] [in_tail] file=/home/pi/logs/disk-usage.log promote to TAIL_EVENT
[2019/05/29 02:27:43] [debug] [storage] [cio scan] opening path /home/pi/config/fluent-bit/buffer/
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: fluent_bit
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: pastri
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: edfa
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: sfp
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: power_scan
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: encoder
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: beacon
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: imu
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: heartbeat
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: reboots
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: repo_status
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: wireguard
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: iperf
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: cpu_usage
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: disk_usage
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: memory_usage
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: net
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: gpu_info
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: pastri_ps
[2019/05/29 02:27:43] [debug] [storage] [cio stream] new stream registered: storage_backlog.19
[2019/05/29 02:27:43] [ info] [storage] initializing...
[2019/05/29 02:27:43] [ info] [storage] root path '/home/pi/config/fluent-bit/buffer/'
[2019/05/29 02:27:43] [ info] [storage] normal synchronization mode, checksum enabled
[2019/05/29 02:27:43] [ info] [storage] backlog input plugin: storage_backlog.19
[2019/05/29 02:27:43] [ info] [engine] started (pid=28984)
[2019/05/29 02:27:43] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=20
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/fbit-metrics.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/fbit-metrics.log, offset=648966
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=27
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/pastri.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/pastri.log, offset=89016696
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=34
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/edfa.log
[2019/05/29 02:27:43] [debug] [in_tail] Cannot read info from: /home/pi/logs/edfa.log
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=40
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/sfp.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/sfp.log, offset=74257695
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=47
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/power-scan.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/power-scan.log, offset=0
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=54
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/aeat6600.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/aeat6600.log, offset=10694270
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=61
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/acm0.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/acm0.log, offset=23875574
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=63
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/acm1.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/acm1.log, offset=192073579
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=74
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/heartbeat.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/heartbeat.log, offset=1520888
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=82
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/reboots.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/reboots.log, offset=0
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=89
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/repo-status.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/repo-status.log, offset=1698
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=96
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/wireguard-keepalive.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/wireguard-keepalive.log, offset=28999
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=103
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/iperf-client.log
[2019/05/29 02:27:43] [debug] [in_tail] Cannot read info from: /home/pi/logs/iperf-client.log
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=109
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/disk-usage.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/disk-usage.log, offset=8060
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=116
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/network.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/network.log, offset=1203112
[2019/05/29 02:27:43] [debug] [in_tail] inotify watch fd=123
[2019/05/29 02:27:43] [debug] [in_tail] scanning path /home/pi/logs/gpu.log
[2019/05/29 02:27:43] [debug] [in_tail] add to scan queue /home/pi/logs/gpu.log, offset=460833
[2019/05/29 02:27:43] [ info] [storage backlog] queue memory limit: 4.8M
[2019/05/29 02:27:43] [debug] [filter_modify] Initialized modify filter with 0 conditions and 6 rules
[2019/05/29 02:27:43] [debug] [filter_modify] Initialized modify filter with 0 conditions and 11 rules
[2019/05/29 02:27:43] [debug] [router] match rule tail.0:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.1:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.2:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.3:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.4:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.5:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.6:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.7:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.8:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.9:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.10:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.11:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.12:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule cpu.13:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.14:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule mem.15:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.16:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule tail.17:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule proc.18:forward.0
[2019/05/29 02:27:43] [debug] [router] match rule storage_backlog.19:forward.0
[2019/05/29 02:27:43] [ info] [http_server] listen iface=127.0.0.1 tcp_port=2020
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/fbit-metrics.log promote to TAIL_EVENT
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/pastri.log read=8465 lines=71
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/sfp.log read=7298 lines=101
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/power-scan.log promote to TAIL_EVENT
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/aeat6600.log read=894 lines=6
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/acm0.log read=4106 lines=24
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/acm1.log read=12437 lines=46
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/heartbeat.log read=212 lines=2
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/reboots.log promote to TAIL_EVENT
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/repo-status.log promote to TAIL_EVENT
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/wireguard-keepalive.log promote to TAIL_EVENT
[2019/05/29 02:27:43] [debug] [in_tail] file=/home/pi/logs/disk-usage.log promote to TAIL_EVENT
[2019/05/29 02:27:49] [debug] [storage] [cio scan] opening path /home/pi/config/fluent-bit/buffer/
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: fluent_bit
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: pastri
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: edfa
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: sfp
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: power_scan
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: encoder
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: beacon
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: imu
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: heartbeat
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: reboots
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: repo_status
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: wireguard
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: iperf
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: cpu_usage
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: disk_usage
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: memory_usage
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: net
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: gpu_info
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: pastri_ps
[2019/05/29 02:27:49] [debug] [storage] [cio stream] new stream registered: storage_backlog.19
[2019/05/29 02:27:49] [ info] [storage] initializing...
[2019/05/29 02:27:49] [ info] [storage] root path '/home/pi/config/fluent-bit/buffer/'
[2019/05/29 02:27:49] [ info] [storage] normal synchronization mode, checksum enabled
[2019/05/29 02:27:49] [ info] [storage] backlog input plugin: storage_backlog.19
[2019/05/29 02:27:49] [ info] [engine] started (pid=28996)
[2019/05/29 02:27:49] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=20
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/fbit-metrics.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/fbit-metrics.log, offset=648966
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=27
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/pastri.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/pastri.log, offset=89025161
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=34
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/edfa.log
[2019/05/29 02:27:49] [debug] [in_tail] Cannot read info from: /home/pi/logs/edfa.log
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=40
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/sfp.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/sfp.log, offset=74264993
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=47
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/power-scan.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/power-scan.log, offset=0
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=48
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/aeat6600.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/aeat6600.log, offset=10695164
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=61
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/acm0.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/acm0.log, offset=23879558
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=67
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/acm1.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/acm1.log, offset=192085953
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=75
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/heartbeat.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/heartbeat.log, offset=1521100
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=81
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/reboots.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/reboots.log, offset=0
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=89
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/repo-status.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/repo-status.log, offset=1698
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=96
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/wireguard-keepalive.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/wireguard-keepalive.log, offset=28999
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=97
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/iperf-client.log
[2019/05/29 02:27:49] [debug] [in_tail] Cannot read info from: /home/pi/logs/iperf-client.log
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=109
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/disk-usage.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/disk-usage.log, offset=8060
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=110
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/network.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/network.log, offset=1203112
[2019/05/29 02:27:49] [debug] [in_tail] inotify watch fd=123
[2019/05/29 02:27:49] [debug] [in_tail] scanning path /home/pi/logs/gpu.log
[2019/05/29 02:27:49] [debug] [in_tail] add to scan queue /home/pi/logs/gpu.log, offset=460833
[2019/05/29 02:27:49] [ info] [storage backlog] queue memory limit: 4.8M
[2019/05/29 02:27:49] [debug] [filter_modify] Initialized modify filter with 0 conditions and 6 rules
[2019/05/29 02:27:49] [debug] [filter_modify] Initialized modify filter with 0 conditions and 11 rules
[2019/05/29 02:27:49] [debug] [router] match rule tail.0:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.1:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.2:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.3:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.4:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.5:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.6:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.7:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.8:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.9:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.10:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.11:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.12:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule cpu.13:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.14:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule mem.15:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.16:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule tail.17:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule proc.18:forward.0
[2019/05/29 02:27:49] [debug] [router] match rule storage_backlog.19:forward.0
[2019/05/29 02:27:49] [ info] [http_server] listen iface=127.0.0.1 tcp_port=2020
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/fbit-metrics.log promote to TAIL_EVENT
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/pastri.log read=8456 lines=71
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/sfp.log read=7298 lines=101
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/power-scan.log promote to TAIL_EVENT
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/aeat6600.log read=894 lines=6
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/acm0.log read=122 lines=0
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/acm1.log read=16447 lines=61
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/heartbeat.log read=106 lines=1
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/reboots.log promote to TAIL_EVENT
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/repo-status.log promote to TAIL_EVENT
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/wireguard-keepalive.log promote to TAIL_EVENT
[2019/05/29 02:27:49] [debug] [in_tail] file=/home/pi/logs/disk-usage.log promote to TAIL_EVENT
[2019/05/29 02:27:54] [debug] [storage] [cio scan] opening path /home/pi/config/fluent-bit/buffer/
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: fluent_bit
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: pastri
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: edfa
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: sfp
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: power_scan
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: encoder
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: beacon
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: imu
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: heartbeat
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: reboots
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: repo_status
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: wireguard
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: iperf
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: cpu_usage
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: disk_usage
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: memory_usage
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: net
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: gpu_info
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: pastri_ps
[2019/05/29 02:27:54] [debug] [storage] [cio stream] new stream registered: storage_backlog.19
[2019/05/29 02:27:54] [ info] [storage] initializing...
[2019/05/29 02:27:54] [ info] [storage] root path '/home/pi/config/fluent-bit/buffer/'
[2019/05/29 02:27:54] [ info] [storage] normal synchronization mode, checksum enabled
[2019/05/29 02:27:54] [ info] [storage] backlog input plugin: storage_backlog.19
[2019/05/29 02:27:54] [ info] [engine] started (pid=29004)
[2019/05/29 02:27:54] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=20
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/fbit-metrics.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/fbit-metrics.log, offset=648966
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=26
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/pastri.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/pastri.log, offset=89033617
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=34
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/edfa.log
[2019/05/29 02:27:54] [debug] [in_tail] Cannot read info from: /home/pi/logs/edfa.log
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=40
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/sfp.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/sfp.log, offset=74272291
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=47
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/power-scan.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/power-scan.log, offset=0
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=54
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/aeat6600.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/aeat6600.log, offset=10696058
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=61
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/acm0.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/acm0.log, offset=23879558
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=68
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/acm1.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/acm1.log, offset=192102359
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=75
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/heartbeat.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/heartbeat.log, offset=1521206
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=82
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/reboots.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/reboots.log, offset=0
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=89
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/repo-status.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/repo-status.log, offset=1698
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=97
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/wireguard-keepalive.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/wireguard-keepalive.log, offset=28999
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=103
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/iperf-client.log
[2019/05/29 02:27:54] [debug] [in_tail] Cannot read info from: /home/pi/logs/iperf-client.log
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=109
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/disk-usage.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/disk-usage.log, offset=8060
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=117
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/network.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/network.log, offset=1203112
[2019/05/29 02:27:54] [debug] [in_tail] inotify watch fd=123
[2019/05/29 02:27:54] [debug] [in_tail] scanning path /home/pi/logs/gpu.log
[2019/05/29 02:27:54] [debug] [in_tail] add to scan queue /home/pi/logs/gpu.log, offset=460833
[2019/05/29 02:27:55] [ info] [storage backlog] queue memory limit: 4.8M
[2019/05/29 02:27:55] [debug] [filter_modify] Initialized modify filter with 0 conditions and 6 rules
[2019/05/29 02:27:55] [debug] [filter_modify] Initialized modify filter with 0 conditions and 11 rules
[2019/05/29 02:27:55] [debug] [router] match rule tail.0:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.1:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.2:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.3:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.4:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.5:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.6:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.7:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.8:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.9:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.10:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.11:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.12:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule cpu.13:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.14:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule mem.15:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.16:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule tail.17:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule proc.18:forward.0
[2019/05/29 02:27:55] [debug] [router] match rule storage_backlog.19:forward.0
[2019/05/29 02:27:55] [ info] [http_server] listen iface=127.0.0.1 tcp_port=2020
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/fbit-metrics.log promote to TAIL_EVENT
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/pastri.log read=8102 lines=68
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/sfp.log read=7298 lines=101
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/power-scan.log promote to TAIL_EVENT
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/aeat6600.log read=894 lines=6
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/acm0.log read=4218 lines=25
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/acm1.log read=16425 lines=61
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/heartbeat.log read=106 lines=1
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/reboots.log promote to TAIL_EVENT
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/repo-status.log promote to TAIL_EVENT
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/wireguard-keepalive.log promote to TAIL_EVENT
[2019/05/29 02:27:55] [debug] [in_tail] file=/home/pi/logs/disk-usage.log promote to TAIL_EVENT
[2019/05/29 02:28:00] [debug] [storage] [cio scan] opening path /home/pi/config/fluent-bit/buffer/
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: fluent_bit
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: pastri
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: edfa
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: sfp
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: power_scan
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: encoder
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: beacon
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: imu
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: heartbeat
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: reboots
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: repo_status
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: wireguard
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: iperf
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: cpu_usage
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: disk_usage
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: memory_usage
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: net
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: gpu_info
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: pastri_ps
[2019/05/29 02:28:00] [debug] [storage] [cio stream] new stream registered: storage_backlog.19

Steps to reproduce the problem:

Expected behavior

Fluent bit should not crash.

Screenshots

Your Environment

Version used: v1.1.1
Configuration:

FLB_FLUSH_INT=5
FLB_PULL_INT=10
FLB_MEM_BUF_LIMIT=1M
FLB_BACKLOG_MEM_LIMIT=5M

# https://docs.fluentbit.io/manual/service
[SERVICE]
    Flush           ${FLB_FLUSH_INT}
    Daemon          Off
    Log_Level       info
    Log_File        ${FLB_LOG_FILE}
    Parsers_File    parsers.conf

    # https://docs.fluentbit.io/manual/configuration/buffering
    storage.path              ${FLB_CONF_DIR}/buffer/
    storage.sync              normal
    storage.checksum          on
    storage.backlog.mem_limit ${FLB_BACKLOG_MEM_LIMIT}

    # https://docs.fluentbit.io/manual/configuration/monitoring
    HTTP_Server     On
    HTTP_Listen     ${FLB_HTTP_HOST}
    HTTP_Port       ${FLB_HTTP_PORT}

@INCLUDE inputs.conf
@INCLUDE filters.conf
@INCLUDE outputs.conf

# parsers.conf
# Parse logs from files generated by easyloggingapp
[PARSER]
    Name         el_json
    Format       json
    Time_Key     datetime
    Time_Format  %Y-%m-%dT%H:%M:%S.%L
    Time_Keep    off

# inputs.conf
# Fluent Bit Monitoring
# https://docs.fluentbit.io/manual/configuration/monitoring
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           fluent_bit
    Tag             fluent-bit
    Path            ${LOGS_DIR}/fbit-metrics.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/fbit.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# Pastri
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           pastri
    Tag             pastri
    Path            ${LOGS_DIR}/pastri.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/pastri.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# EDFA
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           edfa
    Tag             edfa
    Path            ${LOGS_DIR}/edfa.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/edfa.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# SFP
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           sfp
    Tag             sfp
    Path            ${LOGS_DIR}/sfp.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/sfp.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# Power Scan
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           power_scan
    Tag             power-scan
    Path            ${LOGS_DIR}/power-scan.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/power_scan.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# Encoder (AEAT6600)
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           encoder
    Tag             encoder
    Path            ${LOGS_DIR}/aeat6600.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/encoder.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# ACM0
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           beacon
    Tag             beacon
    Path            ${LOGS_DIR}/acm0.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/beacon.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# ACM1
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           imu
    Tag             imu
    Path            ${LOGS_DIR}/acm1.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/imu.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# Heartbeat
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           heartbeat
    Tag             heartbeat
    Path            ${LOGS_DIR}/heartbeat.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/heartbeat.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# Reboots
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           reboots
    Tag             reboots
    Path            ${LOGS_DIR}/reboots.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/reboots.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# Repo Status
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           repo_status
    Tag             repo_status
    Path            ${LOGS_DIR}/repo-status.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/repo_status.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# WireGuard
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           wireguard
    Tag             wireguard
    Path            ${LOGS_DIR}/wireguard-keepalive.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/wireguard.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# iPerf
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           iperf
    Tag             iperf
    Path            ${LOGS_DIR}/iperf-client.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/iperf.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# CPU
# https://docs.fluentbit.io/manual/input/cpu
# =============================================================================
[INPUT]
    Name            cpu
    Alias           cpu_usage
    Tag             cpu
    Interval_Sec    ${FLB_PULL_INT}
    Interval_NSec   0

# Disk
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           disk_usage
    Tag             disk
    Path            ${LOGS_DIR}/disk-usage.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/du.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# Memory
# https://docs.fluentbit.io/manual/input/mem
# =============================================================================
[INPUT]
    Name            mem
    Alias           memory_usage
    Tag             memory
    Interval_Sec    ${FLB_PULL_INT}
    Interval_NSec   0

# Network (WLAN, Ethernet, Loopback, VPN)
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           net
    Tag             net
    Path            ${LOGS_DIR}/network.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/net.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# Raspi GPU
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           gpu_info
    Tag             gpu
    Path            ${LOGS_DIR}/gpu.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/gpu.db
    Mem_Buf_Limit   ${FLB_MEM_BUF_LIMIT}

# TODO: Figure out what other processes we need to keep an eye on
# Process
# NOTE: Group proc messages into 1 tag
# https://docs.fluentbit.io/manual/input/proc
# =============================================================================
[INPUT]
    Name            proc
    Alias           pastri_ps
    Tag             ps
    Proc_Name       pastri
    Interval_Sec    ${FLB_PULL_INT}
    Interval_NSec   0
    Fd              true
    Mem             true

# filters.conf
# Rename mem stats keys
[FILTER]
    Name    modify
    Alias   rename_mem
    Match   memory
    Rename  Mem.total mem_total
    Rename  Mem.used mem_used
    Rename  Mem.free mem_free
    Rename  Swap.total swap_total
    Rename  Swap.used swap_used
    Rename  Swap.free swap_free

# Add {hostname} to every record
# https://docs.fluentbit.io/manual/filter/record_modifier
[FILTER]
    Name    record_modifier
    Alias   add_hostname
    Match   *
    Record  hostname ${HOSTNAME}

# Rename process stats keys
[FILTER]
    Name    modify
    Alias   rename_ps
    Match   ps
    Rename  mem.VmPeak vm_peak
    Rename  mem.VmSize vm_size
    Rename  mem.VmLck vm_lck
    Rename  mem.VmHWM vm_hwm
    Rename  mem.VmRSS vm_rss
    Rename  mem.VmData vm_data
    Rename  mem.VmStk vm_stk
    Rename  mem.VmExe vm_exe
    Rename  mem.VmLib vm_lib
    Rename  mem.VmPTE vm_pte
    Rename  mem.VmSwap vm_swap

# outputs.conf
# Forward logs to Fluentd
# https://docs.fluentbit.io/manual/output/forward#secure_forward_setup
# =============================================================================
[OUTPUT]
    Name            forward
    Alias           forward
    Match           *
    Host            ${FLD_HOST}
    Port            ${FLD_PORT}
    Shared_Key      ${FLD_SHARED_KEY}
    Self_Hostname   ${HOSTNAME}.local
    tls             on
    tls.verify      on
    tls.ca_file     ${FLB_CERTS_DIR}/ca.cert.pem
    tls.crt_file    ${FLB_CERTS_DIR}/fbit.cert.pem
    tls.key_file    ${FLB_CERTS_DIR}/fbit.key.pem
    tls.key_passwd  ${FLB_CERT_PASS}
    Retry_Limit     False

# NOTE: Use for debug when FB debug mode is enabled (journalctl -u fluent-bit -f)
# Log to file
# https://docs.fluentbit.io/manual/output/file
# =============================================================================
# [OUTPUT]
#     Name    stdout
#     Alias   stdout
#     Match   *

Environment name and version (e.g. Kubernetes? What version?): Raspberry Pi 3
Server type and version:
Operating System and version:

Linux raspi 4.14.44-v7+ #1117 SMP Thu May 31 16:57:56 BST 2018 armv7l GNU/Linux

PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Filters and plugins:

Additional context

I'm building fluent-bit instead of using the td-agent-bit with the following script:

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
BIN_DIR=${DIR}/installer/bin

# Fluent Bit
FLB_VERSION=1.1.1
FLB_DOWNLOAD_DIR=/tmp/fluent-bit
FLB_ARCHIVE=fluent-bit-${FLB_VERSION}.tar.gz
FLB_ROOT_DIR=${FLB_DOWNLOAD_DIR}/fluent-bit-${FLB_VERSION}
FLB_BUILD_DIR=${FLB_ROOT_DIR}/build

# Get Fluent Bit
mkdir -p ${FLB_ROOT_DIR}
wget -P ${FLB_DOWNLOAD_DIR} https://fluentbit.io/releases/1.1/${FLB_ARCHIVE}
tar xzf ${FLB_DOWNLOAD_DIR}/${FLB_ARCHIVE} -C ${FLB_DOWNLOAD_DIR}
rm -rf ${FLB_DOWNLOAD_DIR}/${FLB_ARCHIVE}

# Build Fluent Bit
# https://docs.fluentbit.io/manual/installation/build_install
# https://docs.fluentbit.io/manual/installation/build_static_configuration
pushd .
cd ${FLB_BUILD_DIR}
# Note that the FLB_OUT_TD cannot be turned off as it will cause an error at compile time
cmake -DFLB_EXAMPLES=No \
  -DFLB_JEMALLOC=Yes \
  -DFLB_HTTP_SERVER=Yes \
  -DFLB_LUAJIT=No \
  -DFLB_METRICS=Yes \
  -DFLB_PROXY_GO=No \
  -DFLB_STREAM_PROCESSOR=No \
  -DFLB_FILTER_KUBERNETES=Off \
  -DFLB_FILTER_LUA=Off \
  -DFLB_IN_DUMMY=Off \
  -DFLB_IN_FORWARD=Off \
  -DFLB_IN_HEAD=Off \
  -DFLB_IN_KMSG=Off \
  -DFLB_IN_MQTT=Off \
  -DFLB_IN_RANDOM=Off \
  -DFLB_IN_SERIAL=Off \
  -DFLB_IN_SYSLOG=Off \
  -DFLB_IN_STDIN=Off \
  -DFLB_OUT_AZURE=Off \
  -DFLB_OUT_BIGQUERY=Off \
  -DFLB_OUT_COUNTER=Off \
  -DFLB_OUT_EXIT=Off \
  -DFLB_OUT_FLOWCOUNTER=Off \
  -DFLB_OUT_GELF=Off \
  -DFLB_OUT_HTTP=Off \
  -DFLB_OUT_INFLUXDB=Off \
  -DFLB_OUT_KAFKA=Off \
  -DFLB_OUT_KAFKA_REST=Off \
  -DFLB_OUT_NATS=Off \
  -DFLB_OUT_NULL=Off \
  -DFLB_OUT_PLOT=Off \
  -DFLB_OUT_SPLUNK=Off \
  -DFLB_OUT_STACKDRIVER=Off \
  ../

# NOTE: Using all cores (4) might crash the Raspi
make -j 3 # $(getconf _NPROCESSORS_ONLN)

# Copy binaries
cp -a ./bin/. ${BIN_DIR}/

# Verify binary
${BIN_DIR}/fluent-bit --version

popd

And here are the current log file sizes:

-rw-r--r--  1 pi   pi    24M May 29 02:40 acm0.log
-rw-r--r--  1 pi   pi   186M May 29 02:40 acm1.log
-rw-r--r--  1 pi   pi    11M May 29 02:41 aeat6600.log
-rw-r--r--  1 root root 8.0K May 29 02:40 disk-usage.log
-rw-r--r--  1 root root 634K May 29 02:40 fbit-metrics.log
-rw-r--r--  1 root root 5.7M May 29 02:40 fluent-bit.log
-rw-r--r--  1 root root 1.7M May 29 02:40 gpu.log
-rw-r--r--  1 root root 4.1M May 29 02:40 network.log
-rw-r--r--  1 pi   pi    86M May 29 02:41 pastri.log
-rw-r--r--  1 pi   pi      0 Apr 23 06:25 power-scan.log
-rw-r--r--  1 pi   pi    72M May 29 02:40 sfp.log

The text was updated successfully, but these errors were encountered:

rolandjitsu · 2019-05-29T03:11:15Z

I suspect this could happen due to one of the following reasons:

Too many log lines to read/parse, not enough memory (backlog buffer too small or input memory buffer to small)
We logrotate every night, so fluent-bit might have a reference to a line that doesn't exist anymore
Some line(s) cannot be parsed or it causes fluent-bit to crash when it tries to parse it

I do know that clearing all log files fixes the problem, but we cannot do that every time this happens.

UPDATE: I have tried to bump the backpressure to 200M and input buffer to 20M, but it doesn't seem to fix the problem.

rolandjitsu · 2019-05-29T04:16:10Z

I have found the issue, the following line:

{"event": "switchifa", "msg": "Switch to ethernet", "datetime": 2019-05-28T05:52:33.525872728+00:00}

The datetime key is missing the quotes.

Would it be possible to just have fluent-bit ignore this sort of lines instead of crashing without any indication why?

kiich · 2019-06-08T21:45:32Z

+1 to this as i've seen something similar with just the following logs:

Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: [2019/06/08 21:01:07] [ warn] unknown time format e99fb760
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: [engine] caught signal ([2019/06/08 21:01:07] [ warn] unknown time format e99fb760
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: SIGSEGV)
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: [2019/06/08 21:01:07] [ warn] unknown time format e99fb760
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: [2019/06/08 21:01:07] [ warn] unknown time format 65757274
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: #0  0x55ae76a4544a      in  splunk_format() at plugins/out_splunk/splunk.c:114
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: #1  0x55ae76a456d5      in  cb_splunk_flush() at plugins/out_splunk/splunk.c:175
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: #2  0x55ae769f7f4a      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:314
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: #3  0x55ae76d06786      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
Jun 08 21:01:07 ip-10-16-152-90.us-west-2.compute.internal docker[29395]: #4  0xffffffffffffffff  in  ???() at ???:0

not 100% if its related yet though but i am sending logs to splunk also and seen the below stack trace as well:

Jun 08 21:18:40 ip-10-16-152-90.us-west-2.compute.internal docker[3490]: [2019/06/08 21:18:40] [ warn] unknown time format bcf6bba0
Jun 08 21:18:40 ip-10-16-152-90.us-west-2.compute.internal docker[3490]: [engine] caught signal (SIGSEGV)
Jun 08 21:18:40 ip-10-16-152-90.us-west-2.compute.internal docker[3490]: [2019/06/08 21:18:40] [ warn] unknown time format bcf6bba0
Jun 08 21:18:40 ip-10-16-152-90.us-west-2.compute.internal docker[3490]: [2019/06/08 21:18:40] [ warn] unknown time format 65757274

also i've increased the storage.backlog.mem_limit from 32MB to 1024MB just to see if it does anything and the rate of the above SIGSEGV has actually gone down so that seems to have helped.

FYI, once i start seeing the above error, the only way i can recover was to rm -rf /var/log/flb-storage/ and then the restart starts to work - which as you know is not ideal!

edsiper · 2019-06-12T00:56:15Z

Crash on startup will happen when the storage file system path have more files than the fd system limit, I am working in a solution for the problem

edsiper · 2019-06-13T17:13:01Z

Would you please give it a try to the latest version on GIT Master ? I've pushed several improvements on storage startup and chunks handling.

rolandjitsu · 2019-06-16T06:34:36Z

Will try on Monday.

rolandjitsu · 2019-06-22T05:19:51Z

@edsiper tried it and I don't see any obvious issues. It will be difficult to test the scenario though, but I will keep an eye on it and if it occurs I will let you know.

erks · 2019-06-24T19:45:25Z

In our case, it seems to be an issue in one of our filters (possibly Lua) that sometimes ends up producing nil msgpack data like this (note that we send the log to two OUPTUTs, stdout and splunk for debugging):

[233259] kube.datadog-j9ts5_monitoring_datadog-0a8b400f17cf4db9571ac2e38e67b1b1254c4d66a37aaf3214acee00ba723e5f.log: [1561143872.747286319, nil]
[2019/06/22 03:01:16] [ warn] unknown time format 3f0c3e10
[233260] kube.datadog-j9ts5_monitoring_datadog-0a8b400f17cf4db9571ac2e38e67b1b1254c4d66a37aaf3214acee00ba723e5f.log: [1561143872.747286319, nil]
[2019/06/22 03:01:16] [ warn] unknown time format 3f0c3e10
[233261] kube.datadog-j9ts5_monitoring_datadog-0a8b400f17cf4db9571ac2e38e67b1b1254c4d66a37aaf3214acee00ba723e5f.log: [1561143872.747286319, nil]
[2019/06/22 03:01:16] [ warn] unknown time format 3f0c3e10

After a lot of those, fluent-bit (1.1.3-debug) then crashes

[engine] caught signal (SIGSEGV)
[2019/06/22 03:01:16] [ warn] unknown time format 3f0a59a0
#0  0x55a3375ca05a      in  msgpack_pack_object() at lib/msgpack-3.1.1/src/objectc.c:103
#1  0x55a3373186d2      in  splunk_format() at plugins/out_splunk/splunk.c:119
#2  0x55a3373188b3      in  cb_splunk_flush() at plugins/out_splunk/splunk.c:176
#3  0x55a3372c84ac      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:316
#4  0x55a3375e75a6      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#5  0xffffffffffffffff  in  ???() at ???:0

This might be specific to Splunk output, so let me know if you want me to create a new issue for this.

edsiper · 2019-06-24T20:37:40Z

please try to come up with a small reproducible case with Lua where it crashes Fluent Bit (we might need to add extra checks and enforce data types when using Lua scripts)

edsiper · 2019-07-09T17:41:49Z

ping

rolandjitsu · 2019-07-10T06:07:47Z

@edsiper I haven't seen the error anymore.

edsiper · 2019-07-16T10:00:55Z

new storage improvements fix the issue, closing.

enekofb · 2019-09-04T07:07:43Z

It also happens with splunk ... using fluentbit 1.13

  [2019/09/04 07:15:59] [ warn] unknown time format 73fa0920
  [engine] caught signal (SIGSEGV)
  #0  0x555d5e2ee05a      in  msgpack_pack_object() at lib/msgpack-3.1.1/src/objectc.c:103
  #1  0x555d5e03c6d2      in  splunk_format() at plugins/out_splunk/splunk.c:119
  #2  0x555d5e03c8b3      in  cb_splunk_flush() at plugins/out_splunk/splunk.c:176
  #3  0x555d5dfec4ac      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:316
  #4  0x555d5e30b5a6      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
  #5  0x7f03731def64      in  ???() at ???:0

debug logs does not show more than time format warn

[2019/09/04 07:25:54] [debug] [storage] forward.0:1-1567581954.49545733.flb mapped OK
[2019/09/04 07:25:54] [debug] [filter_nest] Lift : Outer map size is 5, will be 7, lifting 1 record(s)
[2019/09/04 07:25:54] [debug] [filter_modify] Input map size 7 elements, output map size 8 elements
[2019/09/04 07:25:54] [debug] [filter_modify] Input map size 8 elements, output map size 9 elements
[2019/09/04 07:25:54] [debug] [filter_modify] Input map size 8 elements, output map size 9 elements
[2019/09/04 07:25:54] [debug] [filter_modify] Input map size 9 elements, output map size 10 elements
[2019/09/04 07:25:54] [debug] [filter_modify] Input map size 9 elements, output map size 10 elements
[2019/09/04 07:25:54] [debug] [filter_nest] Nest : Outer map size is 10, will be 5, nested map size will be 6
[2019/09/04 07:25:54] [debug] [filter_nest] Nest : Outer map size is 10, will be 5, nested map size will be 6
[2019/09/04 07:25:54] [ warn] unknown time format 7abee7e0
[engine] caught signal (SIGSEGV)

enekofb · 2019-09-04T07:43:32Z

Same with fluent-bit 1.2.2

docker[12312]: [2019/09/04 07:41:56] [ warn] unknown time format a5604e20
docker[12312]: [2019/09/04 07:41:56] [ warn] unknown time format a5604e20
docker[12312]: [engine] caught signal ([2019/09/04 07:41:56] [ warn] unknown time format 2e303031
docker[12312]: SIGSEGV)
docker[12312]: #0  0x5639aae9b5eb      in  splunk_format() at plugins/out_splunk/splunk.c:115
docker[12312]: #1  0x5639aae9b877      in  cb_splunk_flush() at plugins/out_splunk/splunk.c:176
docker[12312]: #2  0x5639aae49cda      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:316
docker[12312]: #3  0x5639ab16c606      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
docker[12312]: #4  0x7f5600000011      in  ???() at ???:0

psychomelet · 2019-10-26T10:16:34Z

same with 1.3.2

Oct 26 18:13:01 cn-log.yogorobot.cloud fluentbit[29479]: [engine] caught signal ([2019/10/26 18:13:01] [ warn] unknown time format c088dd50
Oct 26 18:14:47 cn-log.yogorobot.cloud fluentbit[29788]: [engine] caught signal ([2019/10/26 18:14:47] [ warn] unknown time format a1083dd0

edsiper · 2019-10-27T13:38:09Z

Looks like a bad formed record is messing up things, any clue from where those records are coming from? If you disable the filters do u still see the issue?

khsahaji · 2020-05-05T09:32:52Z

How to determine which earliest version has this fix available ? I am guessing 1.2.1 or 1.2.2 but not sure.

rolandjitsu mentioned this issue May 29, 2019

fluent-bit Crash at startup #584

Closed

edsiper self-assigned this Jun 13, 2019

edsiper added bug fixed waiting-for-user Waiting for more information, tests or requested changes labels Jun 13, 2019

edsiper closed this as completed Jul 16, 2019

enekofb mentioned this issue Sep 4, 2019

fluent-bit crashes sending data to Kafka #1525

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash on startup #1362

Crash on startup #1362

rolandjitsu commented May 29, 2019 •

edited

Loading

rolandjitsu commented May 29, 2019 •

edited

Loading

rolandjitsu commented May 29, 2019

kiich commented Jun 8, 2019 •

edited

Loading

edsiper commented Jun 12, 2019

edsiper commented Jun 13, 2019

rolandjitsu commented Jun 16, 2019

rolandjitsu commented Jun 22, 2019

erks commented Jun 24, 2019

edsiper commented Jun 24, 2019

edsiper commented Jul 9, 2019

rolandjitsu commented Jul 10, 2019

edsiper commented Jul 16, 2019

enekofb commented Sep 4, 2019 •

edited

Loading

enekofb commented Sep 4, 2019

psychomelet commented Oct 26, 2019

edsiper commented Oct 27, 2019

khsahaji commented May 5, 2020

Crash on startup #1362

Crash on startup #1362

Comments

rolandjitsu commented May 29, 2019 • edited Loading

Bug Report

rolandjitsu commented May 29, 2019 • edited Loading

rolandjitsu commented May 29, 2019

kiich commented Jun 8, 2019 • edited Loading

edsiper commented Jun 12, 2019

edsiper commented Jun 13, 2019

rolandjitsu commented Jun 16, 2019

rolandjitsu commented Jun 22, 2019

erks commented Jun 24, 2019

edsiper commented Jun 24, 2019

edsiper commented Jul 9, 2019

rolandjitsu commented Jul 10, 2019

edsiper commented Jul 16, 2019

enekofb commented Sep 4, 2019 • edited Loading

enekofb commented Sep 4, 2019

psychomelet commented Oct 26, 2019

edsiper commented Oct 27, 2019

khsahaji commented May 5, 2020

rolandjitsu commented May 29, 2019 •

edited

Loading

rolandjitsu commented May 29, 2019 •

edited

Loading

kiich commented Jun 8, 2019 •

edited

Loading

enekofb commented Sep 4, 2019 •

edited

Loading