kafka output plugin issue

[stand-for-hope]

Bug Report

Describe the bug

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}

• Steps to reproduce the problem:

Expected behavior

Screenshots

Your Environment

• Version used: fluentbit 0.13
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): kubernetes docker
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context
I have used the fluentbit 0.13 and nodejs to create a base image. In my own application dockerfile, i used that base image for "FROM" command, i create a shell script as the ENTRYPOINT for that dockerfile like this:
/fluent-bit/bin/fluent-bit -i exec -p 'command=npm run server' -o kafka -p brokers=**** -p topics=****
The problem is that:
After the application successfully deployed, application can run without any problems which means the "npm run server" script work properly but i can't see any loggers from our kafka.
But i try to go inside that running docker container and trigger fluentbit command manually then i can see the loggers go to kafka.
Can someone help give some advices on what the potential reason could be for this case?

[edsiper]

hi,

please provide the following:

• attach your dockerfile to this github issue
• if you do -o stdout instead of -o kafka do you see anything in docker logs ?
• please try with latest fluent/fluent-bit:0.14.6 image

[stand-for-hope]

hi,

please provide the following:

• attach your dockerfile to this github issue
• if you do -o stdout instead of -o kafka do you see anything in docker logs ?
• please try with latest fluent/fluent-bit:0.14.6 image

Hi Edsiper,
Thanks for your quick reply.
For #2 item, i have tried to use stdout, i can see logs on console
For #3 item, i try to use 0.14.6 to build base image
This is the sample error message i have from kubernetes:
$ kubectl logs {podname}
[2018/11/02 02:54:09] [ info] [engine] started (pid=1)
[2018/11/02 02:54:09] [debug] [in_exec] interval_sec=1 interval_nsec=0
[2018/11/02 02:54:09] [ info] [out_kafka] brokers='********' topics=''
[2018/11/02 02:54:09] [debug] [router] match rule exec.0:kafka.0
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 130
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 289
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 451
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 619
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 661
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 717
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 1436
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 2156
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 2921
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 3088
Fluent-Bit v0.13.8
Copyright (C) Treasure Data

[0] cathy-kafka: [1541127251.610221105, {"exec"=>"[Nest] 13 - 2018-11-2 02:54:11 [NestFactory] Starting Nest application..."}]
[0] cathy-kafka: [1541127251.673910264, {"exec"=>"[Nest] 13 - 2018-11-2 02:54:11 [InstanceLoader] AppModule dependencies initialized +63ms"}]
[0] cathy-kafka: [1541127251.674051651, {"exec"=>"[Nest] 13 - 2018-11-2 02:54:11 [InstanceLoader] TypeOrmModule dependencies initialized +1ms"}]
[0] cathy-kafka: [1541127251.674149722, {"exec"=>"[Nest] 13 - 2018-11-2 02:54:11 [InstanceLoader] AccessControlModule dependencies initialized +0ms"}]
[0] cathy-kafka: [1541127251.757934422, {"exec"=>"query: START TRANSACTION"}]
[0] cathy-kafka: [1541127251.772324745, {"exec"=>"query: SELECT DATABASE() AS db_name"}]
[0] cathy-kafka: [1541127251.787789127, {"exec"=>"query: SELECT * FROM ******"}]
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 4077
[2018/11/02 02:54:11] [debug] [input exec.0] [mem buf] size = 5292

Dockerfile:
FROM {base image contains both fluentbit and nodejs}

WORKDIR /dev-business-server

COPY package*.json ./
RUN npm install --production

COPY src/ /dev-business-server/src
COPY tsconfig.json /dev-business-server/tsconfig.json
RUN npm run build

COPY fluentbit.kafka.conf /dev-business-server
COPY fluent-bit-parsers.conf /dev-business-server
EXPOSE 7000
CMD [ "/fluent-bit/bin/fluent-bit", "-c", "fluentbit.kafka.conf"]

Fluentbit.kafka.conf
[FILTER]
Name stdout
Match cathy-*
[SERVICE]
Flush 5
Daemon Off
Log_Level error
Parsers_File fluent-bit-parsers.conf
[INPUT]
Name exec
Tag cathy-kafka
Command node dist/main.js
Interval_Sec 1
Interval_NSec 0
[OUTPUT]
Name kafka
Match *
Brokers {domain}:{port}
Topics {topicName}
Retry_Limit 2
rdkafka.log.connection.close false
rdkafka.queue.buffering.max.ms 1000
rdkafka.queue.buffering.max.messages 10000
rdkafka.batch.num.messages 2000
rdkafka.compression.codec none
rdkafka.request.required.acks 1

Another wired thing is that, if i change to use "i -cpu " and keep the rest out put config same, then i can see logs in our kafka server. So from my observation, maybe i miss some necessary configuration for "exec" input plugin?
Can you please help point to the right direction?

[edsiper]

In order to speed up the review of this issue please do:

• attach (not copy paste) your Docker file to this ticket.
• attach your Fluent Bit configuration file

Also make sure that your configuration file have the order of sections as: SERVICE, INPUT, FILTER, OUTPUT

[edsiper]

ping

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.