Support k8s log file in syslog exporter #30746
Labels
Comments
oppokui
added
enhancement
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Hello @oppokui, since there are a few moving parts here it would be good to try to narrow down which component isn't working as expected. It sounds like you're not receiving logs at your syslog server, is that the issue you're trying to resolve? Is the collector logging any relevant messages about the syslog exporter? Can you please set up your collector using the debug exporter instead of the syslog exporter, and confirm you're seeing the logs you expect in the output? This will help us determine if there's a problem with your receiver configuration, or something wrong with the exporter configuration. |
|
In addition to @crobert-1's suggestion, it may be helpful to try sending the logs back to a syslog receiver on the same collector. This loopback test could remove your backend from the equation and demonstrate whether the exporter is working correctly. receivers:
filelog: ... # reads original logs
syslog: ... # reads from syslog exporter
exporters:
syslog: ... # sends to syslog receiver
debug: ... # writes final result
service:
logs/in:
receivers: [ filelog ]
exporters: [ syslog ]
logs/out:
receivers: [ syslog ]
exporters: [ debug ] |
|
Thanks, @crobert-1 and @djaglowski. I am pretty sure filelog generate events, just no data coming out of syslog. I do try the debug and see event printed in the following way: The detail log printed like this: The log format is not syslog format, I guess it is the reason why syslog exporter didn't generate output. Are you sure it should work? When I saw the example of syslog exporter of github, the file from filelog was with syslog format and use syslog parser in operators. I didn't find any other example from google. If you have, please let me know. |
|
@djaglowski , I tried your config, but failed. I configure as follows: I open port 5140 in collector daemonset yaml. |
The collector has an internal representation of logs. Receivers convert other formats into the internal representation, and exporters convert from the internal representation into other formats. So you should be able to use the syslog exporter with any logs that are successfully received by the collector. |
|
@djaglowski , thanks for the confirmation. I didn't see any log printed for syslog exporter. The log only print one .log file from filelog receiver, then stuck there. I assume it should find all k8s logs in current EC2 node, but it didn't. By the way, in original example of syslog https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/syslogexporter/examples/config_with_filelog_receiver.yaml, I saw extension "file_storage/syslog", I comment it out because it didn't work in my env. It is the possible reason that syslog didn't have tmp folder to generate output? The file_storage extension need a hostpath, but it always reports permission issue. How to resolve it? |
|
@djaglowski , file_storage extension is not a problem now. I configure initContainer to create it. But still no syslog event sent out. Here is the complete deployment yaml. The config file: The daemonset of OTLP collector: When it is deployed, the log from debug exporter is printed: In target machine, I watch 5140 syslog server port, no data printed: But if I manually send UDP data in the same machine of OTLP collector, the message will be printed in target machine. The message: It means if syslog exporter send the UDP package out, the target should receive it. How to print more logs of syslog exporter? Don't know the internal logic. Any idea how to debug it? |
|
@sumo-drosiek can you take a look at this issue? It is a bug of syslog exporter? I am using helm chart 0.77.0 which includes docker image otel/opentelemetry-collector-contrib:0.89.0. I try 0.93.0 or dev build, none of them work. |
|
@oppokui AFAIR Syslog exporter requires some data preparation. The following attributes should be set, so data can be properly serialized: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/syslogexporter#rfc5424 Could you use |
|
Also I would check metrics: |
|
@sumo-drosiek , great! it is the one exact problem I meet. When I use source code to debug the same case, I can see the empty sylsog event in tcp mode: In this case, I read /var/log/syslog file which should be standard syslog format, but it can't parse the file. The error log: After I comment parser from operators, the error log disappear. But the syslog event body is empty.
|
|
@oppokui Could you try the following parser? I believe Regarding |
|
@sumo-drosiek , I tried rfc5424, no help. For the attribute enrichment, I saw filelog already configured some operators. So I just add appname/hostname/message, as follows. But for "timeUnixNano", do you know how to enrich it from timestamp string format? |
|
You probably looking for time parser operator or combination of regex_parser and time_parser You can then use transformprocessor to move it from log property o log attribute. Is this what you are looking for? |
|
I don't know "timeUnixNano" is required or not as filelog can identify the timestamp from the k8s log file. I assume syslog can identify the format too. No need to construct it manually, right? I add 2 processors, k8sattribute and transform. The transform will enrich appname, hostname and message, not sure k8s.pod.name/k8s.node.name are in the attributes after k8s attribute processor. Do you know it? After it, I still didn't see a complete syslog event send out. |
What is missing? Most of the fields are optional, so the question if there are required by you/syslog server or rfc? If you want to send additional attributes (no specified explicitly in rfc) you need add them to Could you share, example input, actual output of it and expected output? It would be easier for me to understand the context. Ideally outputs should be both from syslog server and debug(logging) exporter |
|
I am not sure which part is missing. Originally I assume the format is not complete, but as you said, event will be still sent out even attribute is missing. The end-2-end data as follows:
When I sent UDP package in the same machine of OTLP collector, I can see the data in target machine. In my local env, I can modify the .go code and print log to indicate whether the connect() and sender.write() are executed or not, but I fail to deploy it to EKS env. So don't know whether the UDP connect/send work or not. Any idea how to diagnose it? |
|
@sumo-drosiek , I checkout collector-contrib code in my EC2 machine, and modify exporter.go to print more logs, then create same pipeline with my description above, from k8s logs to syslog exporter. My code change is: The log message only print "[Kui] format data" logs before connect() line. There are lot of new lines in k8s pods, but no messages in collector log, it sounds hang there. As I said, I can send TCP/UDP package through nc command, to target machine. In target machine, I can see the UDP package: Do you think any reason syslog exporter hang in connect()? |
|
@sumo-drosiek , I assume it is a bug. I didn't configure TLS, but the code still get tlsConfig then hang there. My config file: My code change like this: The log is: (it goes to tlsConfig path and hang there) After I comment out tlsConfig path and retry, the TCP/UDP package can be sent tout. I can see the empty syslog event in target machine now. Then log look like: @sumo-drosiek , can you help to fix it in next release? Now the left issue for me is the syslog event mapping. I still get the empty body. |
Yes, log tismtamp is used to built syslog message
Oh, I forget about that. In order to disable tls, the following configuration has to be used:
I recommend to change |
|
woooooo, it definitely is a bug. I didn't set tls, or set tls as empty map, it should mean insecure. Hang there without any warning is definitely wrong. It blocked me two weeks. Now I can see syslog format in target machine, with timestamp, appname and message. Just no hostname. |
|
@sumo-drosiek I enable detailed verbosity in debug exporter, now I can see the structure of input event to syslog exporter. |
Yes, please use the following: |
|
Well done! For structure data, how can I enrich multi sub-nodes into attributes.structured_data? |
The following should do the job: |
|
Thanks very much, @sumo-drosiek ! You save my life! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
exporter/syslog
Is your feature request related to a problem? Please describe.
We want to collect k8s logs through openTelemetry Collector, and then analyze the logs outside. The initial idea is to setup syslog server and forward logs to HDFS cluster. The openTelemetry Collector helps to forward k8s logs to syslog server.
I create a pipeline with filelog receiver and syslog exporter. But failed to receive any UDP package in external syslog server. It sounds the k8s log is not syslog format then syslog exporter didn't do anything.
I didn't find any syslog processor or formatter in pipeline techs.
Any suggestion?
Describe the solution you'd like
Is it possible to create sylog processor like this;
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: