Filelog Receiver output is not as JSON format

[Prabakaranmkd]

I'm trying to export application log file to datadog using opentelemetery filelog receiver and oltphttp exporter

Steps to reproduce

receivers:
  filelog:
    include: [ /var/log/myservice/*.json ]
    operators:    
      - type: json_parser
        trace:
          trace_id:
            parse_from: trace
            preserve_to: trace
          span_id:
            parse_from: span
            preserve_to: span

exporters:
  otlphttp:
    logs_endpoint: https://http-intake.logs.us3.datadoghq.com/api/v2/logs
    headers:
      DD-API-KEY: xxxxxx
      Content-Type: application/json
    tls:
      insecure: true
  logging:
    loglevel: debug
    sampling_initial: 5
    sampling_thereafter: 200

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [logging,otlphttp]

What did you expect to see?

Expecting JSON format log like

{
  "timestamp": "",
  "attributes": {},
  "body": {
    "key1": "value1",
    "key2": {
      "nested_key1": "nested_value1",
      "nested_key2": "nested_value2"
    }
  }
}

What did you see instead?

getting log output as plain text

Timestamp: 2021-12-07 19:35:13.7383266 +0000 UTC
Severity:
ShortName:
Body: {
     -> @timestamp: STRING(2021-12-07T19:35:13.306Z)
     -> baggage: STRING()
     -> class: STRING(LogMonitor)
     -> dd.span_id: STRING(17263862035368123456)
     -> dd.trace_id: STRING(94514300960145454545)
     -> pid: STRING(33920)
     -> rest: STRING(LogMonitor)
     -> service: STRING(service-name)
     -> severity: STRING(DEBUG)
     -> span: STRING(dbf1beda33864533)
     -> thread: STRING(http-nio-8087-exec-9)
     -> trace: STRING(8628b320bfad7c02a3998b4343446f)
     -> xRequestId: STRING()
}
Attributes:
     -> file.name: STRING(logfile.json)
Trace ID: 8628b320bfad7c02a3998b4343446f
Span ID: dbf1beda33864533
Flags: 0

[jpkrohling]

getting log output as plain text

That's the logging exporter in action, not related to the Datadog exporter in any way. Or are you saying that you see this in Datadog?

[Prabakaranmkd]

Yes, each line exported to datadog as different log entry

[Prabakaranmkd]

I want to put them all as single log entry with JSON format

[jpkrohling]

I believe this might be some config problem on your side, but I'll tag someone from datadog to help you further. Perhaps @mx-psi? Or @KSerrania?

[mx-psi]

Hey there @Prabakaranmkd! We currently do not support sending OTLP logs directly to Datadog (neither on the http-intake.logs.us3.datadoghq.com/api/v2/logs endpoint nor in any other endpoint). To send logs to Datadog, they need to be in this documented API format.

#2651 tracks logs support for our Datadog exporter and there is a community implementation of the Datadog exporter in #2651 (comment). We are keeping track of demand for logs and stability of the OpenTelemetry logs signal more generally and may add logs support in the future to our exporter.

I am closing this since I think the questions are resolved, feel free to upvote #2651 and comment anything about your use case there.