[exporter/datadog] Log exporter support

[lchayoun]

What are the plans for exporting logs to DataDog?

[ericmustin]

@lchayoun 👋 Hey, thanks for asking about this. Datadog 100% plans to support OpenTelemetry Logging, and in general all the telemetry from OpenTelemetry. Currently, OpenTelemetry Logs and the Log Data Model are marked as experimental according to the specification, https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md and, as far as I know, subject to many breaking changes and not suggested for production use. So at this time we don't yet have an Otel Logging exporter option in the datadogexporter, but it's on our roadmap and something we're keen to add as Logging in Otel matures.

In the mean time there are a variety of other oss log shippers we support (fluentbit, fluentd, logstash, vector, etc) or you can use the datadog-agent.
- https://docs.datadoghq.com/integrations/fluentbit/#overview
- https://docs.datadoghq.com/integrations/fluentd/?tab=host#overview
- https://docs.datadoghq.com/integrations/logstash/?tab=host#overview
- https://vector.dev/

(cc @tigrannajaryan @bogdandrutu If I've mispoken here about the suitability of otel logs for production uses or the possibility of breaking changes, please let me know. 🙇. we'd love to be able to add an exporter for logs as soon as it becomes realistic or logging approaches stability)

[tigrannajaryan]

@ericmustin correct, logs are not ready for production use. However we encourage people to try logs and give us feedback. Implementing a log exporter is a good way to validate if the data model works for DataDog and if not what you want to be different.

[lchayoun]

@ericmustin Thank you for the response!
The reason I proposed it is in order to start and test it at scale and provide feedback.
I'll be more than happy to try it even at an early stage.

We are currently working with the collector and DataDog exporter at large scale for metrics and traces.

[ericmustin]

@lchayoun Thanks for the additional context. That's super useful to know, i will update here when there's a timeline for this!

[adriangb]

Hi @ericmustin , could you update here when there's any progress made? I too would like to test whenever you have an early version out

[ericmustin]

@adriangb sure, we've synced on slack but just reposting here for anyone else, that we have nothing to update at this time and original answer still applies. I'm discussing internally on when this work might get accomplished but nothing to share publicly yet unfort

[alexvanboxel]

If someone is interested, we have a completely unsupproted branch that we use for pushing logs to datadog: https://github.com/collibra/opentelemetry-collector-contrib/tree/feature/datadoglog

Expect, force pushes on this branch, and as stated. no support

[ericmustin]

cool stuff @alexvanboxel , took a quick look i think this probably is something folks can use with some level of confidence, and covers a lot of the special cases/reserved attribs specific to datadog. But yes, to re-iterate, this isn't maintained or supported by datadog. Normally I'd encourage you to contribute upstream (and i could work with you on getting it merged) but i need to get some internal agreement on roadmap/maintenance/ownership of these components long term, especially bc the signal is still experimental, so i don't want you to spin your wheels unnecessarily. But again, awesome stuff!

[carlreid]

i need to get some internal agreement on roadmap/maintenance/ownership of these components long term

@ericmustin Were there any agreements/plans? Think it would be a shame not to build on top of the work already in place. Especially because then the community could at the least maintain the exporter on any changes that might happen from the experimental state

[ericmustin]

@carlreid generally agreed, but @olivielpeau or @mx-psi would probably be the right folks to sync with regarding roadmap and direction here, and not me, as I am no longer employed by datadog

[bertysentry]

Hey guys, now that Log data model is stable, will we get support for Logs in the Datadog exporter? Thank you!

[gfonseca-tc]

This is a very needed feature for the datadog exporter! Currently there is no way of sending logs to DD using only otel.

[carlreid]

@mx-psi Do you have any update to share regarding logging support from Datadog side; now that the data model is stable as earlier reported? I guess there hasn't been a meaningful update since July 2021. Also is there anything the community can do to help to get something working?

Also for anyone reading/subscribed, @mx-psi stated on related issue #9409:

could you '👍' that issue instead? That will help us track user interest.

So make sure to 👍 this issue if you're needing this. Though this issue is 2nd place on most thumbed up issues in this repo.

[mx-psi]

@mx-psi Do you have any update to share regarding logging support from Datadog side; now that the data model is stable as earlier reported? I guess there hasn't been a meaningful update since July 2021. Also is there anything the community can do to help to get something working?

Also for anyone reading/subscribed, @mx-psi stated on related issue #9409:

could you '👍' that issue instead? That will help us track user interest.

So make sure to 👍 this issue if you're needing this. Though this issue is 2nd place on most thumbed up issues in this repo.

@carlreid I relayed this internally to the logs team and can act as a liaison here. The team shared that Datadog is committed to supporting logs on the Datadog exporter and that while they can't provide a ship date, there will be an update for the community in the coming months.

As I said on the duplicate issue, if this feature would be useful for you, please upvote it, and if we are missing something or you think something about your use case provides important context, feel free to share it here.

[michalsteyn]

I would argue that this should be a top priority for DataDog. OpenTelemetry has undertaken a mammoth effort to consolidate observability across the industry.

Elastic already added support for OpenTelemetry Logging very early on. And APM isn't even their core business.

We have a code base that runs both in Cloud Native environments as well as air gapped self hosted environments. For our Cloud environments, DataDog is our preferred. But for air gapped environments where we have not connectivity to DataDog, we can only use Elastic APM.

OpenTelementry makes our code base so elegant and simple. We only need to support OpenTelementry, and from the OTEL Agent we can send our logs, traces and metrics wherever we want.

Currently we run a OTEL agent as well as a Vector Agent (only for logs) to decouple our code from the endpoints. Once DataDog finally adds logging support we will be able to do everything using only the OTEL agent.

I imagine there are numerous companies that seek this type of decoupled infrastructure.

[julealgon]

@michalsteyn can you elaborate how you setup Vector as an intermediate in your scenario?

I tried using DD agent for this purpose but the agent just pushes raw OTEL-formatted Json log batches as-is to Datadog which is not really what I wanted:

• How to properly parse OTEL-format logs collected by Datadog Agent? DataDog/datadog-agent#12701

Does your integration with Vector properly generate all individual log events in Datadog as if they were pushed directly by the application?

[michalsteyn]

@julealgon: Ahh, I see how my comment was confusing. Just to clarify:

• I'm basically using two agents running parallel (and not interconnected): OTEL for Tracing and Metrics, and Vector for Logging.
• I'm not using OTEL at all for logging, not from the client application side, nor from the Open Telemetry Receiver point of view.
• So Logging is totally running in a non-OTEL silo at the moment.

Obviously, this is not at all the type of architecture I want to have. I too want to have a single OTEL Collector deployed to my infra, and have my code decoupled from vendor specific libraries.

I have decided to live with this solution until DataDog properly supports OTEL Logging. At that point we will refactor our code to make use of OTEL Logging which will fully unify our solution.

If Vecor does end up supporting OTEL Logging, I would probably refactor our client code to switch to using OTEL Logging for exporting the logs, and will then use OTEL and Vector in series. That way I can at least decouple our code while we wait for DataDog.

Hope that adds some clarification.

[julealgon]

Hope that adds some clarification.

It absolutely does, thanks @michalsteyn .

I'm now in the process of seeing what options I have to make sure I don't need to setup DD directly on every application. Tried a few different approaches but without much success thus far... the log events that do arrive are really cumbersome to handle.

If I find a workable solution that gives me proper correlated entries in DD I'll update here with the details.

[adriangb]

FWIW, the way I set this up is:

• DataDog Agent as the only collector. Traces come via OLTP ingest (supported in the latest version), logs come via stdout/stderr scraping.
• I made a little (maybe like 100LOC) library (internal for now) that hooks into Python's logging API and emits structured logs that conform to the OTEL log data model and inject trace/span information into every log message emitted. It's similar to https://github.com/madzak/python-json-logger.
• I wrote a DataDog log processing pipeline to massage the OTEL log data model into something DataDog understands (converting timestamps from ns to ms, etc.). Without this, the backend would just drop the logs on the floor/fail to parse them.

Yes this is coupled to the DataDog agent, but I found that easier to set up/manage compared to the OTEL collector. It's also codified in a single IaC repo, so I can always swap it out with zero changes to individual applications.

I guess as far as I can tell there isn't much advantage to shipping logs over OLTP vs. just scraping them off of stdout/stderr, but maybe I'm missing something. The important thing to me is that the schema/data model be the OTEL model, because that bleeds into application logic and you need to have the same data model to search across services (ideally).

So what I would really like to see is better backend/frontend support from DataDog so that I don't have to do a bunch of custom stuff to make logs following the OTEL log data model usable. Ideally there would be a pre-populated log processing pipeline that detects that the logs are in the OTEL log data model schema and does any processing necessary to get trace correlation, aliases, etc. all working.

[julealgon]

I guess as far as I can tell there isn't much advantage to shipping logs over OLTP vs. just scraping them off of stdout/stderr

@adriangb unless you have an extremely simplistic deployment scenario, I don't see how that could be the case. When you have multiple service instances, some serverless, how would you even tap into their output streams? You'd both need an agent per application (dramatically ramping up maintenance costs and complexity) and a way to even be able to read the outputs from services like Azure Functions (which I don't believe you can normally?).

Do you mind if I ask how exactly you are fetching from the stdin/err? Do you have a single service that needs OTel integration?

[adriangb]

We don't do serverless, but we have multiple k8s clusters in multiple clouds with at least two dozen services; I wouldn't call it "simplistic". Each cluster has the DataDog agent helm chart installed: https://github.com/DataDog/helm-charts

[michalsteyn]

Hi @mx-psi, would it be possible to get a progress update on this? As documented the community is investing a lot of effort to make OTEL logging work with DD while we wait for the official support. It is risky for all of us to invest time into this, as DD might announce any day that it now supports OTEL Logging in the exporter.

And kind of roadmap update, even if it is merely a rough estimate would really help us be strategic on our end, and help us not to waste our time.

Given DD's tremendous growth and popularity, it still feels disappointing that this is taking so long.

Thanks.

[dineshg13]

Hi @michalsteyn , Thanks for following. Am from Datadog, and can I confirm that we started actively working on logs support. You can expect to hear from us soonish.

[carlreid]

For those following this, the latest release has support for Datadog logs. Thanks @dineshg13 👏

[julealgon]

For those following this, the latest release has support for Datadog logs. Thanks @dineshg13 👏

Awesome to hear that. We'll be using it right away.

@carlreid / @dineshg13 any particular reason this issue wasn't associated with the work and closed?

[mx-psi]

@julealgon We just forgot to close this, thanks for pointing it out, closing this now!