[exporter/datadog] add example k8s configs

[ericmustin]

Description:

Adds an example k8s manifest for deploying the collector as agent and as collector in a k8s environment, with datadog specific processors and exporters enabled.

Link to tracking Issue:

Testing:

Documentation:

[ericmustin]

would like to have this reviewed by colleagues at datadog before merging, thanks! cc @mx-psi

[codecov]

Codecov Report

Merging #2193 (20287e6) into master (d7ed62f) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2193      +/-   ##
==========================================
- Coverage   90.48%   90.46%   -0.02%     
==========================================
  Files         397      397              
  Lines       19593    19593              
==========================================
- Hits        17728    17725       -3     
- Misses       1403     1406       +3     
  Partials      462      462              

Flag	Coverage Δ
integration	69.41% <ø> (ø)
unit	89.26% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
receiver/prometheusexecreceiver/receiver.go	85.83% <0.00%> (-2.50%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d7ed62f...20287e6. Read the comment docs.

[mx-psi]

Something is missing in this sentence, right?

[ericmustin]

ah yup, updated.

[jpkrohling]

Do you mind if I ask why this component must have a Kubernetes example? We have Kubernetes examples in the main repository, and no individual component has one, to my knowledge. Nothing really against it, just trying to understand if there's something special about this component.

[ericmustin]

@jpkrohling no worries at all, don't mind you asking. So, we've been working on onboarding documentation for datadog's docs (this is public, feel free to take a look: DataDog/documentation#9529) , and noticed that a lot of users of otel at datadog are new to both otel and datadog at the same time, and so documenting things in a way that is just an incremental "add these exporters or processors to an existing setup" is confusing because there is no existing setup.

Additionally, the current k8s docs we've found to be not a wonderful fit for capturing metadata that is especially important for a vendor like ddog that is doing host count-based billing. Specifically there's no node or pod level information captured here via a k8s_tagger processor: https://github.com/open-telemetry/opentelemetry-collector/blob/master/examples/k8s/otel-config.yaml . We additionally have some preferences around the batch processor that may not be something everyone would want setup globally.

So, all those things combined, it's just far easier and simpler to have a single doc that helps users onboard to datadog and OpenTelemetry simultaneously. And we'd prefer that doc not be in-line markdown within our docs site, but hosted in the otel-collector-contrib repo so folks can find it more easily if they're coming from within the contrib repo instead of via our docs site.

Additionally, when we have our vendor specific docs in a good place and verified with users, we'd like to contribute back upstream to the main docs repo for otel. we've found that the docker examples and k8s examples there have room for improvement. Once those are in a better place we'll try to migrate to using those, but in the meantime we'd prefer to have strong examples within the datadogexporter repo.

For context, we've found there are a number of other vendors that are taking a similar approach to what we're doing, but choosing to keep their examples and docs proprietary or hosted largely within their own repos. We feel the approach we're taking is more community friendly.

Let me know if that sounds reasonable or if there's anything we can provide here to improve the comfort level around this.

[jpkrohling]

Thanks for the explanation, sounds good to me.

we've found that the docker examples and k8s examples there have room for improvement

+1, I have a blog post draft ready to be submitted on this topic, as I don't think the examples are satisfactory.

[mx-psi]

LGTM, we should open the issues needed for these docs to be updated in the k8s_tagger and in the OpenTelemetry docs.