otel: make sure everything existing actually WAI

[deniseli]

Code Links:

• otelInterceptor: internal/rpc/context.go#L83
• otel client: internal/observability/client.go#L36-L75
• templates/ftl-controller.yml#L50 - Exporter is how we send data to otel (docs)

Out of scope: this logging interface we provide through the Go runtime SDK that no one currently uses: go-runtime/ftl/observability/metrics.go

Datadog:

• go/tbddd - all TBD services
• FTL Dashboard
• Controller logs
• Example of a call

[deniseli]

Also part of this: make sure metrics + logs work as intended. Add a toy one, deploy to the dev instance, make some verb calls in the dev instance, and then check datadog to make sure it all pipes all the way through. That unblocks us adding metrics/logs/traces in general.

[deniseli]

Metrics are looking pretty solid now: https://app.datadoghq.com/metric/explorer?fromUser=false&graph_layout=split&start=1722446967244&end=1722533367244&paused=false#N4Ig7glgJg5gpgFxALlAGwIYE8D2BXJVEADxQEYAaELcqyKBAC1pEbghkcLIF8qo4AMwgA7CAgg4RKUAiwAHOChASAtnADOcAE4RNIKtrgBHPJoQaUAbVBGN8qVoD6gnNtUZCKiOq279VKY6epbINiAiGOrKQdpYZAYgUJ4YThr42gDGSsgg6gi6mZaBZnHKGniqyIIIaAB0GBpYIplOmRho9aoaTgg4bTiq8miIcMAAVDwNPZn4IggAFACUIDwAulSu7niYoeFbqjsYMaXxqxsgGnIjMiDyHYgIOUk4MG27GhCZiWiicL0KZ7pX5QH5-Jz0JjKERuDxoVb8CD2TBYAGKZS-ERKdY8PiXYbiADCUmEMFuyPETigPjgIk+jmsoCk-xg2gw8kYTkU2mUNXqjWarXanTqbg4onOVF+qnEKAATAAWKjpbSEJnaAQ83ICDTfXE8IA

[deniseli]

Marking this as done now that the correctly parented traces are in prod and the new metrics+attrs are continuing to function as expected