web: initial Datadog RUM integration

[valerybugakov]

Context

This PR integrates Datadog RUM SDK into the web application.

Closes https://github.com/sourcegraph/sourcegraph/issues/33992
Related to: https://github.com/sourcegraph/sourcegraph/issues/29651 and RFC 575 APPROVED: Cloud Observability tooling

Changes

• Datadog RUM SDK is initialized via async script along with Sentry SDK.
• Datadog RUM configuration is added to the site-config under the observability.logging key.
• @datadog/browser-rum-slim is added to devDependencies to allow usage of Typescript types, and it's not added to the application bundle.

Notes integration trade-offs

Since we didn't decide which solution we would want to use for the observability OKR set for the Frontend Platform team next quarter, it's essential to integrate the Datadog RUM SDK as a POC without hurting our performance metrics on production.

• The bundle size of the default Datadog browser package is pretty big.
• They have a slim version that doesn't include the SessionReplay feature, but it's still significant.
• The SDK doesn't provide a lazy-loading out of the box without the possibility of missing some events.

The best strategy for POC is to load it via async script to avoid hurting our bundle size. This way, we keep Sentry in place and see how they compare.

Production

Merging this PR won't enable Datadog RUM on any production instance automatically. To enable it, the Datadog RUM configuration should be added to the site configuration via site-admin.

Test plan

Local development

Datadog is initialized only if:

1. The SDK script is included in the index.html template (app.html). Meaning that SourcegraphDotComMode == true.
2. Datadog RUM is configured using Sourcegraph site configuration.
3. process.env.ENABLE_MONITORING || process.env.NODE_ENV === 'production' to prevent log spam in the development environment.

Based on the requirements above to test Datadog RUM locally run: ENABLE_MONITORING=true sg start dotcom

Datadog access

Datadog RUM application logs are available here. If the link is inaccessible for you, check if Datadog is added to your Okta profile. If it's not there — reach out to #it-tech-ops

Testing

1. Run ENABLE_MONITORING=true sg start dotcom
2. Add Dagadog configuration to the site config.
3. Check out the network tab:
   • The Datadog SDK should be loaded.
   • Pageviews should be logged to Datadog servers.

[jasongornall]

loving these comments!