Consider changing batch span processor defaults

[alanwest]

Related #2954

Related discussions in the 12/20/2022 spec meeting (video link) and 1/4/2023 log meeting (video link).

The default scheduledDelayMillis for the batch span processor has come under question in the context of defining the default for the equivalent setting of the batch log processor (#3002 (comment)).

The purpose of this issue is to determine if we are comfortable revisiting the default value for the batch span processor.

Below is a summary of discussion points raised from SIG meetings.

5 seconds is too long

The current default scheduledDelayMillis is 5000 milliseconds. There seems to be a general sense that this is too lengthy for both log and span data. This is especially true in "live-tailing" use cases.

What should the default be?

A default value in the range of 200ms and 1s has been proposed as a more reasonable default.

Some considerations in choosing a default:

• Install experience. It may seem odd to end users if default values differ between spans and logs. If they do differ, we should at least have some rationale for the differences.
• Human perception. Users may find the system sluggish or non-responsive if the delay chosen is too long.
• Not wreck havoc when deployed in a production environment.

The collector should influence the defaults we choose

• The equivalent default for the collector's batch processor is 200ms. To some, it seems unintuitive that the collector's default would be shorter than the SDK's default i.e., seems that, if anything, it should be the other way around.
• Defaults configuration values of other SDK components have been influenced by assuming data will be exported to a locally running OpenTelemetry Collector (e.g., the default endpoint of the OTLP exporter is localhost).

Is this a breaking change?

It has been proposed that the current 5 second default for the batch span processor should be considered a specification bug. So, while this may be considered a breaking change, we should correct our mistake.

[reyang]

The SDK can be used to send data to a nearby place (e.g. a local collector) or faraway place (e.g. an IoT device sending data to a satellite).

5 seconds delay was chosen because:

1. it is not going to blow the production system away due to the too frequent exports
2. it is still responsive enough, so a user won't be frustrated (e.g. if the default delay is 5 minutes, a new user might give up after 30 seconds assuming that things were wrong, but 5 seconds is good enough)

The current default scheduledDelayMillis is 5000 milliseconds. There seems to be a general sense that this is too lengthy for both log and span data. This is especially true in "live-tailing" use cases.

200ms (or anything less than 300ms) seems to be very interactive (e.g. if the telemetry is associated with some UI activity - a user clicks a button from the UI and see the telemetry behavior change "immediately"), and I don't think the defaults should optimize for these scenarios.