update defaults for the otel collector to reduce CPU/Memory

[strongjz]

The Otel collector defaults are cuasing high CPU and memory spikes for folks, we need to set the defaults to help reduce those.

#9016 (comment)

@sereinity I had the same issue. This was fixed when I set otel-schedule-delay-millis to something like 5000 and otel-max-export-batch-size to 512 in the ConfigMap. The defaults are causing every span to be sent to the collector without any batching/delay, which results in huge CPU usage.

[strongjz]

/triage accepted
/priority backlog
/good-first-issue
/help-wanted

[k8s-ci-robot]

@strongjz:
This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/triage accepted
/priority backlog
/good-first-issue
/help-wanted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chansuke]

/assign

[esigo]

@strongjz this is already fixed in #9978

[strongjz]

/close

[k8s-ci-robot]

@strongjz: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.