Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some data couldn't be associated with k8s metadata when the agent was just started. #32556

Closed
h0cheung opened this issue Apr 19, 2024 · 6 comments · Fixed by #32622
Closed

Some data couldn't be associated with k8s metadata when the agent was just started. #32556

h0cheung opened this issue Apr 19, 2024 · 6 comments · Fixed by #32622
Assignees
Labels
bug Something isn't working processor/k8sattributes k8s Attributes processor

Comments

@h0cheung
Copy link
Contributor

Component(s)

processor/k8sattributes

What happened?

Description

When the agent is first started, the k8s metadata has not been fully synchronized by the k8sattributes processor. If any data passes through the component at this point, they will not be associated with metadata.

Steps to Reproduce

  • start a service that continuously sends a large amount of data to the agent.
  • (re)start the agent.

Expected Result

All data should be associated with k8s metadata.

Actual Result

There may be a little data that hasn't be assosicated.

Collector version

v0.95.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

I think it's the best solution to wait for an initial sync when starting k8sattributes processor.

Or we can block data util an initial sync is finished.

@h0cheung h0cheung added bug Something isn't working needs triage New item requiring triage labels Apr 19, 2024
@github-actions github-actions bot added the processor/k8sattributes k8s Attributes processor label Apr 19, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@TylerHelmuth TylerHelmuth removed the needs triage New item requiring triage label Apr 22, 2024
@TylerHelmuth
Copy link
Member

I wonder if we could add something to the processor's start function that waits for the kube client connections to be made.

@h0cheung
Copy link
Contributor Author

I wonder if we could add something to the processor's start function that waits for the kube client connections to be made.

I think it's not enough to only wait for connections. If the connections are just made but metadata hasn't been synced yet, data won't be associated still. We need to wait for an initial complete sync like the implementation in #32622.

Copy link
Contributor

github-actions bot commented Aug 8, 2024

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Aug 8, 2024
Copy link
Contributor

github-actions bot commented Oct 7, 2024

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 7, 2024
@orloffv
Copy link

orloffv commented Oct 7, 2024

Oh noo

@TylerHelmuth TylerHelmuth reopened this Oct 8, 2024
RutvikS-crest pushed a commit to RutvikS-crest/opentelemetry-collector-contrib that referenced this issue Dec 9, 2024
…ibutes processor. (open-telemetry#32622)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

When starting `k8sattributes` processor, block until an initial
synchronization has been completed. This solves open-telemetry#32556

**Link to tracking Issue:** <Issue number if applicable>

fix open-telemetry#32556

**Testing:** <Describe what testing was performed and which tests were
added.>

Tested in a cluster with constant high span traffic, no more spans with
unassociated k8s metadata after adding this pr.

**Documentation:** <Describe the documentation added.>

---------

Co-authored-by: Christos Markou <[email protected]>
sbylica-splunk pushed a commit to sbylica-splunk/opentelemetry-collector-contrib that referenced this issue Dec 17, 2024
…ibutes processor. (open-telemetry#32622)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

When starting `k8sattributes` processor, block until an initial
synchronization has been completed. This solves open-telemetry#32556

**Link to tracking Issue:** <Issue number if applicable>

fix open-telemetry#32556

**Testing:** <Describe what testing was performed and which tests were
added.>

Tested in a cluster with constant high span traffic, no more spans with
unassociated k8s metadata after adding this pr.

**Documentation:** <Describe the documentation added.>

---------

Co-authored-by: Christos Markou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working processor/k8sattributes k8s Attributes processor
Projects
None yet
4 participants