Target Allocator loses targets discovered when collector instances take time to come up

[rashmichandrashekar]

Component(s)

target allocator

What happened?

Description

In a kubernetes cluster, have scrape jobs configured using TargetAllocator. Due to timing issue(which is very common with kubernetes clusters, where no ordering can be guaranteed), the instances are not discovered until after the first set of targets are discovered. The configuration applied until the instances are discovered are lost.

Steps to Reproduce

Delay the collector instance creation (hard to reproduce since it happens due to timing issue)

Expected Result

The targets discovered should be retained so that they can be assigned whenever the collector instances come up

Actual Result

The targets discovered before the collector instances come up are lost.

Kubernetes Version

1.25.11

Operator version

v0.85.0

Collector version

v0.85.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

No collector instances present, cannot set targets

Additional context

Have created a PR to fix this - #2351

[jaronoff97]

closed by #2351