Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/prometheus] syncTargetAllocator now detects regex changes in relabel config #32127

Merged
merged 73 commits into from
Jul 11, 2024

Conversation

rashmichandrashekar
Copy link
Contributor

Description:
Fixing a bug - With the Targetallocator component in prometheus receiver, when prometheus scrape config is updated for metric relabel config with just regex change, the prometheus metrics receiver doesn't update the hash and hence doesn't pick up the metrics relabel config with the new regex.
This is because the regex struct has unexported fields. This fix similar to the fix made in opentelemetry-operator fixes this issue by taking into account the unexported fields as well.

Link to tracking Issue: - #29313

Testing: Tested to make sure that just the regex changes in metric relabeling gets picked up with TargetAllocator enabled.

Reopening this PR since the old one(#30258) got closed due to inactivity.

@swiatekm
Copy link
Contributor

@rashmichandrashekar yeah, in my opinion we should do something very similar to what the target allocator does and just serialize the configs to yaml, and hash the output. They should be sorted, to avoid map ordering inconsistencies.

Copy link
Contributor

@swiatekm swiatekm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach looks good to me. I left some comments about testing.

@@ -145,20 +147,47 @@ func (r *pReceiver) startTargetAllocator(allocConf *TargetAllocator, baseCfg *Pr
return nil
}

// Calculate a hash for a scrape config map.
// This is done by marshaling to YAML because it's the most straightforward and doesn't run into problems with unexported fields.
func getScrapeConfigHash(jobToScrapeConfig map[string]*config.ScrapeConfig) (hash.Hash64, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add some simple unit tests for this function? There was concern about map iteration order inconsistency, so we should at least check if this function ran on two copies of the same config gives the same result.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @swiatekm. Added one. Please take a look.

@@ -532,6 +644,17 @@ func TestTargetAllocatorJobRetrieval(t *testing.T) {
// which is identical to the source url
s.Labels["__meta_url"] = model.LabelValue(sdConfig.URL)
require.Equal(t, s.Labels, group.Labels)
if s.MetricRelabelConfig != nil {
// Adding wait here so that the latest scrape config is applied with the updated regex
time.Sleep(5 * time.Second)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do need to wait here, could we use assert.Eventually instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@swiatekm - This is not needed. Removed it.

Copy link
Contributor

@swiatekm swiatekm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rashmichandrashekar
Copy link
Contributor Author

LGTM

Thanks @swiatekm for the quick review and approval. @dashpole - Could you pls help get this merged?

@rashmichandrashekar
Copy link
Contributor Author

rashmichandrashekar commented Jul 10, 2024

LGTM

Thanks @swiatekm for the quick review and approval. @dashpole - Could you pls help get this merged?

@dashpole - Following up, could you pls help get this merged as this has been waiting for several months?

@dashpole
Copy link
Contributor

Ack. I'm OK with the general approach now. I'll try to review soon

@dashpole dashpole added the ready to merge Code review completed; ready to merge by maintainers label Jul 10, 2024
@jpkrohling jpkrohling changed the title Fix for - syncTargetAllocator in prometheus metrics receiver doesnt detect regex changes in metrics relabel config [receiver/prometheus] syncTargetAllocator now detects regex changes in relabel config Jul 11, 2024
@jpkrohling jpkrohling merged commit 42c03a2 into open-telemetry:main Jul 11, 2024
160 of 161 checks passed
@github-actions github-actions bot added this to the next release milestone Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants