Memory improvements first pass #1293
Conversation
| } | ||
|
|
||
| return tgs | ||
| func GetAllTargetsByCollectorAndJob(allocator Allocator, collector string, job string) []*target.Item { |
There was a problem hiding this comment.
in a future PR i'm going to remove this file altogether, for now i'm trying to keep changes to a minimum
| jsonConfig, err := yaml.YAMLToJSON(configBytes) | ||
| if err != nil { | ||
| s.errorHandler(w, err) | ||
| // if the hashes are different, we need to recompute the scrape config |
There was a problem hiding this comment.
this logic is similar to what the collector is doing on the receiving end i.e. it should only update this byte array when there's an updated scrape config list
| } | ||
|
|
||
| func (t Item) Hash() string { | ||
| return t.JobName + t.TargetURL + t.Label.Fingerprint().String() | ||
| return t.hash |
There was a problem hiding this comment.
by precomputing the hash, we're saving a lot of allocations
|
@secustor i've tested this in cluster and it has been running successfully for over a week now. I did some benchmarking before and after for the number of allocations made and the reduction was over 99.9%. I haven't tested this with a cluster at scale, but I can do that before we release this if that would give us more confidence? |
…7/opentelemetry-operator into 1257-memory-improvements-first
| additions := map[string]T{} | ||
| removals := map[string]T{} | ||
| // Used as a set to check for removed items | ||
| newMembership := map[string]bool{} |
* Memory improvements first pass * Comments, store hash * Fix linting and tests * Update, more tests and benchmarks, notes * linting
* Memory improvements first pass * Comments, store hash * Fix linting and tests * Update, more tests and benchmarks, notes * linting
Closes #1257
This PR is the first in a series of two. In this PR, we add new fields to the allocators and servers to cache the values they need to return to prevent excessive allocations on http calls. This also removes the intermediary of the targetGroupJSON and instead just uses the target item with json hints. This also removes the allocations needed in the map diffing. This also fixes a potential bug in the map which would have caused targets to not update when processed. Finally, this has the server keep track of when it needs to recompute the scrape configs response it has to return.
NOTES!