[prometheusremotewrite] - Prioritize "external_labels" on prometheusremotewrite exporer

[gillg]

Is your feature request related to a problem? Please describe.
Actualy the final label set pushed to prometheus is constructed with :

1. external_labels config
2. OTEL resources (marked as "usefull")
3. OTEL attributes

See :

opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter/helper.go

Line 115 in 140f00a

for key, value := range externalLabels {

Describe the solution you'd like
We should use this preceence instead to allow any override in external_labels I think...

1. OTEL resources (marked as "usefull")
2. OTEL attributes
3. external_labels config

Additional context
As an exemple I would override in my context the instance resource to put a hostname on it.
I will do it by a resource processor, but this order seems more smart anyway.

[gouthamve]

Hi @gillg making this change would make it incompatible with how Prometheus does it unfortunately. Prometheus prioritises metric labels over external_labels:

https://github.com/prometheus/prometheus/blob/57f4aab27d53872b109c8513846765df6b51eb57/storage/remote/queue_manager.go#L665

and

https://github.com/prometheus/prometheus/blob/57f4aab27d53872b109c8513846765df6b51eb57/storage/remote/queue_manager.go#L741-L742

I think using the resource processor is the right way to go for your usecase.