metric adjusters in prometheus receiver should have a better fallback strategy.

[h0cheung]

Component(s)

receiver/prometheus

Is your feature request related to a problem? Please describe.

There are two metric adjusters in the Prometheus receiver:

• initialPointAdjuster, used by default. It will use the timestamp of the first sample as start_timestamp. There is a function gate receiver.prometheusreceiver.UseCreatedMetric, which, if enabled, will use *_created so that the timestamp of the first sample will be used as a backup.
• startTimeMetricAdjuster, used when use_start_time_metric is set to true. It will take a metric representing the start time of the process and use it as the start_timestamp. If the metric is missing, an error is returned and all metrics are discarded.

It seems odd that using *_created will fall back to using the first sample, but using the process start time will not fallback to anything.

I have a number of targets to scrape, some of which have *_created metrics, others don't have *_created but may have process_start_time_seconds. There is no way to optimally handle all these metrics with a unified configuration.

Describe the solution you'd like

Allow to config a strategy like this:

• At first, try *_created.
• If no *_created found, try process_start_time_seconds. (can be customized by setting start_time_metric_regex)
• Finnaly, use the timestamp of the first sample.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

I'm supportive of that change. If you are interested, feel free to work on a PoC to see how much refactoring, etc. is needed to make that work

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ArthurSens]

I have the impression that this was solved already. I see something similar in the docs.