chore: [k207] fix: Redo ingester profile tagging

[grafanabot]

Backport 32097c8 from #13239

---

What this PR does / why we need it:

This pr reverts 00d3c7a and takes a different approach to adding profile tagging.

Previously we were using the pprof.Do method which internal does this:

func Do(ctx context.Context, labels LabelSet, f func(context.Context)) {
	defer SetGoroutineLabels(ctx)
	ctx = WithLabels(ctx, labels)
	SetGoroutineLabels(ctx)
	f(ctx)
}

To make this pattern work we had to wrap a lot of business logic and change error handling within many of the ingester functions, this unfortunately introduced a few bugs and made the code more brittle.

This PR instead uses the same mechanisms in the Do method directly in the various ingester methods, which should hopefully be a lot less error prone and invasive.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

This PR is two commits, the first is a revert for 00d3c7a and the second is a re-implementation using the new method, it's probably a lot easier to review this PR by looking at just the second commit

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR