[release-2.9.x] Series Index Store: fix race in GetSeries (#10346)

Backport cf353bb from #10310

---

**What this PR does / why we need it**:
~Lock around data that is modified from multiple goroutines
concurrently.~
Replaced with the implementation from #9984 since @akhilanarayanan did
it more efficiently.

**Which issue(s) this PR fixes**:
Relates to #8586

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- NA Documentation added
- NA Tests updated
- [x] `CHANGELOG.md` updated
- NA If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- NA Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- NA For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)

Co-authored-by: Bryan Boreham <[email protected]>

CHANGELOG.md

@@ -73,6 +73,7 @@
 * [9773](https://github.com/grafana/loki/pull/9773) **ssncferreira**: Fix instant query summary statistic's `splits` corresponding to the number of subqueries a query is split into based on `split_queries_by_interval`.
 * [9949](https://github.com/grafana/loki/pull/9949) **masslessparticle**: Fix pipelines to clear caches when tailing to avoid resource exhaustion.
 * [9936](https://github.com/grafana/loki/pull/9936) **masslessparticle**: Fix the way query stages are reordered when `unpack` is present.
+* [10309](https://github.com/grafana/loki/pull/10309) **akhilanarayanan**: Fix race condition in series index store.
 
 ##### Changes
 

pkg/storage/stores/series/series_index_store.go

@@ -279,7 +279,7 @@ func (c *indexReaderWriter) chunksToSeries(ctx context.Context, in []logproto.Ch
 		}))
 	}
 
-	results := make([]labels.Labels, 0, len(chunksBySeries))
+	perJobResults := make([][]labels.Labels, len(jobs))
 
 	// Picking an arbitrary bound of 20 numConcurrent jobs.
 	numConcurrent := len(jobs)
@@ -294,14 +294,18 @@ func (c *indexReaderWriter) chunksToSeries(ctx context.Context, in []logproto.Ch
 		func(_ context.Context, idx int) error {
 			res, err := jobs[idx]()
 			if res != nil {
-				results = append(results, res...)
+				perJobResults[idx] = res
 			}
 			return err
 		},
 	); err != nil {
 		return nil, err
 	}
 
+	results := make([]labels.Labels, len(chunksBySeries)) // Flatten out the per-job results.
+	for _, innerSlice := range perJobResults {
+		results = append(results, innerSlice...)
+	}
 	sort.Slice(results, func(i, j int) bool {
 		return labels.Compare(results[i], results[j]) < 0
 	})