test: 2 data race fixes

[paul1r]

What this PR does / why we need it:
This PR fixes two data races.

First, in the BoltDB table manager, before the fix:

go test -count=1000 -race .  -run TestLoadTables
==================
WARNING: DATA RACE
Write at 0x00c000a4c0b0 by goroutine 2270:
  runtime.racewrite()
      <autogenerated>:1 +0x10
  github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb.(*TableManager).Stop()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/boltdb/table_manager.go:120 +0x1bc
  github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb.TestLoadTables.func3()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/boltdb/table_manager_test.go:135 +0x34
  runtime.deferreturn()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/panic.go:477 +0x34
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40

Previous read at 0x00c000a4c0b0 by goroutine 2279:
  runtime.raceread()
      <autogenerated>:1 +0x10
  github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb.(*TableManager).loop()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/boltdb/table_manager.go:98 +0x40
  github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb.NewTableManager.func1()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/boltdb/table_manager.go:93 +0x34

Goroutine 2270 (running) created at:
  testing.(*T).Run()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x5e8
  testing.runTests.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:2054 +0x80
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.runTests()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x6e4
  testing.(*M).Run()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0x9ec
  main.main()
      _testmain.go:63 +0x294

Goroutine 2279 (running) created at:
  github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb.NewTableManager()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/boltdb/table_manager.go:93 +0x2e0
  github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb.TestLoadTables()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/boltdb/table_manager_test.go:133 +0x878
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40
==================
--- FAIL: TestLoadTables (0.18s)
    testing.go:1465: race detected during execution of test
FAIL
FAIL	github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb	183.552s
FAIL

After the fix:

go test -count=1000 -race .  -run TestLoadTables
ok  	github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/boltdb	189.300s

Second is a test routine that was accessing variables outside of the locking mechanism. Before the fix:

WARNING: DATA RACE
Write at 0x00c000698b40 by goroutine 37:
  runtime.mapassign_faststr()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/map_faststr.go:203 +0x40c
  github.com/grafana/loki/v3/pkg/storage/chunk/cache.(*EmbeddedCache[go.shape.string,go.shape.[]uint8]).remove()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/chunk/cache/embeddedcache.go:263 +0x2c8
  github.com/grafana/loki/v3/pkg/storage/chunk/cache.(*EmbeddedCache[go.shape.string,go.shape.[]uint8]).pruneExpiredItems()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/chunk/cache/embeddedcache.go:210 +0x1e8
  github.com/grafana/loki/v3/pkg/storage/chunk/cache.(*EmbeddedCache[go.shape.string,go.shape.[]uint8]).runPruneJob()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/chunk/cache/embeddedcache.go:197 +0xbc
  github.com/grafana/loki/v3/pkg/storage/chunk/cache.NewTypedEmbeddedCache[go.shape.string,go.shape.[]uint8].func1()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/chunk/cache/embeddedcache.go:182 +0x58

Previous read at 0x00c000698b40 by goroutine 36:
  github.com/grafana/loki/v3/pkg/storage/chunk/cache.TestEmbeddedCacheExpiry()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/chunk/cache/embeddedcache_test.go:210 +0x9b4
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40

Goroutine 37 (running) created at:
  github.com/grafana/loki/v3/pkg/storage/chunk/cache.NewTypedEmbeddedCache[go.shape.string,go.shape.[]uint8]()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/chunk/cache/embeddedcache.go:182 +0xb04
  github.com/grafana/loki/v3/pkg/storage/chunk/cache.TestEmbeddedCacheExpiry()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/storage/chunk/cache/embeddedcache_test.go:193 +0x4a4
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40

Goroutine 36 (running) created at:
  testing.(*T).Run()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x5e8
  testing.runTests.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:2054 +0x80
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.runTests()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x6e4
  testing.(*M).Run()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0x9ec
  main.main()
      _testmain.go:79 +0x294
==================
--- FAIL: TestEmbeddedCacheExpiry (0.20s)
    testing.go:1465: race detected during execution of test
FAIL
FAIL	github.com/grafana/loki/v3/pkg/storage/chunk/cache	0.702s
FAIL

After the fix:

go test -count=500 -race . -run TestEmbeddedCacheExpiry
ok  	github.com/grafana/loki/v3/pkg/storage/chunk/cache	102.491s

Which issue(s) this PR fixes:
Relates to #8586

Special notes for your reviewer:

Checklist

• [ x] Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• [x ] Tests updated
• Title matches the required conventional commits format, see here
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR