Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mutex for t.stopped to prevent data races #11933

Merged
merged 3 commits into from
Feb 13, 2024

Conversation

paul1r
Copy link
Collaborator

@paul1r paul1r commented Feb 13, 2024

What this PR does / why we need it:
This addresses the data race present on the t.stopped variable in tail.go.

==================
WARNING: DATA RACE
Write at 0x00c00098b198 by goroutine 568:
  github.com/grafana/loki/pkg/querier.(*Tailer).close()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:272 +0x104
  github.com/grafana/loki/pkg/querier.TestTailer.func7.2()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:169 +0x34
  runtime.deferreturn()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/panic.go:477 +0x34
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40

Previous read at 0x00c00098b198 by goroutine 569:
  github.com/grafana/loki/pkg/querier.(*Tailer).loop()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:88 +0x13c
  github.com/grafana/loki/pkg/querier.newTailer.func1()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:342 +0x34

Goroutine 568 (running) created at:
  testing.(*T).Run()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x5e8
  github.com/grafana/loki/pkg/querier.TestTailer()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:158 +0x10dc
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40

Goroutine 569 (running) created at:
  github.com/grafana/loki/pkg/querier.newTailer()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:342 +0x300
  github.com/grafana/loki/pkg/querier.TestTailer.func7()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:168 +0x138
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40
==================

Which issue(s) this PR fixes:
Relates to: #8586

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@paul1r paul1r requested a review from a team as a code owner February 13, 2024 15:04
@@ -52,6 +52,7 @@ type Tailer struct {
querierTailClientsMtx sync.RWMutex

stopped bool
stoppedMtx sync.RWMutex
Copy link
Contributor

@MasslessParticle MasslessParticle Feb 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than a mutex here, what do you think of using atomic.Bool? That way, there's less performance impact than mutexes and anyone who needs to use stopped in the future knows it's synchronized

@paul1r paul1r merged commit b2e4cc3 into main Feb 13, 2024
8 checks passed
@paul1r paul1r deleted the paul1r/querier_tail_data_race_fix branch February 13, 2024 17:45
rhnasc pushed a commit to inloco/loki that referenced this pull request Apr 12, 2024
**What this PR does / why we need it**:
This addresses the data race present on the `t.stopped` variable in
`tail.go`.

```
==================
WARNING: DATA RACE
Write at 0x00c00098b198 by goroutine 568:
  github.com/grafana/loki/pkg/querier.(*Tailer).close()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:272 +0x104
  github.com/grafana/loki/pkg/querier.TestTailer.func7.2()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:169 +0x34
  runtime.deferreturn()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/panic.go:477 +0x34
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40

Previous read at 0x00c00098b198 by goroutine 569:
  github.com/grafana/loki/pkg/querier.(*Tailer).loop()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:88 +0x13c
  github.com/grafana/loki/pkg/querier.newTailer.func1()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:342 +0x34

Goroutine 568 (running) created at:
  testing.(*T).Run()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x5e8
  github.com/grafana/loki/pkg/querier.TestTailer()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:158 +0x10dc
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40

Goroutine 569 (running) created at:
  github.com/grafana/loki/pkg/querier.newTailer()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:342 +0x300
  github.com/grafana/loki/pkg/querier.TestTailer.func7()
      /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:168 +0x138
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40
==================
```


**Which issue(s) this PR fixes**:
Relates to: grafana#8586

**Special notes for your reviewer**:

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](grafana@d10549e)
- [ ] If the change is deprecating or removing a configuration option,
update the `deprecated-config.yaml` and `deleted-config.yaml` files
respectively in the `tools/deprecated-config-checker` directory.
[Example
PR](grafana@0d4416a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants