Switch from counter to a gauge for partitions held #1485
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bobrik
requested review from
black-adder,
jpkrohling,
objectiser,
pavolloffay,
tiffon,
vprithvi and
yurishkuro
as code owners
Counters cannot be decremented in Prometheus:
```
panic: counter cannot decrease in value
goroutine 895 [running]:
github.com/jaegertracing/jaeger/vendor/github.com/prometheus/client_golang/prometheus.(*counter).Add(0xc000790600, 0xbff0000000000000)
/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/prometheus/client_golang/prometheus/counter.go:71 +0xa3
github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-lib/metrics/prometheus.(*counter).Inc(0xc0006b42a0, 0xffffffffffffffff)
/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-lib/metrics/prometheus/factory.go:183 +0x46
github.com/jaegertracing/jaeger/cmd/ingester/app/consumer.(*Consumer).handleMessages(0xc0004c4300, 0xf08c60, 0xc00054e630)
/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/ingester/app/consumer/consumer.go:124 +0x893
created by github.com/jaegertracing/jaeger/cmd/ingester/app/consumer.(*Consumer).Start.func1
/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/ingester/app/consumer/consumer.go:87 +0xbd
```
Gauges can, even though we have to keep an extra variable
around to keep count. In Prometheus Go library itself that
is not necessary as Gauge type provides `Inc` and `Dec`,
but Jaeger's wrapper does not have those exposed.
Fixes jaegertracing#1200.
Signed-off-by: Ivan Babrou <[email protected]>
bobrik
force-pushed
the
only-dec-for-gauge
branch
from
5e32ed2 to
fa84803
Compare
Codecov Report
@@ Coverage Diff @@
## master #1485 +/- ##
==========================================
+ Coverage 99.81% 99.81% +<.01%
==========================================
Files 180 180
Lines 8631 8637 +6
==========================================
+ Hits 8615 8621 +6
Misses 8 8
Partials 8 8
Continue to review full report at Codecov.
|
vprithvi
reviewed
Apr 23, 2019
| @@ -109,8 +110,12 @@ func (c *Consumer) Close() error { | |||
|
|
|||
| func (c *Consumer) handleMessages(pc sc.PartitionConsumer) { | |||
| c.logger.Info("Starting message handler", zap.Int32("partition", pc.Partition())) | |||
| c.partitionsHeld.Inc(1) | |||
| defer c.partitionsHeld.Inc(-1) | |||
| c.partitionsHeld++ | |||
There was a problem hiding this comment.
Wouldn't this require some sort of synchronization? L92 starts handleMessages as a goroutine
There was a problem hiding this comment.
You're probably right, so I moved partitionMapLock.Lock() around to encompass this.
There was a problem hiding this comment.
seems like c.partitionMapLock.Lock() should be enough
|
similar PR #1254 |
bobrik
force-pushed
the
only-dec-for-gauge
branch
from
3cd65d2 to
d3c9ccd
Compare
yurishkuro
reviewed
Apr 23, 2019
| c.partitionsHeld-- | ||
| c.partitionsHeldGauge.Update(c.partitionsHeld) | ||
| c.partitionMapLock.Unlock() | ||
| }() |
There was a problem hiding this comment.
I recommend moving this after L123 (after unlock) and combining L124-125 into the same defer func()
There was a problem hiding this comment.
Makes sense, amended.
Signed-off-by: Ivan Babrou <[email protected]>
bobrik
force-pushed
the
only-dec-for-gauge
branch
from
d3c9ccd to
2582040
Compare
yurishkuro
approved these changes
Apr 23, 2019
|
CI is happy ✅ |
|
Thanks for the PR! |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Counters cannot be decremented in Prometheus:
Gauges can, even though we have to keep an extra variable around to keep count. In Prometheus Go library itself that is not necessary as Gauge type provides
IncandDec, but Jaeger's wrapper does not have those exposed.Fixes #1200.
Signed-off-by: Ivan Babrou [email protected]