Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic sending allocation metrics #3188

Closed
epipho opened this issue Sep 11, 2017 · 3 comments
Closed

Panic sending allocation metrics #3188

epipho opened this issue Sep 11, 2017 · 3 comments
Assignees
@chelseakomlo

Comments

@epipho
Copy link

epipho commented Sep 11, 2017

Attempting to test support for tagged metrics using master latest and the following telemetry stanza.

   "telemetry": {
        "publish_allocation_metrics": true,
        "publish_node_metrics": true,
        "datadog_address": "<local host ip>:8129",
        "disable_tagged_metrics": false,
        "backwards_compatible_metrics": false
    }

Nomad version

v0.6.3-dev
Built from 03e63d0

Operating system and Environment details

CoreOS Linux / In Docker

Issue

Panic attempting to send metrics.

Suspect it was introduced here:

7a96f92#diff-1cd41a2611da49ce63630d5a51bd82c5R1942

where c.configLock.RLock() was removed.

Reproduction steps

Turn on tagged metrics + allocation metrics on latest build.

Nomad Client logs (if appropriate)

fatal error: sync: RUnlock of unlocked RWMutex

goroutine 62 [running]:
runtime.throw(0x1395956, 0x21)
        /usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc420335c88 sp=0xc420335c68 pc=0x42d895
sync.throw(0x1395956, 0x21)
        /usr/local/go/src/runtime/panic.go:594 +0x35 fp=0xc420335ca8 sp=0xc420335c88 pc=0x42d7f5
sync.(*RWMutex).RUnlock(0xc4200aa2f0)
        /usr/local/go/src/sync/rwmutex.go:71 +0x50 fp=0xc420335cc8 sp=0xc420335ca8 pc=0x468290
github.com/hashicorp/nomad/client.(*Client).setGaugeForAllocationStats(0xc4200aa2c0, 0xc42017d2c0, 0x24)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:1944 +0x5a fp=0xc420335e80 sp=0xc420335cc8 pc=0xeccdda
github.com/hashicorp/nomad/client.(*Client).emitHostStats(0xc4200aa2c0)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:2038 +0x11d fp=0xc420335ec8 sp=0xc420335e80 pc=0xece8cd
github.com/hashicorp/nomad/client.(*Client).emitStats(0xc4200aa2c0)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:1870 +0x346 fp=0xc420335fd8 sp=0xc420335ec8 pc=0xecac66
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420335fe0 sp=0xc420335fd8 pc=0x45ebb1
created by github.com/hashicorp/nomad/client.NewClient
        /root/go/src/github.com/hashicorp/nomad/client/client.go:297 +0xd79

goroutine 1 [runnable]:
os/signal.signal_enable(0xf)
        /usr/local/go/src/runtime/sigqueue.go:190 +0x7c
os/signal.enableSignal(0xf)
        /usr/local/go/src/os/signal/signal_unix.go:49 +0x2a
os/signal.Notify.func1(0xf)
        /usr/local/go/src/os/signal/signal.go:129 +0xa3
os/signal.Notify(0xc42005afc0, 0xc4204c77b8, 0x4, 0x4)
        /usr/local/go/src/os/signal/signal.go:141 +0x131
github.com/hashicorp/nomad/command/agent.(*Command).handleSignals(0xc420012900, 0xc42041ab40, 0xc420012900)
        /root/go/src/github.com/hashicorp/nomad/command/agent/command.go:523 +0x105
github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc420012900, 0xc4200100e0, 0xa, 0xa, 0x0)
        /root/go/src/github.com/hashicorp/nomad/command/agent/command.go:517 +0xd52
github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc42009dc20, 0xc42009dc20, 0x2e, 0x13d1a18)
        /root/go/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:242 +0x1eb
main.RunCustom(0xc4200100d0, 0xb, 0xb, 0xc420454e70, 0xc42017df60)
        /root/go/src/github.com/hashicorp/nomad/main.go:55 +0xfc3
main.Run(0xc4200100d0, 0xb, 0xb, 0x10078b6)
        /root/go/src/github.com/hashicorp/nomad/main.go:24 +0x56
main.main()
        /root/go/src/github.com/hashicorp/nomad/main.go:20 +0x63

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 11 [select]:
github.com/hashicorp/nomad/client.(*AllocGarbageCollector).Run(0xc42006c190)
        /root/go/src/github.com/hashicorp/nomad/client/gc.go:75 +0x10e
created by github.com/hashicorp/nomad/client.NewClient
        /root/go/src/github.com/hashicorp/nomad/client/client.go:221 +0x6de

goroutine 10 [select]:
github.com/hashicorp/nomad/nomad.(*ConnPool).reap(0xc42006c0f0)
        /root/go/src/github.com/hashicorp/nomad/nomad/pool.go:380 +0x37b
created by github.com/hashicorp/nomad/nomad.NewPool
        /root/go/src/github.com/hashicorp/nomad/nomad/pool.go:155 +0x179

goroutine 9 [select]:
github.com/hashicorp/nomad/command/agent/consul.(*ServiceClient).Run(0xc420090370)
        /root/go/src/github.com/hashicorp/nomad/command/agent/consul/client.go:274 +0x1fc
created by github.com/hashicorp/nomad/command/agent.(*Agent).setupConsul
        /root/go/src/github.com/hashicorp/nomad/command/agent/agent.go:744 +0x4e9

goroutine 25 [select, locked to thread]:
runtime.gopark(0x13d22f0, 0x0, 0x1370d8e, 0x6, 0x18, 0x1)
        /usr/local/go/src/runtime/proc.go:277 +0x12c
runtime.selectgo(0xc420023750, 0xc420064420)
        /usr/local/go/src/runtime/select.go:395 +0x1138
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal_unix.go:511 +0x220
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 26 [select]:
github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*InmemSignal).run(0xc42038d500)
        /root/go/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:64 +0xf8
created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.NewInmemSignal
        /root/go/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:38 +0x162

goroutine 27 [sleep]:
time.Sleep(0x3b9aca00)
        /usr/local/go/src/runtime/time.go:65 +0x130
github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*Metrics).collectStats(0xc4201ce420)
        /root/go/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/metrics.go:168 +0x2f
created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.New
        /root/go/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/start.go:75 +0x12b

goroutine 33 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc4200aa2c0, 0x13703da, 0x6, 0x1c92fc0, 0xc420484240, 0x37e11d600)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:919 +0x224
created by github.com/hashicorp/nomad/client.(*Client).fingerprint
        /root/go/src/github.com/hashicorp/nomad/client/client.go:905 +0x6d0

goroutine 60 [select]:
github.com/hashicorp/nomad/client.(*Client).allocSync(0xc4200aa2c0)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:1248 +0x209
created by github.com/hashicorp/nomad/client.NewClient
        /root/go/src/github.com/hashicorp/nomad/client/client.go:291 +0xd35

goroutine 31 [IO wait]:
internal/poll.runtime_pollWait(0x7f76e8d7deb0, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4201b7a98, 0x72, 0xffffffffffffff00, 0x1c91c80, 0x1c8ac10)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4201b7a98, 0xc4203e6000, 0x1000, 0x1000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4201b7a80, 0xc4203e6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc4201b7a80, 0xc4203e6000, 0x1000, 0x1000, 0x42f55b, 0xc420037a40, 0x45ac30)
        /usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc4201b0228, 0xc4203e6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6d
net/http.(*persistConn).Read(0xc42027b7a0, 0xc4203e6000, 0x1000, 0x1000, 0xc4202d64a0, 0xc420064ad8, 0x458cd0)
        /usr/local/go/src/net/http/transport.go:1391 +0x140
bufio.(*Reader).fill(0xc42031fda0)
        /usr/local/go/src/bufio/bufio.go:97 +0x11a
bufio.(*Reader).Peek(0xc42031fda0, 0x1, 0x0, 0x0, 0x0, 0xc4200649c0, 0x0)
        /usr/local/go/src/bufio/bufio.go:129 +0x3a
net/http.(*persistConn).readLoop(0xc42027b7a0)
        /usr/local/go/src/net/http/transport.go:1539 +0x185
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1186 +0xa2e

goroutine 32 [select]:
net/http.(*persistConn).writeLoop(0xc42027b7a0)
        /usr/local/go/src/net/http/transport.go:1759 +0x165
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1187 +0xa53

goroutine 12 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc4200aa2c0, 0x1370392, 0x6, 0x1c92f40, 0xc420492000, 0x37e11d600)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:919 +0x224
created by github.com/hashicorp/nomad/client.(*Client).fingerprint
        /root/go/src/github.com/hashicorp/nomad/client/client.go:905 +0x6d0

goroutine 35 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc4200aa2c0, 0x136f902, 0x5, 0x1c93200, 0xc420484540, 0x37e11d600)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:919 +0x224
created by github.com/hashicorp/nomad/client.(*Client).fingerprint
        /root/go/src/github.com/hashicorp/nomad/client/client.go:905 +0x6d0

goroutine 61 [select]:
github.com/hashicorp/nomad/client.(*Client).run(0xc4200aa2c0)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:1071 +0x153
created by github.com/hashicorp/nomad/client.NewClient
        /root/go/src/github.com/hashicorp/nomad/client/client.go:294 +0xd57

goroutine 16 [IO wait]:
internal/poll.runtime_pollWait(0x7f76e8d7ddf0, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4203ee598, 0x72, 0xffffffffffffff00, 0x1c91c80, 0x1c8ac10)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4203ee598, 0xc420354000, 0x1000, 0x1000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4203ee580, 0xc420354000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc4203ee580, 0xc420354000, 0x1000, 0x1000, 0x42f55b, 0xc42033ba40, 0x45ac30)
        /usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42000e508, 0xc420354000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6d
net/http.(*persistConn).Read(0xc4203057a0, 0xc420354000, 0x1000, 0x1000, 0xc4204255e0, 0xc420307198, 0x458cd0)
        /usr/local/go/src/net/http/transport.go:1391 +0x140
bufio.(*Reader).fill(0xc42005a7e0)
        /usr/local/go/src/bufio/bufio.go:97 +0x11a
bufio.(*Reader).Peek(0xc42005a7e0, 0x1, 0x0, 0x0, 0x0, 0xc420307080, 0x0)
        /usr/local/go/src/bufio/bufio.go:129 +0x3a
net/http.(*persistConn).readLoop(0xc4203057a0)
        /usr/local/go/src/net/http/transport.go:1539 +0x185
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1186 +0xa2e

goroutine 34 [select]:
net/http.(*persistConn).writeLoop(0xc4203057a0)
        /usr/local/go/src/net/http/transport.go:1759 +0x165
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1187 +0xa53

goroutine 59 [select]:
github.com/hashicorp/nomad/client.(*Client).periodicSnapshot(0xc4200aa2c0)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:1051 +0x103
created by github.com/hashicorp/nomad/client.NewClient
        /root/go/src/github.com/hashicorp/nomad/client/client.go:288 +0xd13

goroutine 63 [IO wait]:
internal/poll.runtime_pollWait(0x7f76e8d7dbb0, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4203ee918, 0x72, 0xc420538d00, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4203ee918, 0xffffffffffffff00, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc4203ee900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc4203ee900, 0x413008, 0x30, 0x1278ca0)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc42000e570, 0x5a071a, 0x1278ca0, 0xc42035db30)
        /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e570, 0xc4200160b8, 0x1173840, 0x1c75e10, 0x133c9a0)
        /usr/local/go/src/net/tcpsock.go:247 +0x49
net/http.(*Server).Serve(0xc4205300d0, 0x1c99440, 0xc42000e570, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:2695 +0x1b2
net/http.Serve(0x1c99440, 0xc42000e570, 0x1c91180, 0xc420552680, 0x1c99440, 0xc42000e570)
        /usr/local/go/src/net/http/server.go:2323 +0x73
created by github.com/hashicorp/nomad/command/agent.NewHTTPServer
        /root/go/src/github.com/hashicorp/nomad/command/agent/http.go:84 +0x36d

goroutine 58 [select]:
github.com/hashicorp/nomad/client.(*Client).retryRegisterNode(0xc4200aa2c0)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:1119 +0x1ff
github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat(0xc4200aa2c0)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:997 +0x43
created by github.com/hashicorp/nomad/client.NewClient
        /root/go/src/github.com/hashicorp/nomad/client/client.go:285 +0xcf1

goroutine 47 [select]:
github.com/hashicorp/nomad/client.(*Client).watchAllocations(0xc4200aa2c0, 0xc4201cb6e0)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:1352 +0x536
created by github.com/hashicorp/nomad/client.(*Client).run
        /root/go/src/github.com/hashicorp/nomad/client/client.go:1068 +0x7c

goroutine 56 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc4200aa2c0, 0x13704ac, 0x6, 0x7f76e8d4bfc8, 0xc4201caa80, 0x37e11d600)
        /root/go/src/github.com/hashicorp/nomad/client/client.go:919 +0x224
created by github.com/hashicorp/nomad/client.(*Client).setupDrivers
        /root/go/src/github.com/hashicorp/nomad/client/client.go:971 +0x5fb

goroutine 64 [runnable]:
github.com/hashicorp/nomad/command/agent.(*Command).retryJoin(0xc420012900, 0xc42041ab40)
        /root/go/src/github.com/hashicorp/nomad/command/agent/command.go:784
created by github.com/hashicorp/nomad/command/agent.(*Command).Run
        /root/go/src/github.com/hashicorp/nomad/command/agent/command.go:514 +0xd34
@epipho epipho changed the title [ISSUE] Panic sending allocation metrics Panic sending allocation metrics Sep 11, 2017
@shantanugadgil shantanugadgil mentioned this issue Sep 11, 2017
Merged
@chelseakomlo chelseakomlo self-assigned this Sep 11, 2017
@chelseakomlo chelseakomlo mentioned this issue Sep 11, 2017
Merged
@epipho
Copy link
Author

epipho commented Sep 12, 2017

Tested fix from #3191, all is better, metrics are emitting correctly. Thanks for the quick fix.

@dadgar
Copy link
Contributor

dadgar commented Sep 21, 2017

Closing as it has been fixed.

@dadgar dadgar closed this as completed Sep 21, 2017
@github-actions
Copy link

github-actions bot commented Dec 7, 2022

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 7, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@chelseakomlo chelseakomlo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@epipho @dadgar @chelseakomlo