Panics in /v1/agent/monitor endpoint on clients

[dcarbone]

Nomad version

1.10.3

Operating system and Environment details

• Centos 7.6
• Docker version 1.13.1, build 4ef4b30/1.13.1
• Three servers and three clients, all 1.10.3

Issue

We are reliably seeing panics from nomad clients on boot, unsure of specific root cause.

Will add more details once we finish rollback to 10.2.

Reproduction steps

Unclear at this time

Nomad Client logs

Feb 03 08:15:09 nomad: goroutine 448144 [running]:
Feb 03 08:15:09 nomad: net/http.(*conn).serve.func1(0xc002b4c0a0)
Feb 03 08:15:09 nomad: /usr/local/go/src/net/http/server.go:1769 +0x139
Feb 03 08:15:09 nomad: panic(0x1f88340, 0x3eb90e0)
Feb 03 08:15:09 nomad: /usr/local/go/src/runtime/panic.go:522 +0x1b5
Feb 03 08:15:09 nomad: github.com/hashicorp/nomad/nomad.(*Server).ResolveToken(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Feb 03 08:15:09 nomad: /opt/gopath/src/github.com/hashicorp/nomad/nomad/acl.go:17 +0x4e
Feb 03 08:15:09 nomad: github.com/hashicorp/nomad/command/agent.(*HTTPServer).AgentMonitor(0xc00114e140, 0x27bc580, 0xc001734240, 0xc002060100, 0xc00022f840, 0xc0044feac0, 0x38, 0xc0044feab8)
Feb 03 08:15:09 nomad: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent_endpoint.go:162 +0x9b
Feb 03 08:15:09 nomad: github.com/hashicorp/nomad/command/agent.(*HTTPServer).wrap.func1(0x27bc580, 0xc001734240, 0xc002060100)
Feb 03 08:15:09 nomad: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/http.go:388 +0x149
Feb 03 08:15:09 nomad: net/http.HandlerFunc.ServeHTTP(0xc0011fc200, 0x27bc580, 0xc001734240, 0xc002060100)
Feb 03 08:15:09 nomad: /usr/local/go/src/net/http/server.go:1995 +0x44
Feb 03 08:15:09 nomad: net/http.(*ServeMux).ServeHTTP(0xc00101d500, 0x27bc580, 0xc001734240, 0xc002060100)
Feb 03 08:15:09 nomad: /usr/local/go/src/net/http/server.go:2375 +0x1d6
Feb 03 08:15:09 nomad: github.com/hashicorp/nomad/vendor/github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x27bf780, 0xc002926ee0, 0xc002060100)
Feb 03 08:15:09 nomad: /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/NYTimes/gziphandler/gzip.go:269 +0x1db
Feb 03 08:15:09 nomad: net/http.HandlerFunc.ServeHTTP(0xc001140f90, 0x27bf780, 0xc002926ee0, 0xc002060100)
Feb 03 08:15:09 nomad: /usr/local/go/src/net/http/server.go:1995 +0x44
Feb 03 08:15:09 nomad: net/http.serverHandler.ServeHTTP(0xc00113c340, 0x27bf780, 0xc002926ee0, 0xc002060100)
Feb 03 08:15:09 nomad: /usr/local/go/src/net/http/server.go:2774 +0xa8
Feb 03 08:15:09 nomad: net/http.(*conn).serve(0xc002b4c0a0, 0x27cf040, 0xc00255e000)
Feb 03 08:15:09 nomad: /usr/local/go/src/net/http/server.go:1878 +0x851
Feb 03 08:15:09 nomad: created by net/http.(*Server).Serve
Feb 03 08:15:09 nomad: /usr/local/go/src/net/http/server.go:2884 +0x2f4
Feb 03 08:15:09 nomad: 
                                                net/http.(*conn).serve.func1(0xc002b4c0a0)
                                                        /usr/local/go/src/net/http/server.go:1769 +0x139
                                                panic(0x1f88340, 0x3eb90e0)
                                                        /usr/local/go/src/runtime/panic.go:522 +0x1b5
                                                github.com/hashicorp/nomad/nomad.(*Server).ResolveToken(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
                                                        /opt/gopath/src/github.com/hashicorp/nomad/nomad/acl.go:17 +0x4e
                                                github.com/hashicorp/nomad/command/agent.(*HTTPServer).AgentMonitor(0xc00114e140, 0x27bc580, 0xc001734240, 0xc002060100, 0xc00022f840, 0xc0044feac0, 0x38, 0xc0044feab8)
                                                        /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent_endpoint.go:162 +0x9b
                                                github.com/hashicorp/nomad/command/agent.(*HTTPServer).wrap.func1(0x27bc580, 0xc001734240, 0xc002060100)
                                                        /opt/gopath/src/github.com/hashicorp/nomad/command/agent/http.go:388 +0x149
                                                net/http.HandlerFunc.ServeHTTP(0xc0011fc200, 0x27bc580, 0xc001734240, 0xc002060100)
                                                        /usr/local/go/src/net/http/server.go:1995 +0x44
                                                net/http.(*ServeMux).ServeHTTP(0xc00101d500, 0x27bc580, 0xc001734240, 0xc002060100)
                                                        /usr/local/go/src/net/http/server.go:2375 +0x1d6
                                                github.com/hashicorp/nomad/vendor/github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x27bf780, 0xc002926ee0, 0xc002060100)
                                                        /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/NYTimes/gziphandler/gzip.go:269 +0x1db
                                                net/http.HandlerFunc.ServeHTTP(0xc001140f90, 0x27bf780, 0xc002926ee0, 0xc002060100)
                                                        /usr/local/go/src/net/http/server.go:1995 +0x44
                                                net/http.serverHandler.ServeHTTP(0xc00113c340, 0x27bf780, 0xc002926ee0, 0xc002060100)
                                                        /usr/local/go/src/net/http/server.go:2774 +0xa8
                                                net/http.(*conn).serve(0xc002b4c0a0, 0x27cf040, 0xc00255e000)
                                                        /usr/local/go/src/net/http/server.go:1878 +0x851
                                                created by net/http.(*Server).Serve
                                                        /usr/local/go/src/net/http/server.go:2884 +0x2f4

The panic is seen across all 3 clients, and afterwards they may become unresponsive.

[notnoop]

@dcarbone Very sorry that this occurred in your cluster. We have identified the cause being that the monitor http endpoint not handling running on client and causing the panic. The bug exists in both 0.10.2 and 0.10.3. I would recommend avoiding hitting /v1/agent/monitor endpoint on clients for now.

We will follow up with a fix shortly.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.