Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic on Windows Server 2016 #2343

Closed
schmichael opened this issue Feb 22, 2017 · 15 comments · Fixed by #2378
Closed

Panic on Windows Server 2016 #2343

schmichael opened this issue Feb 22, 2017 · 15 comments · Fixed by #2378

Comments

@schmichael
Copy link
Member

schmichael commented Feb 22, 2017

Panic on Windows Server 2016 with Docker 1.13.1-cs1 first encountered on #2193 (comment)

Attached is an executable built from ff5ea7a

nomad-amd64.zip

Workaround

The bug is triggered when a docker image is downloaded. You can workaround the bug by running docker pull redis:3.0-windowsservercore manually before submitting the job.

The nomad process will still panic on its next exit (perhaps only if it tries to stop/cleanup the Docker image?).

Reproduce

Download, extract, and run the attached binary with:

nomad agent -dev

Then submit the following job file with nomad run ...:

job "example" {
  datacenters = ["dc1"]
  type = "service"
  update {
    stagger = "10s"
    max_parallel = 1
  }
  group "cache" {
    count = 1
    restart {
      attempts = 10
      interval = "5m"
      delay = "25s"
      mode = "delay"
    }
    ephemeral_disk {
      size = 300
    }
    task "redis" {
      driver = "docker"
      config {
        image = "redis:3.0-windowsservercore"
        port_map {
          db = 6379
        }
      }
      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
        network {
          mbits = 10
          port "db" {}
        }
      }

      service {
        name = "global-redis-check"
        tags = ["global", "cache"]
        port = "db"
        check {
          name     = "alive"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}
@schmichael
Copy link
Member Author

Turns out this wasn't fixed. Testing must have missed the race initially, but further testing shows it to still occur whenever docker images are pulled or removed.

Seems to be caused by these upstream issues:
microsoft/go-winio#32
microsoft/go-winio#41

If anyone knows of a mitigation we'd love to implement it!

Workaround

In the mean time I think the only workaround for Docker on Windows is to pull images using Docker's CLI ahead of time. That should skip both of the points Nomad can trigger this crash.

@cvandal
Copy link

cvandal commented Apr 26, 2017

Hey @schmichael, I'm now facing this issue as well but unfortunately pulling the Docker image before running the job doesn't help, I still get the panic.

Nomad Version: 0.5.6
Docker Version: 17.03.1-ee-3, build 3fcee33

More than happy to help troubleshoot this problem.

@schmichael
Copy link
Member Author

@cvandal Any chance you could post the panic to a gist or if it's too large as a compressed attachment in a comment?

I suspect this is a race condition in much if not all IO on Windows and that for whatever reason pulling is just the easiest place to hit it.

I'll try to make a build with this fix for testing: microsoft/go-winio#31

@cvandal
Copy link

cvandal commented Apr 26, 2017

Standard startup logs

==> Starting Nomad agent...
==> Nomad agent configuration:

             Atlas: <disabled>
            Client: true
         Log Level: INFO
            Region: global (DC: ap-southeast-2)
            Server: false
           Version: 0.5.6

==> Nomad agent started! Log data will stream in below:

2017/04/26 20:50:33.210497 [INFO] client: using state directory C:\tmp\client
2017/04/26 20:50:33.210497 [INFO] client: using alloc directory C:\tmp\alloc
2017/04/26 20:50:33.371501 [INFO] fingerprint.consul: consul agent is available
2017/04/26 20:50:35.512477 [WARN]: fingerprint.env_aws: Could not read value for attribute "public-hostname"
2017/04/26 20:50:35.512477 [WARN]: fingerprint.env_aws: Could not read value for attribute "public-ipv4"
2017/04/26 20:50:35.521477 [INFO] client: Node ID "e8fc34f9-92cd-4dfb-b6c4-d6b1135bfe16"
2017/04/26 20:50:35.528476 [INFO] client.consul: discovered following Servers: 10.123.28.153:4647,10.123.27.155:4647

,10.123.26.132:4647
2017/04/26 20:50:35.556479 [INFO] client: node registration complete

2017/04/26 20:51:08.395080 [INFO] driver.docker: created container 21df8120eefebd09a829b5344d985279b7a72035eafd71339

8e615334352d94b
2017/04/26 20:51:08.397079 [INFO] driver.docker: created container ce6782e273520703b49874998e3b700e93cd7b964d60ac45e
d6746a79a5f1c72
2017/04/26 20:51:09.424047 [INFO] driver.docker: started container ce6782e273520703b49874998e3b700e93cd7b964d60ac45e
d6746a79a5f1c72
2017/04/26 20:51:09.430067 [WARN] client: error fetching stats of task WebApplication1: stats collection hasn't star
ted yet
2017/04/26 20:51:09.445066 [INFO] driver.docker: started container 21df8120eefebd09a829b5344d985279b7a72035eafd71339
8e615334352d94b
2017/04/26 20:51:09.450068 [WARN] client: error fetching stats of task WebApplication1: stats collection hasn't star
ted yet
fatal error: unexpected signal during runtime execution
[signal 0xc0000005 code=0x0 addr=0x1000002b8 pc=0x40e892]

goroutine 123 [running]:
runtime.throw(0x12a2e10, 0x2a)
/opt/go/src/runtime/panic.go:596 +0x9c fp=0xc0424cbe18 sp=0xc0424cbdf8
runtime.sigpanic()
/opt/go/src/runtime/signal_windows.go:155 +0x18b fp=0xc0424cbe48 sp=0xc0424cbe18
runtime.unlock(0x184e198)
/opt/go/src/runtime/lock_sema.go:107 +0x72 fp=0xc0424cbe70 sp=0xc0424cbe48
runtime.chansend(0x10511c0, 0x184e140, 0xc0424cbfb0, 0xc0424cbf01, 0xac192a, 0xc0424cbf9c)
/opt/go/src/runtime/chan.go:179 +0x8af fp=0xc0424cbf20 sp=0xc0424cbe70
runtime.chansend1(0x10511c0, 0x184e140, 0xc0424cbfb0)
/opt/go/src/runtime/chan.go:113 +0x4d fp=0xc0424cbf60 sp=0xc0424cbf20
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.ioCompletionProcessor(0x3a8)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:133 +0xea fp=0xc0424cbfd
8 sp=0xc0424cbf60
runtime.goexit()
/opt/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc0424cbfe0 sp=0xc0424cbfd8
created by github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.initIo
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:57 +0x87

Full goroutine dump

goroutine 1 [select, 1 minutes]:
github.com/hashicorp/nomad/command/agent.(*Command).handleSignals(0xc0423c8280, 0xc0423e8l2c0, 0xc0423c8280)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:526 +0x72d
github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc0423c8280, 0xc042010160, 0x3, 0x3, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:515 +0xf2b
github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc0423c40c0, 0xc0423c40c0, 0x1e, 0xc0420fe220)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:154 +0x1af
main.RunCustom(0xc042010150, 0x4, 0x4, 0xc0420ea540, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/main.go:57 +0xa61
main.Run(0xc042010150, 0x4, 0x4, 0xc042020000)
/opt/gopath/src/github.com/hashicorp/nomad/main.go:20 +0x5d
main.main()
/opt/gopath/src/github.com/hashicorp/nomad/main.go:16 +0x6b

goroutine 17 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
/opt/go/src/runtime/sigqueue.go:116 +0x10b
os/signal.loop()
/opt/go/src/os/signal/signal_unix.go:22 +0x29
created by os/signal.init.1
/opt/go/src/os/signal/signal_unix.go:28 +0x48

goroutine 22 [sleep]:
time.Sleep(0x3b9aca00)
/opt/go/src/runtime/time.go:59 +0x107
github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*Metrics).collectStats(0xc0423b2410)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/metrics.go:67 +0x36
created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.New
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/start.go:61 +0xde

goroutine 21 [select, 1 minutes]:
github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.(*InmemSignal).run(0xc0423b43c0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:63 +0x112
created by github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics.NewInmemSignal
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/armon/go-metrics/inmem_signal.go:37 +0x17f

goroutine 23 [select]:
github.com/hashicorp/nomad/nomad.(*ConnPool).reap(0xc0423b2500)
/opt/gopath/src/github.com/hashicorp/nomad/nomad/pool.go:380 +0x435
created by github.com/hashicorp/nomad/nomad.NewPool
/opt/gopath/src/github.com/hashicorp/nomad/nomad/pool.go:155 +0x19f

goroutine 24 [select]:
github.com/hashicorp/nomad/client.(*AllocGarbageCollector).run(0xc0423b4880)
/opt/gopath/src/github.com/hashicorp/nomad/client/gc.go:63 +0x1c4
created by github.com/hashicorp/nomad/client.NewAllocGarbageCollector
/opt/gopath/src/github.com/hashicorp/nomad/client/gc.go:56 +0x24c

goroutine 129 [select]:
github.com/hashicorp/nomad/client.(*Client).consulReaper(0xc0423ae780)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:2149 +0x1ff
created by github.com/hashicorp/nomad/client.NewClient
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:289 +0x9b2

goroutine 35 [IO wait]:
net.runtime_pollWait(0x47e0f70, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0424581a0, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc042458060, 0x126fe30, 0x7, 0x12c78e8, 0x1, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc042458000, 0xc04245a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc042100050, 0xc04245a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:181 +0x77
net/http.(*persistConn).Read(0xc0423d8120, 0xc04245a000, 0x1000, 0x1000, 0x4, 0x454540, 0xc04203c540)
/opt/go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc0423bc6c0)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc0423bc6c0, 0x1, 0xc042117be5, 0x1, 0x0, 0xc042512f60, 0x0)
/opt/go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc0423d8120)
/opt/go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1117 +0xa3c

goroutine 36 [select]:
net/http.(*persistConn).writeLoop(0xc0423d8120)
/opt/go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1118 +0xa61

goroutine 9 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc0423ae780, 0x126e91d, 0x6, 0x184f340, 0xc04244d540, 0
x37e11d600)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:853 +0x396
created by github.com/hashicorp/nomad/client.(*Client).fingerprint
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:839 +0x6c5

goroutine 10 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc0423ae780, 0x126dd96, 0x5, 0x184f580, 0xc042002e80, 0
x37e11d600)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:853 +0x396
created by github.com/hashicorp/nomad/client.(*Client).fingerprint
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:839 +0x6c5

goroutine 277 [syscall, locked to thread]:
syscall.Syscall6(0x7ffaad984750, 0x5, 0x43c, 0xc0425b0000, 0x8000, 0xc042475dac, 0x0, 0x0, 0x4, 0x3, ...)
/opt/go/src/runtime/syscall_windows.go:174 +0x6b
syscall.ReadFile(0x43c, 0xc0425b0000, 0x8000, 0x8000, 0xc042475dac, 0x0, 0xc042475dc8, 0xc042475dc0)
/opt/go/src/syscall/zsyscall_windows.go:311 +0xe6
syscall.Read(0x43c, 0xc0425b0000, 0x8000, 0x8000, 0xc042475e18, 0x463cb7, 0xc042300548)
/opt/go/src/syscall/syscall_windows.go:296 +0x6f
os.(*File).read(0xc0420f6660, 0xc0425b0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/opt/go/src/os/file_windows.go:289 +0xd0
os.(*File).Read(0xc0420f6660, 0xc0425b0000, 0x8000, 0x8000, 0x48, 0x0, 0x0)
/opt/go/src/os/file.go:101 +0x7d
io.copyBuffer(0x184bbc0, 0xc0420f6638, 0x184c640, 0xc0420f6660, 0xc0425b0000, 0x8000, 0x8000, 0xc042234c00, 0x6e0259, 0x
180001)
/opt/go/src/io/io.go:390 +0x107
io.Copy(0x184bbc0, 0xc0420f6638, 0x184c640, 0xc0420f6660, 0x0, 0x0, 0x0)
/opt/go/src/io/io.go:360 +0x6f
os/exec.(*Cmd).writerDescriptor.func1(0x434d8f, 0x12c7f78)
/opt/go/src/os/exec/exec.go:254 +0x54
os/exec.(*Cmd).Start.func1(0xc0421f3080, 0xc04222e6c0)
/opt/go/src/os/exec/exec.go:371 +0x2e
created by os/exec.(*Cmd).Start
/opt/go/src/os/exec/exec.go:372 +0x55e

goroutine 278 [syscall, locked to thread]:
syscall.Syscall(0x7ffaad984360, 0x2, 0x440, 0xffffffff, 0x0, 0x3f12bf0, 0xc04211bf48, 0x2)
/opt/go/src/runtime/syscall_windows.go:163 +0x6b
syscall.WaitForSingleObject(0x440, 0xffffffff, 0x45b550, 0xc04202b368, 0x0)
/opt/go/src/syscall/zsyscall_windows.go:718 +0x6d
os.(*Process).wait(0xc042168000, 0x0, 0x0, 0x0)
/opt/go/src/os/exec_windows.go:18 +0xa0
os.(*Process).Wait(0xc042168000, 0xc04211bfb0, 0x5e150b, 0xc04211bf30)
/opt/go/src/os/exec.go:115 +0x32
os/exec.(*Cmd).Wait(0xc0421f3080, 0x12c7870, 0xc0420f6628)
/opt/go/src/os/exec/exec.go:435 +0x69
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func3(0xc0420f6638, 0xc0420f6628, 0xc04
21f3080, 0xc0421ed680, 0xc042446640)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:378 +0x8a
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:391 +0xc3c

goroutine 279 [semacquire]:
sync.runtime_notifyListWait(0xc042300580, 0xc000000001)
/opt/go/src/runtime/sema.go:297 +0x119
sync.(*Cond).Wait(0xc042300570)
/opt/go/src/sync/cond.go:57 +0x90
io.(*pipe).read(0xc042300540, 0xc04255c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/io/pipe.go:47 +0x10b
io.(*PipeReader).Read(0xc0420f6630, 0xc04255c000, 0x1000, 0x1000, 0x1e0, 0xc042110180, 0x76)
/opt/go/src/io/pipe.go:130 +0x53
bufio.(*Reader).fill(0xc042477f08)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).ReadSlice(0xc042477f08, 0x42c60a, 0x12c8250, 0xc042477d80, 0x0, 0xc0424801a0, 0x12c8250)
/opt/go/src/bufio/bufio.go:338 +0xc2
bufio.(*Reader).ReadBytes(0xc042477f08, 0x65680a, 0xc0420103c0, 0x2, 0xc042210000, 0x62, 0x0)
/opt/go/src/bufio/bufio.go:416 +0x6d
bufio.(*Reader).ReadString(0xc042477f08, 0xa, 0xc042477ee8, 0x2, 0x2, 0x0)
/opt/go/src/bufio/bufio.go:456 +0x3f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).logStderr(0xc042446640, 0x184bb80, 0xc0420f66
30)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:526 +0xaf
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:394 +0xc7a

goroutine 221 [select]:
github.com/hashicorp/nomad/client.(*AllocRunner).Run(0xc0423ec2d0)
/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:530 +0x741
created by github.com/hashicorp/nomad/client.(*Client).addAlloc
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1896 +0x2b1

goroutine 299 [select]:
github.com/hashicorp/nomad/client/driver.(*DockerHandle).collectStats(0xc0423c4480)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1354 +0x7a2
created by github.com/hashicorp/nomad/client/driver.(*DockerDriver).Start
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:555 +0x1052

goroutine 284 [semacquire]:
sync.runtime_notifyListWait(0xc04221c580, 0xc000000001)
/opt/go/src/runtime/sema.go:297 +0x119
sync.(*Cond).Wait(0xc04221c570)
/opt/go/src/sync/cond.go:57 +0x90
io.(*pipe).read(0xc04221c540, 0xc042563000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/io/pipe.go:47 +0x10b
io.(*PipeReader).Read(0xc042004140, 0xc042563000, 0x1000, 0x1000, 0x1e0, 0xc04221ed00, 0x76)
/opt/go/src/io/pipe.go:130 +0x53
bufio.(*Reader).fill(0xc0422b1f08)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).ReadSlice(0xc0422b1f08, 0x42c60a, 0x12c8250, 0xc0422b1d80, 0x0, 0xc04203b380, 0x12c8250)
/opt/go/src/bufio/bufio.go:338 +0xc2
bufio.(*Reader).ReadBytes(0xc0422b1f08, 0x65680a, 0xc0420103c0, 0x2, 0xc042145180, 0x62, 0x0)
/opt/go/src/bufio/bufio.go:416 +0x6d
bufio.(*Reader).ReadString(0xc0422b1f08, 0xa, 0xc0422b1ee8, 0x2, 0x2, 0x0)
/opt/go/src/bufio/bufio.go:456 +0x3f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).logStderr(0xc04200c600, 0x184bb80, 0xc0420041
40)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:526 +0xaf
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:394 +0xc7a

goroutine 159 [select]:
github.com/hashicorp/nomad/client.(*TaskRunner).run(0xc0424f2f20)
/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:907 +0x11e1
github.com/hashicorp/nomad/client.(*TaskRunner).Run(0xc0424f2f20)
/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:442 +0x55d
created by github.com/hashicorp/nomad/client.(*AllocRunner).Run
/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:519 +0x8e9

goroutine 280 [semacquire]:
sync.runtime_notifyListWait(0xc0423004c0, 0xc000000001)
/opt/go/src/runtime/sema.go:297 +0x119
sync.(*Cond).Wait(0xc0423004b0)
/opt/go/src/sync/cond.go:57 +0x90
io.(*pipe).read(0xc042300480, 0xc0425ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/io/pipe.go:47 +0x10b
io.(*PipeReader).Read(0xc0420f6620, 0xc0425ae000, 0x1000, 0x1000, 0x18, 0xc042036000, 0x25716d0)
/opt/go/src/io/pipe.go:130 +0x53
bufio.(*Reader).fill(0xc042033f10)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).ReadSlice(0xc042033f10, 0xc042033e0a, 0xc042033e38, 0xc042334040, 0x18, 0x0, 0x0)
/opt/go/src/bufio/bufio.go:338 +0xc2
bufio.(*Reader).ReadBytes(0xc042033f10, 0xc0421ed60a, 0xc042033ef8, 0x18, 0x18, 0x0, 0x0)
/opt/go/src/bufio/bufio.go:416 +0x6d
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func4(0xc0421ed6e0, 0xc0420f6620)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:404 +0xde
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:413 +0xcca

goroutine 303 [select]:
net/http.(*persistConn).roundTrip(0xc04231c000, 0xc04222f500, 0x0, 0x0, 0x0)
/opt/go/src/net/http/transport.go:1898 +0x97b
net/http.(*Transport).RoundTrip(0xc04220f590, 0xc04214ea00, 0xc04220f590, 0x0, 0x0)
/opt/go/src/net/http/transport.go:391 +0x753
net/http.send(0xc04214ea00, 0x184c040, 0xc04220f590, 0x0, 0xc000000000, 0x0, 0x0, 0x8, 0xc0420f6028, 0x4107ef)
/opt/go/src/net/http/client.go:249 +0x169
net/http.(*Client).send(0xc0422f5fb0, 0xc04214ea00, 0x0, 0x0, 0x0, 0xc0420f6028, 0x0, 0x1, 0x100)
/opt/go/src/net/http/client.go:173 +0x10f
net/http.(*Client).Do(0xc0422f5fb0, 0xc04214ea00, 0xc0420084d8, 0xc04214ea00, 0xc0420084d8)
/opt/go/src/net/http/client.go:595 +0x25b
github.com/hashicorp/nomad/vendor/golang.org/x/net/context/ctxhttp.Do(0x2745900, 0xc0420084d8, 0xc0422f5fb0, 0xc04214e90
0, 0xc0420084d8, 0x0, 0xc04214e900)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:30 +0x99
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).do(0xc042224320, 0x126c022, 0x4, 0xc0421ed
aa0, 0x51, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:478 +0x37f
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).waitContainer(0xc042224320, 0xc0424f83c0,
0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1272 +0x105
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).WaitContainer(0xc042224320, 0xc0424f83c0,
0x40, 0x0, 0x30004, 0xc04202dfc0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1259 +0x6c
github.com/hashicorp/nomad/client/driver.(*DockerHandle).run(0xc0423c4840)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1300 +0x60
created by github.com/hashicorp/nomad/client/driver.(*DockerDriver).Start
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:556 +0x1077

goroutine 285 [semacquire]:
sync.runtime_notifyListWait(0xc04221c4c0, 0xc000000001)
/opt/go/src/runtime/sema.go:297 +0x119
sync.(*Cond).Wait(0xc04221c4b0)
/opt/go/src/sync/cond.go:57 +0x90
io.(*pipe).read(0xc04221c480, 0xc042562000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/io/pipe.go:47 +0x10b
io.(*PipeReader).Read(0xc0420040b0, 0xc042562000, 0x1000, 0x1000, 0x18, 0xc04202bc00, 0x2570490)
/opt/go/src/io/pipe.go:130 +0x53
bufio.(*Reader).fill(0xc0423ddf10)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).ReadSlice(0xc0423ddf10, 0xc0423dde0a, 0xc0423dde38, 0xc04244caa0, 0x18, 0x0, 0x0)
/opt/go/src/bufio/bufio.go:338 +0xc2
bufio.(*Reader).ReadBytes(0xc0423ddf10, 0xc0421ed80a, 0xc0423ddef8, 0x18, 0x18, 0x0, 0x0)
/opt/go/src/bufio/bufio.go:416 +0x6d
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func4(0xc0421ed8c0, 0xc0420040b0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:404 +0xde
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:413 +0xcca

goroutine 283 [syscall, locked to thread]:
syscall.Syscall(0x7ffaad984360, 0x2, 0x5e0, 0xffffffff, 0x0, 0x2, 0xc042439f98, 0x2)
/opt/go/src/runtime/syscall_windows.go:163 +0x6b
syscall.WaitForSingleObject(0x5e0, 0xffffffff, 0xc04203b860, 0xc042439e58, 0x449aa1)
/opt/go/src/syscall/zsyscall_windows.go:718 +0x6d
os.(*Process).wait(0xc042168300, 0x0, 0x0, 0x0)
/opt/go/src/os/exec_windows.go:18 +0xa0
os.(*Process).Wait(0xc042168300, 0xc000000018, 0xc042439ed0, 0xc042439f9e)
/opt/go/src/os/exec.go:115 +0x32
os/exec.(*Cmd).Wait(0xc042072b00, 0x12c7870, 0xc042004130)
/opt/go/src/os/exec/exec.go:435 +0x69
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func3(0xc042004148, 0xc042004130, 0xc04
2072b00, 0xc0421ed860, 0xc04200c600)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:378 +0x8a
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:391 +0xc3c

goroutine 160 [IO wait]:
net.runtime_pollWait(0x47e0eb0, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0421488a0, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc042148760, 0x126fe30, 0x7, 0x12c78e8, 0xc042236bd0, 0x184bb40, 0xc04244c2c0)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc042148700, 0xc04223d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc0424b6328, 0xc04223d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:181 +0x77
bufio.(*Reader).Read(0xc0423e6a80, 0xc0424c0010, 0xc, 0xc, 0x10, 0xc0421aa770, 0x0)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc0423e6a80, 0xc0424c0010, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc0423e6a80, 0xc0424c0010, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:325 +0x5f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc04275a000, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:418 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc04275a000)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:406 +0x32
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x467

goroutine 289 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc04275a000)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:368 +0x468
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x489

goroutine 228 [select]:
github.com/hashicorp/nomad/client.(*AllocRunner).dirtySyncState(0xc0423ec3c0)
/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:332 +0x115
created by github.com/hashicorp/nomad/client.(*AllocRunner).Run
/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:459 +0x92

goroutine 62 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).keepalive(0xc04250a000)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:302 +0x1e3
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:111 +0x4d8

goroutine 282 [syscall, locked to thread]:
syscall.Syscall6(0x7ffaad984750, 0x5, 0x270, 0xc042488000, 0x8000, 0xc042031dac, 0x0, 0x0, 0x8, 0x3, ...)
/opt/go/src/runtime/syscall_windows.go:174 +0x6b
syscall.ReadFile(0x270, 0xc042488000, 0x8000, 0x8000, 0xc042031dac, 0x0, 0xc042031dc8, 0xc042031dc0)
/opt/go/src/syscall/zsyscall_windows.go:311 +0xe6
syscall.Read(0x270, 0xc042488000, 0x8000, 0x8000, 0xc042031e18, 0x463cb7, 0xc04221c548)
/opt/go/src/syscall/syscall_windows.go:296 +0x6f
os.(*File).read(0xc0420f6688, 0xc042488000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/opt/go/src/os/file_windows.go:289 +0xd0
os.(*File).Read(0xc0420f6688, 0xc042488000, 0x8000, 0x8000, 0x48, 0x0, 0x0)
/opt/go/src/os/file.go:101 +0x7d
io.copyBuffer(0x184bbc0, 0xc042004148, 0x184c640, 0xc0420f6688, 0xc042488000, 0x8000, 0x8000, 0x180001, 0x0, 0x0)
/opt/go/src/io/io.go:390 +0x107
io.Copy(0x184bbc0, 0xc042004148, 0x184c640, 0xc0420f6688, 0x0, 0xc04203cd20, 0x6dc44a)
/opt/go/src/io/io.go:360 +0x6f
os/exec.(*Cmd).writerDescriptor.func1(0x0, 0x0)
/opt/go/src/os/exec/exec.go:254 +0x54
os/exec.(*Cmd).Start.func1(0xc042072b00, 0xc04222e840)
/opt/go/src/os/exec/exec.go:371 +0x2e
created by os/exec.(*Cmd).Start
/opt/go/src/os/exec/exec.go:372 +0x55e

goroutine 281 [syscall, locked to thread]:
syscall.Syscall6(0x7ffaad984750, 0x5, 0x5bc, 0xc042490000, 0x8000, 0xc042175dac, 0x0, 0x0, 0x4, 0x3, ...)
/opt/go/src/runtime/syscall_windows.go:174 +0x6b
syscall.ReadFile(0x5bc, 0xc042490000, 0x8000, 0x8000, 0xc042175dac, 0x0, 0xc042175dc8, 0xc042175dc0)
/opt/go/src/syscall/zsyscall_windows.go:311 +0xe6
syscall.Read(0x5bc, 0xc042490000, 0x8000, 0x8000, 0xc042175e18, 0x463cb7, 0xc04221c488)
/opt/go/src/syscall/syscall_windows.go:296 +0x6f
os.(*File).read(0xc0420f6670, 0xc042490000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/opt/go/src/os/file_windows.go:289 +0xd0
os.(*File).Read(0xc0420f6670, 0xc042490000, 0x8000, 0x8000, 0x18, 0x0, 0x0)
/opt/go/src/os/file.go:101 +0x7d
io.copyBuffer(0x184bbc0, 0xc042004130, 0x184c640, 0xc0420f6670, 0xc042490000, 0x8000, 0x8000, 0x0, 0x0, 0x1)
/opt/go/src/io/io.go:390 +0x107
io.Copy(0x184bbc0, 0xc042004130, 0x184c640, 0xc0420f6670, 0xc042175fc0, 0xc042175f30, 0xc04203ccc0)
/opt/go/src/io/io.go:360 +0x6f
os/exec.(*Cmd).writerDescriptor.func1(0xc04203cd20, 0x6dd861)
/opt/go/src/os/exec/exec.go:254 +0x54
os/exec.(*Cmd).Start.func1(0xc042072b00, 0xc04222e7e0)
/opt/go/src/os/exec/exec.go:371 +0x2e
created by os/exec.(*Cmd).Start
/opt/go/src/os/exec/exec.go:372 +0x55e

goroutine 260 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc0421dc410, 0xc042760000, 0x2000, 0x2000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
io/ioutil.devNull.ReadFrom(0x0, 0x184b880, 0xc0421dc410, 0x10f82a0, 0xc042480601, 0x4b100c8)
/opt/go/src/io/ioutil/ioutil.go:144 +0x8c
io/ioutil.(*devNull).ReadFrom(0x18b4e08, 0x184b880, 0xc0421dc410, 0x4b100c8, 0x18b4e08, 0x449601)
:6 +0x68
io.copyBuffer(0x184d640, 0x18b4e08, 0x184b880, 0xc0421dc410, 0x0, 0x0, 0x0, 0x0, 0x30001, 0xc0424f5440)
/opt/go/src/io/io.go:384 +0x2d2
io.Copy(0x184d640, 0x18b4e08, 0x184b880, 0xc0421dc410, 0x180001, 0x0, 0x0)
/opt/go/src/io/io.go:360 +0x6f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x126f31f, 0x6, 0x184d640, 0x18b4e08, 0x184b
880, 0xc0421dc410)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x80
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCClient).SyncStreams
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:71 +0xa4

goroutine 108 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).keepalive(0xc04212a210)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:302 +0x1e3
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:111 +0x4d8

goroutine 259 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc0420416c0, 0xc04275e000, 0x1000, 0x1000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
bufio.(*Reader).Read(0xc0424f5ec0, 0xc0422ade00, 0x1, 0x9, 0xc042015300, 0x10000c0424809c0, 0xc0420ee3c0)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc0424f5ec0, 0xc0422ade00, 0x1, 0x9, 0x1, 0xc0422f3d00, 0x42c34d, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc0424f5ec0, 0xc0422ade00, 0x1, 0x9, 0xc04203b520, 0x4, 0x5)
/opt/go/src/io/io.go:325 +0x5f
encoding/gob.decodeUintReader(0x184a080, 0xc0424f5ec0, 0xc0422ade00, 0x9, 0x9, 0xc0422f3d48, 0x42bc75, 0xc000000008, 0xc
0424f8080)
/opt/go/src/encoding/gob/decode.go:119 +0x6a
encoding/gob.(*Decoder).recvMessage(0xc0421f6300, 0xc0422f3d58)
/opt/go/src/encoding/gob/decoder.go:76 +0x5e
encoding/gob.(*Decoder).decodeTypeSequence(0xc0421f6300, 0x12c8200, 0xc0421f6300)
/opt/go/src/encoding/gob/decoder.go:140 +0x16e
encoding/gob.(*Decoder).DecodeValue(0xc0421f6300, 0x1036340, 0xc04229f1d0, 0x16, 0x0, 0x0)
/opt/go/src/encoding/gob/decoder.go:208 +0xef
encoding/gob.(*Decoder).Decode(0xc0421f6300, 0x1036340, 0xc04229f1d0, 0x0, 0x0)
/opt/go/src/encoding/gob/decoder.go:185 +0x174
net/rpc.(*gobClientCodec).ReadResponseHeader(0xc0424405a0, 0xc04229f1d0, 0xc0422ade0c, 0x0)
/opt/go/src/net/rpc/client.go:223 +0x4c
net/rpc.(*Client).input(0xc042106000)
/opt/go/src/net/rpc/client.go:109 +0xd0
created by net/rpc.NewClientWithCodec
/opt/go/src/net/rpc/client.go:201 +0xcc

goroutine 107 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc04212a210)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:368 +0x468
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x489

goroutine 106 [IO wait]:
net.runtime_pollWait(0x47e0a30, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0421ae8a0, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc0421ae760, 0x126fe30, 0x7, 0x12c78e8, 0xc, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc0421ae700, 0xc04213e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc0421ea0c0, 0xc04213e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:181 +0x77
bufio.(*Reader).Read(0xc042106d80, 0xc0424c0720, 0xc, 0xc, 0xc0423e1ea0, 0xb1bbf9, 0x10)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc042106d80, 0xc0424c0720, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc042106d80, 0xc0424c0720, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:325 +0x5f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc04212a210, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:418 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc04212a210)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:406 +0x32
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x467

goroutine 276 [syscall, locked to thread]:
syscall.Syscall6(0x7ffaad984750, 0x5, 0x5d0, 0xc042554000, 0x8000, 0xc042425dac, 0x0, 0x0, 0x3, 0x3, ...)
/opt/go/src/runtime/syscall_windows.go:174 +0x6b
syscall.ReadFile(0x5d0, 0xc042554000, 0x8000, 0x8000, 0xc042425dac, 0x0, 0xc042425dc8, 0xc042425dc0)
/opt/go/src/syscall/zsyscall_windows.go:311 +0xe6
syscall.Read(0x5d0, 0xc042554000, 0x8000, 0x8000, 0xc042425e18, 0x463cb7, 0xc042300488)
/opt/go/src/syscall/syscall_windows.go:296 +0x6f
os.(*File).read(0xc0420f6648, 0xc042554000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/opt/go/src/os/file_windows.go:289 +0xd0
os.(*File).Read(0xc0420f6648, 0xc042554000, 0x8000, 0x8000, 0x18, 0x0, 0x0)
/opt/go/src/os/file.go:101 +0x7d
io.copyBuffer(0x184bbc0, 0xc0420f6628, 0x184c640, 0xc0420f6648, 0xc042554000, 0x8000, 0x8000, 0xc04230c201, 0x6e3c06, 0x
180001)
/opt/go/src/io/io.go:390 +0x107
io.Copy(0x184bbc0, 0xc0420f6628, 0x184c640, 0xc0420f6648, 0xc0421d4640, 0x4, 0xc04235c090)
/opt/go/src/io/io.go:360 +0x6f
os/exec.(*Cmd).writerDescriptor.func1(0x434d8f, 0x12c7f78)
/opt/go/src/os/exec/exec.go:254 +0x54
os/exec.(*Cmd).Start.func1(0xc0421f3080, 0xc04222e660)
/opt/go/src/os/exec/exec.go:371 +0x2e
created by os/exec.(*Cmd).Start
/opt/go/src/os/exec/exec.go:372 +0x55e

goroutine 61 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc04250a000)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:368 +0x468
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x489

goroutine 60 [IO wait]:
net.runtime_pollWait(0x47e08b0, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0421e4360, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc0421e4220, 0x126fe30, 0x7, 0x12c78e8, 0xc, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc0421e41c0, 0xc04243d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc0420f6090, 0xc04243d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:181 +0x77
bufio.(*Reader).Read(0xc0422353e0, 0xc0424e45b0, 0xc, 0xc, 0xc04242bea0, 0xb1bbf9, 0x10)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc0422353e0, 0xc0424e45b0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc0422353e0, 0xc0424e45b0, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:325 +0x5f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc04250a000, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:418 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc04250a000)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:406 +0x32
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x467

goroutine 120 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc0423ae780, 0x126b7d5, 0x3, 0x27458d0, 0xc0420f0cd0, 0
x37e11d600)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:853 +0x396
created by github.com/hashicorp/nomad/client.(*Client).setupDrivers
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:905 +0x685

goroutine 222 [select]:
github.com/hashicorp/nomad/client.(*AllocRunner).Run(0xc0423ec3c0)
/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:530 +0x741
created by github.com/hashicorp/nomad/client.(*Client).addAlloc
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1896 +0x2b1

goroutine 261 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc0421dc4e0, 0xc042764000, 0x2000, 0x2000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
io/ioutil.devNull.ReadFrom(0x0, 0x184b880, 0xc0421dc4e0, 0x10f82a0, 0xc0421f0301, 0x4b100c8)
/opt/go/src/io/ioutil/ioutil.go:144 +0x8c
io/ioutil.(*devNull).ReadFrom(0x18b4e08, 0x184b880, 0xc0421dc4e0, 0x4b100c8, 0x18b4e08, 0xc0422ebe01)
:6 +0x68
io.copyBuffer(0x184d640, 0x18b4e08, 0x184b880, 0xc0421dc4e0, 0x0, 0x0, 0x0, 0x0, 0x30001, 0xc0422ebf28)
/opt/go/src/io/io.go:384 +0x2d2
io.Copy(0x184d640, 0x18b4e08, 0x184b880, 0xc0421dc4e0, 0x180001, 0xc0422ebfa8, 0x6e3c06)
/opt/go/src/io/io.go:360 +0x6f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x126f319, 0x6, 0x184d640, 0x18b4e08, 0x184b
880, 0xc0421dc4e0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x80
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCClient).SyncStreams
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:72 +0x120

goroutine 302 [select]:
github.com/hashicorp/nomad/client/driver.(*DockerHandle).collectStats(0xc0423c4840)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1354 +0x7a2
created by github.com/hashicorp/nomad/client/driver.(*DockerDriver).Start
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:555 +0x1052

goroutine 126 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc0423ae780, 0x126e9d7, 0x6, 0x2745940, 0xc042145b90, 0
x37e11d600)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:853 +0x396
created by github.com/hashicorp/nomad/client.(*Client).setupDrivers
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:905 +0x685

goroutine 75 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc0421dc680, 0xc0420c5000, 0x1000, 0x1000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
bufio.(*Reader).fill(0xc042107260)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).ReadByte(0xc042107260, 0xc0421dc6a0, 0xc042017900, 0x0)
/opt/go/src/bufio/bufio.go:239 +0x62
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*ioDecReader).readn1(0xc0424449f0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:90 +0x81
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*msgpackDecDriver).initReadNext(0xc04244d080)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/msgpack.go:540 +0x4c
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*Decoder).decode(0xc0421072c0, 0x1036340, 0xc04
22f52f0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:635 +0x50
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec.(*Decoder).Decode(0xc0421072c0, 0x1036340, 0xc04
22f52f0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-msgpack/codec/decode.go:630 +0x8b
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.(*MsgpackCodec).read(0xc0424f9600, 0x1036340,
0xc0422f52f0, 0x1, 0xc0422f52f0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/codec.go:121 +0x5e
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.(*MsgpackCodec).ReadResponseHeader(0xc0424f960
0, 0xc0422f52f0, 0x117ac80, 0xc04203d320)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/codec.go:74 +0x48
github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc.CallWithCodec(0x1855780, 0xc0424f9600, 0x1285c
7f, 0x14, 0x117ac80, 0xc04203d320, 0x1117c40, 0xc042126720, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/net-rpc-msgpackrpc/client.go:29 +0x125
github.com/hashicorp/nomad/nomad.(*ConnPool).RPC(0xc0423b2500, 0x126eb39, 0x6, 0x18503c0, 0xc042441a40, 0x1, 0x1285c7f,
0x14, 0x117ac80, 0xc04203d320, ...)
/opt/gopath/src/github.com/hashicorp/nomad/nomad/pool.go:363 +0x1af
github.com/hashicorp/nomad/client.(*Client).RPC(0xc0423ae780, 0x1285c7f, 0x14, 0x117ac80, 0xc04203d320, 0x1117c40, 0xc04
2126720, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:452 +0x187
github.com/hashicorp/nomad/client.(*Client).watchAllocations(0xc0423ae780, 0xc04203d2c0)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1287 +0x2c3
created by github.com/hashicorp/nomad/client.(*Client).run
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1002 +0x83

goroutine 127 [select]:
github.com/hashicorp/nomad/client.(*Client).fingerprintPeriodic(0xc0423ae780, 0x126c3ce, 0x4, 0x2745970, 0xc0420f1a90, 0
x37e11d600)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:853 +0x396
created by github.com/hashicorp/nomad/client.(*Client).setupDrivers
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:905 +0x685

goroutine 128 [select, 1 minutes]:
github.com/hashicorp/nomad/client.(*Client).consulDiscovery(0xc0423ae780)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:2027 +0x198
created by github.com/hashicorp/nomad/client.NewClient
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:281 +0xcca

goroutine 197 [chan receive]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func5.1(0xc0421ed6e0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:419 +0x63
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func5
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:421 +0x46

goroutine 263 [chan receive]:
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc042410730, 0xc04255f830, 0x0, 0x
0, 0x0, 0xc000000000, 0x184e140, 0x18485a0, 0x4f9, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:167 +0x24d
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc042410730, 0xc042140000, 0x1000, 0x
1000, 0x4f9, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:190 +0xf1
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc04222f420, 0xc042140000,
0x1000, 0x1000, 0x4f9, 0x0, 0x54ff09)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/pipe.go:116 +0x63
bufio.(*Reader).fill(0xc0421ed7a0)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).ReadSlice(0xc0421ed7a0, 0xc0421ed70a, 0xc04229e8c0, 0x2, 0x2, 0x2, 0x0)
/opt/go/src/bufio/bufio.go:338 +0xc2
net/http/internal.readChunkLine(0xc0421ed7a0, 0x42c34d, 0xc0421ed7a0, 0x42f1df, 0x12c80d0, 0x2)
/opt/go/src/net/http/internal/chunked.go:122 +0x3b
net/http/internal.(*chunkedReader).beginChunk(0xc04229e8a0)
/opt/go/src/net/http/internal/chunked.go:48 +0x39
net/http/internal.(*chunkedReader).Read(0xc04229e8a0, 0xc0427a4000, 0x8000, 0x8000, 0xc0424aa000, 0x463cb7, 0x6d1d74)
/opt/go/src/net/http/internal/chunked.go:93 +0x12f
net/http.(*body).readLocked(0xc0423cca40, 0xc0427a4000, 0x8000, 0x8000, 0x8000, 0x4f2, 0x0)
/opt/go/src/net/http/transfer.go:761 +0x68
net/http.(*body).Read(0xc0423cca40, 0xc0427a4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/opt/go/src/net/http/transfer.go:753 +0x104
io.copyBuffer(0x184bbc0, 0xc042004450, 0x4b50b00, 0xc0423cca40, 0xc0427a4000, 0x8000, 0x8000, 0x118c000, 0x1000000000000
00, 0x4b50b00)
/opt/go/src/io/io.go:390 +0x107
io.Copy(0x184bbc0, 0xc042004450, 0x4b50b00, 0xc0423cca40, 0xc0423cca40, 0xc0422b5c70, 0xc0422b5c70)
/opt/go/src/io/io.go:360 +0x6f
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.handleStreamResponse(0xc0423d75f0, 0xc0422b5e58, 0xc
0423cca40, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:630 +0x1fe
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).stream(0xc042224320, 0x126b328, 0x3, 0xc04
21ed500, 0x5e, 0x10100, 0x0, 0x0, 0x0, 0x184bbc0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:607 +0x73b
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats.func2(0xc042224320, 0xc0424f87c0, 0x
40, 0xc0423a61e0, 0x1, 0xc0423e69c0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1062 +0x1ea
created by github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1076 +0x232

goroutine 130 [select]:
github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat(0xc0423ae780)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:947 +0x434
created by github.com/hashicorp/nomad/client.NewClient
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:302 +0xb4e

goroutine 131 [select]:
github.com/hashicorp/nomad/client.(*Client).periodicSnapshot(0xc0423ae780)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:985 +0x1c8
created by github.com/hashicorp/nomad/client.NewClient
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:305 +0xb70

goroutine 132 [runnable]:
github.com/hashicorp/nomad/client.(*Client).allocSync(0xc0423ae780)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1202 +0x70d
created by github.com/hashicorp/nomad/client.NewClient
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:308 +0xb92

goroutine 133 [select]:
github.com/hashicorp/nomad/client.(*Client).run(0xc0423ae780)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1005 +0x16d
created by github.com/hashicorp/nomad/client.NewClient
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:311 +0xbb4

goroutine 134 [select]:
github.com/hashicorp/nomad/client.(*Client).emitStats(0xc0423ae780)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:2197 +0x26f
created by github.com/hashicorp/nomad/client.NewClient
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:314 +0xbd6

goroutine 135 [select]:
github.com/hashicorp/nomad/command/agent/consul.(*Syncer).Run(0xc0423f85b0)
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/consul/syncer.go:819 +0x365
created by github.com/hashicorp/nomad/command/agent.NewAgent
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:86 +0x259

goroutine 158 [select]:
github.com/hashicorp/nomad/client.(*AllocRunner).dirtySyncState(0xc0423ec2d0)
/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:332 +0x115
created by github.com/hashicorp/nomad/client.(*AllocRunner).Run
/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:459 +0x92

goroutine 264 [select]:
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats.func3(0xc0424f87c0, 0x40, 0xc0423a61
e0, 0x1, 0xc0423e69c0, 0x0, 0x0, 0x0, 0x0, 0xc042004448, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1082 +0xf2
created by github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1088 +0x2df

goroutine 148 [IO wait]:
net.runtime_pollWait(0x47e0c70, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc042148de0, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc042148ca0, 0x126fe30, 0x7, 0x12c78e8, 0x1, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc042148c40, 0xc0421b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc0424b77d8, 0xc0421b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:181 +0x77
net/http.(*persistConn).Read(0xc04216d0e0, 0xc0421b2000, 0x1000, 0x1000, 0x4, 0x454540, 0xc04203d3e0)
/opt/go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc0423f0300)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc0423f0300, 0x1, 0xc0424cfbe5, 0x1, 0x0, 0xc0421062a0, 0x0)
/opt/go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc04216d0e0)
/opt/go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1117 +0xa3c

goroutine 149 [select]:
net/http.(*persistConn).writeLoop(0xc04216d0e0)
/opt/go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1118 +0xa61

goroutine 136 [IO wait]:
net.runtime_pollWait(0x47e0bb0, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0421ae1a0, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc0421ae060, 0x12712d4, 0x8, 0xc04244dc60, 0xc0421e48c0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).acceptOne(0xc0421ae000, 0xc042013ea0, 0x2, 0x2, 0xc0421ae060, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:591 +0x1ae
net.(*netFD).accept(0xc0421ae000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:621 +0x10a
net.(*TCPListener).accept(0xc042412010, 0xc0421bbe90, 0x6cb6b5, 0x456fa0)
/opt/go/src/net/tcpsock_posix.go:136 +0x35
net.(*TCPListener).Accept(0xc042412010, 0x12c7a80, 0xc04210b5e0, 0x1855200, 0xc0421de0c0)
/opt/go/src/net/tcpsock.go:228 +0x50
net/http.(*Server).Serve(0xc0420702c0, 0x1853b00, 0xc042412010, 0x0, 0x0)
/opt/go/src/net/http/server.go:2643 +0x22f
net/http.Serve(0x1853b00, 0xc042412010, 0x184d780, 0xc0423e26e0, 0x1853b00, 0xc042412010)
/opt/go/src/net/http/server.go:2307 +0x7f
created by github.com/hashicorp/nomad/command/agent.NewHTTPServer
/opt/gopath/src/github.com/hashicorp/nomad/command/agent/http.go:97 +0x389

goroutine 137 [select, 1 minutes]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-checkpoint.CheckInterval.func1(0x4e94914f0000, 0xc0424ea180, 0
xc042464390, 0xc0423e6240)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-checkpoint/checkpoint.go:197 +0x14e
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-checkpoint.CheckInterval
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-checkpoint/checkpoint.go:205 +0xc2

goroutine 258 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).AcceptStream(0xc04275a000, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:201 +0x1c2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*MuxBroker).Run(0xc042197b20)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/mux_broker.go:139 +0x3a
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.NewRPCClient
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:51 +0x232

goroutine 177 [IO wait]:
net.runtime_pollWait(0x47e0af0, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0421d61a0, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc0421d6060, 0x126fe30, 0x7, 0x12c78e8, 0x10, 0x1, 0x2)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc0421d6000, 0xc0421ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc0421ea000, 0xc0421ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:181 +0x77
net/http.(*persistConn).Read(0xc0421b6000, 0xc0421ee000, 0x1000, 0x1000, 0x4, 0x454540, 0xc0424f45a0)
/opt/go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc0421ec000)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc0421ec000, 0x1, 0xc0421cdbe5, 0x1, 0x0, 0xc0421068a0, 0x0)
/opt/go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc0421b6000)
/opt/go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1117 +0xa3c

goroutine 178 [select]:
net/http.(*persistConn).writeLoop(0xc0421b6000)
/opt/go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1118 +0xa61

goroutine 304 [runnable]:
github.com/hashicorp/nomad/client.(*TaskRunner).collectResourceUsageStats(0xc0424f2f20, 0xc0423e7a40)
/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:1264 +0x50a
created by github.com/hashicorp/nomad/client.(*TaskRunner).run
/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:937 +0x487

goroutine 140 [select]:
github.com/hashicorp/nomad/client.(*Client).watchNodeUpdates(0xc0423ae780)
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:1430 +0x338
created by github.com/hashicorp/nomad/client.(*Client).registerAndHeartbeat
/opt/gopath/src/github.com/hashicorp/nomad/client/client.go:934 +0x6f

goroutine 166 [IO wait]:
net.runtime_pollWait(0x47e0970, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc0421d6360, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc0421d6220, 0x126fe30, 0x7, 0x12c78e8, 0xc, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc0421d61c0, 0xc042260000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc0421a8040, 0xc042260000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:181 +0x77
bufio.(*Reader).Read(0xc0424f4b40, 0xc0424e41f0, 0xc, 0xc, 0xc042243ea0, 0xb1bbf9, 0x10)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc0424f4b40, 0xc0424e41f0, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc0424f4b40, 0xc0424e41f0, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:325 +0x5f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc04225a210, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:418 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc04225a210)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:406 +0x32
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x467

goroutine 167 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc04225a210)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:368 +0x468
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x489

goroutine 168 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).keepalive(0xc04225a210)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:302 +0x1e3
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:111 +0x4d8

goroutine 242 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).AcceptStream(0xc04225a160, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:201 +0x1c2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*MuxBroker).Run(0xc0420fed00)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/mux_broker.go:139 +0x3a
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.NewRPCClient
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:51 +0x232

goroutine 290 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).keepalive(0xc04275a000)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:302 +0x1e3
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:111 +0x4d8

goroutine 229 [select]:
github.com/hashicorp/nomad/client.(*TaskRunner).run(0xc0423e8160)
/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:907 +0x11e1
github.com/hashicorp/nomad/client.(*TaskRunner).Run(0xc0423e8160)
/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:442 +0x55d
created by github.com/hashicorp/nomad/client.(*AllocRunner).Run
/opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:519 +0x8e9

goroutine 199 [IO wait]:
net.runtime_pollWait(0x47e0df0, 0x72, 0x18485a0)
/opt/go/src/runtime/netpoll.go:164 +0x60
net.(*pollDesc).wait(0xc04249a1a0, 0x72, 0x18485a0, 0x0)
/opt/go/src/net/fd_poll_runtime.go:75 +0x3f
net.(*ioSrv).ExecIO(0xc042100040, 0xc04249a060, 0x126fe30, 0x7, 0x12c78e8, 0xc042211e30, 0x184bb40, 0xc0423626a0)
/opt/go/src/net/fd_windows.go:196 +0xfe
net.(*netFD).Read(0xc04249a000, 0xc04249c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/fd_windows.go:452 +0x13d
net.(*conn).Read(0xc042412000, 0xc04249c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/opt/go/src/net/net.go:181 +0x77
bufio.(*Reader).Read(0xc04203c0c0, 0xc0422add70, 0xc, 0xc, 0x10, 0xc0420093e0, 0x0)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc04203c0c0, 0xc0422add70, 0xc, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc04203c0c0, 0xc0422add70, 0xc, 0xc, 0xc, 0x0, 0x0)
/opt/go/src/io/io.go:325 +0x5f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recvLoop(0xc04225a160, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:418 +0xf2
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).recv(0xc04225a160)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:406 +0x32
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:108 +0x467

goroutine 200 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).send(0xc04225a160)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:368 +0x468
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:109 +0x489

goroutine 201 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Session).keepalive(0xc04225a160)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:302 +0x1e3
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.newSession
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/session.go:111 +0x4d8

goroutine 230 [chan receive]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func5.1(0xc0421ed8c0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:419 +0x63
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*Client).Start.func5
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/client.go:421 +0x46

goroutine 232 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc042103d40, 0xc042205000, 0x1000, 0x1000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
bufio.(*Reader).Read(0xc04230dc80, 0xc04240b7d0, 0x1, 0x9, 0xc042015300, 0xc04257c1a0, 0xc0421d97c0)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc04230dc80, 0xc04240b7d0, 0x1, 0x9, 0x1, 0xc042527d00, 0x42c34d, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc04230dc80, 0xc04240b7d0, 0x1, 0x9, 0xc04203b520, 0x4, 0x7)
/opt/go/src/io/io.go:325 +0x5f
encoding/gob.decodeUintReader(0x184a080, 0xc04230dc80, 0xc04240b7d0, 0x9, 0x9, 0xc042527d48, 0x42bc75, 0xc000000008, 0xc
0424f8780)
/opt/go/src/encoding/gob/decode.go:119 +0x6a
encoding/gob.(*Decoder).recvMessage(0xc0424ea480, 0xc042527d58)
/opt/go/src/encoding/gob/decoder.go:76 +0x5e
encoding/gob.(*Decoder).decodeTypeSequence(0xc0424ea480, 0x12c8200, 0xc0424ea480)
/opt/go/src/encoding/gob/decoder.go:140 +0x16e
encoding/gob.(*Decoder).DecodeValue(0xc0424ea480, 0x1036340, 0xc0422f5560, 0x16, 0x0, 0x0)
/opt/go/src/encoding/gob/decoder.go:208 +0xef
encoding/gob.(*Decoder).Decode(0xc0424ea480, 0x1036340, 0xc0422f5560, 0x0, 0x0)
/opt/go/src/encoding/gob/decoder.go:185 +0x174
net/rpc.(*gobClientCodec).ReadResponseHeader(0xc0420fcea0, 0xc0422f5560, 0xc0420099d0, 0x0)
/opt/go/src/net/rpc/client.go:223 +0x4c
net/rpc.(*Client).input(0xc04230dd40)
/opt/go/src/net/rpc/client.go:109 +0xd0
created by net/rpc.NewClientWithCodec
/opt/go/src/net/rpc/client.go:201 +0xcc

goroutine 243 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc04225c000, 0xc04257f000, 0x1000, 0x1000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
bufio.(*Reader).Read(0xc0423cf500, 0xc042333500, 0x1, 0x9, 0xc042015300, 0x10000c04246b380, 0xc0420ee3c0)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc0423cf500, 0xc042333500, 0x1, 0x9, 0x1, 0xc042523d00, 0x42c34d, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc0423cf500, 0xc042333500, 0x1, 0x9, 0xc0421f0340, 0x4, 0x11)
/opt/go/src/io/io.go:325 +0x5f
encoding/gob.decodeUintReader(0x184a080, 0xc0423cf500, 0xc042333500, 0x9, 0x9, 0xc042523d48, 0x42bc75, 0xc000000008, 0xc
0421d8680)
/opt/go/src/encoding/gob/decode.go:119 +0x6a
encoding/gob.(*Decoder).recvMessage(0xc0424fe200, 0xc042523d58)
/opt/go/src/encoding/gob/decoder.go:76 +0x5e
encoding/gob.(*Decoder).decodeTypeSequence(0xc0424fe200, 0x12c8200, 0xc0424fe200)
/opt/go/src/encoding/gob/decoder.go:140 +0x16e
encoding/gob.(*Decoder).DecodeValue(0xc0424fe200, 0x1036340, 0xc042440480, 0x16, 0x0, 0x0)
/opt/go/src/encoding/gob/decoder.go:208 +0xef
encoding/gob.(*Decoder).Decode(0xc0424fe200, 0x1036340, 0xc042440480, 0x0, 0x0)
/opt/go/src/encoding/gob/decoder.go:185 +0x174
net/rpc.(*gobClientCodec).ReadResponseHeader(0xc04219c6c0, 0xc042440480, 0xc04233350c, 0x0)
/opt/go/src/net/rpc/client.go:223 +0x4c
net/rpc.(*Client).input(0xc0423cf5c0)
/opt/go/src/net/rpc/client.go:109 +0xd0
created by net/rpc.NewClientWithCodec
/opt/go/src/net/rpc/client.go:201 +0xcc

goroutine 244 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc0423bb040, 0xc04258c000, 0x2000, 0x2000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
io/ioutil.devNull.ReadFrom(0x0, 0x184b880, 0xc0423bb040, 0x10f82a0, 0x421e42f600000001, 0x4b100c8)
/opt/go/src/io/ioutil/ioutil.go:144 +0x8c
io/ioutil.(*devNull).ReadFrom(0x18b4e08, 0x184b880, 0xc0423bb040, 0x4b100c8, 0x18b4e08, 0x411101)
:6 +0x68
io.copyBuffer(0x184d640, 0x18b4e08, 0x184b880, 0xc0423bb040, 0x0, 0x0, 0x0, 0x3183579d1e9668b0, 0x410, 0x82)
/opt/go/src/io/io.go:384 +0x2d2
io.Copy(0x184d640, 0x18b4e08, 0x184b880, 0xc0423bb040, 0x0, 0x0, 0xc0425861a8)
/opt/go/src/io/io.go:360 +0x6f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x126f31f, 0x6, 0x184d640, 0x18b4e08, 0x184b
880, 0xc0423bb040)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x80
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCClient).SyncStreams
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:71 +0xa4

goroutine 245 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc0423bb110, 0xc042750000, 0x2000, 0x2000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
io/ioutil.devNull.ReadFrom(0x0, 0x184b880, 0xc0423bb110, 0x10f82a0, 0xe3494e32869e3401, 0x4b100c8)
/opt/go/src/io/ioutil/ioutil.go:144 +0x8c
io/ioutil.(*devNull).ReadFrom(0x18b4e08, 0x184b880, 0xc0423bb110, 0x4b100c8, 0x18b4e08, 0x8c90dfb4dc94fb01)
:6 +0x68
io.copyBuffer(0x184d640, 0x18b4e08, 0x184b880, 0xc0423bb110, 0x0, 0x0, 0x0, 0xff64bdf4237af0a, 0x952794fc859e96bf, 0xb55
8f6a82f82e1a2)
/opt/go/src/io/io.go:384 +0x2d2
io.Copy(0x184d640, 0x18b4e08, 0x184b880, 0xc0423bb110, 0xc970a5fb09e1ac10, 0x75eaf00a97250b77, 0xfbdc15bae04baa95)
/opt/go/src/io/io.go:360 +0x6f
github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.copyStream(0x126f319, 0x6, 0x184d640, 0x18b4e08, 0x184b
880, 0xc0423bb110)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/stream.go:15 +0x80
created by github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin.(*RPCClient).SyncStreams
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/go-plugin/rpc_client.go:72 +0x120

goroutine 286 [select]:
github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux.(*Stream).Read(0xc0421dcf70, 0xc0421cb000, 0x1000, 0x1000,
0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/hashicorp/yamux/stream.go:125 +0x26f
bufio.(*Reader).Read(0xc04203ca80, 0xc042414b24, 0x1, 0x9, 0xc042014000, 0xc0424a6d00, 0xc04200db80)
/opt/go/src/bufio/bufio.go:213 +0x319
io.ReadAtLeast(0x184a080, 0xc04203ca80, 0xc042414b24, 0x1, 0x9, 0x1, 0xc04222dd00, 0x42c34d, 0x0)
/opt/go/src/io/io.go:307 +0xb0
io.ReadFull(0x184a080, 0xc04203ca80, 0xc042414b24, 0x1, 0x9, 0xc0421f0340, 0x4, 0x8)
/opt/go/src/io/io.go:325 +0x5f
encoding/gob.decodeUintReader(0x184a080, 0xc04203ca80, 0xc042414b24, 0x9, 0x9, 0xc04222dd48, 0x42bc75, 0xc000000008, 0xc
0424f8580)
/opt/go/src/encoding/gob/decode.go:119 +0x6a
encoding/gob.(*Decoder).recvMessage(0xc042110500, 0xc04222dd58)
/opt/go/src/encoding/gob/decoder.go:76 +0x5e
encoding/gob.(*Decoder).decodeTypeSequence(0xc042110500, 0x12c8200, 0xc042110500)
/opt/go/src/encoding/gob/decoder.go:140 +0x16e
encoding/gob.(*Decoder).DecodeValue(0xc042110500, 0x1036340, 0xc042339e00, 0x16, 0x0, 0x0)
/opt/go/src/encoding/gob/decoder.go:208 +0xef
encoding/gob.(*Decoder).Decode(0xc042110500, 0x1036340, 0xc042339e00, 0x0, 0x0)
/opt/go/src/encoding/gob/decoder.go:185 +0x174
net/rpc.(*gobClientCodec).ReadResponseHeader(0xc04219c930, 0xc042339e00, 0xc042008630, 0x0)
/opt/go/src/net/rpc/client.go:223 +0x4c
net/rpc.(*Client).input(0xc04203cb40)
/opt/go/src/net/rpc/client.go:109 +0xd0
created by net/rpc.NewClientWithCodec
/opt/go/src/net/rpc/client.go:201 +0xcc

goroutine 262 [semacquire]:
sync.runtime_notifyListWait(0xc04221c940, 0xc00000001a)
/opt/go/src/runtime/sema.go:297 +0x119
sync.(*Cond).Wait(0xc04221c930)
/opt/go/src/sync/cond.go:57 +0x90
io.(*pipe).read(0xc04221c900, 0xc04235f801, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
/opt/go/src/io/pipe.go:47 +0x10b
io.(*PipeReader).Read(0xc042004448, 0xc04235f801, 0x5ff, 0x5ff, 0xa000000000004, 0xa, 0x1d)
/opt/go/src/io/pipe.go:130 +0x53
encoding/json.(*Decoder).refill(0xc0421e4380, 0x30a, 0x9)
/opt/go/src/encoding/json/stream.go:152 +0x110
encoding/json.(*Decoder).readValue(0xc0421e4380, 0x0, 0x0, 0x4107ef)
/opt/go/src/encoding/json/stream.go:128 +0x1d1
encoding/json.(*Decoder).Decode(0xc0421e4380, 0x10195c0, 0xc042485b00, 0x0, 0x0)
/opt/go/src/encoding/json/stream.go:57 +0x8c
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats(0xc042224320, 0xc0424f87c0, 0x40, 0x
c0423a61e0, 0x1, 0xc0423e69c0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1092 +0x40b
github.com/hashicorp/nomad/client/driver.(*DockerHandle).collectStats.func1(0xc0423c4480, 0xc0424f87c0, 0x40, 0xc0423a61
e0, 0x1, 0xc0423e69c0, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1348 +0x78
created by github.com/hashicorp/nomad/client/driver.(*DockerHandle).collectStats
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1351 +0x125

goroutine 301 [runnable]:
github.com/hashicorp/nomad/client.(*TaskRunner).collectResourceUsageStats(0xc0423e8160, 0xc0423e73e0)
/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:1264 +0x50a
created by github.com/hashicorp/nomad/client.(*TaskRunner).run
/opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:937 +0x487

goroutine 300 [select]:
net/http.(*persistConn).roundTrip(0xc0424577a0, 0xc04244c240, 0x0, 0x0, 0x0)
/opt/go/src/net/http/transport.go:1898 +0x97b
net/http.(*Transport).RoundTrip(0xc04220f590, 0xc042038c00, 0xc04220f590, 0x0, 0x0)
/opt/go/src/net/http/transport.go:391 +0x753
net/http.send(0xc042038c00, 0x184c040, 0xc04220f590, 0x0, 0xc000000000, 0x0, 0x0, 0x8, 0xc0421ead18, 0x4107ef)
/opt/go/src/net/http/client.go:249 +0x169
net/http.(*Client).send(0xc0422f5fb0, 0xc042038c00, 0x0, 0x0, 0x0, 0xc0421ead18, 0x0, 0x1, 0x100)
/opt/go/src/net/http/client.go:173 +0x10f
net/http.(*Client).Do(0xc0422f5fb0, 0xc042038c00, 0xc0420084d8, 0xc042038c00, 0xc0420084d8)
/opt/go/src/net/http/client.go:595 +0x25b
github.com/hashicorp/nomad/vendor/golang.org/x/net/context/ctxhttp.Do(0x2745900, 0xc0420084d8, 0xc0422f5fb0, 0xc042038b0
0, 0xc0420084d8, 0x0, 0xc042038b00)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:30 +0x99
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).do(0xc042224320, 0x126c022, 0x4, 0xc04203d
740, 0x51, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:478 +0x37f
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).waitContainer(0xc042224320, 0xc0424f87c0,
0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1272 +0x105
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).WaitContainer(0xc042224320, 0xc0424f87c0,
0x40, 0xc0421ec5a0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1259 +0x6c
github.com/hashicorp/nomad/client/driver.(*DockerHandle).run(0xc0423c4480)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1300 +0x60
created by github.com/hashicorp/nomad/client/driver.(*DockerDriver).Start
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:556 +0x1077

goroutine 287 [chan receive]:
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).stream.func1(0x3f24910, 0xc042446c80, 0x18
5a340, 0xc04222f420)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:564 +0x5b
created by github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).stream
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:566 +0x427

goroutine 311 [select]:
net/http.(*persistConn).writeLoop(0xc0424577a0)
/opt/go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1118 +0xa61

goroutine 310 [chan receive]:
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc0424fa8c0, 0xc042398de0, 0x0, 0x
0, 0x0, 0xc000000000, 0x184e140, 0x18485a0, 0xc04278fb30, 0x6d6da4, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:167 +0x24d
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc0424fa8c0, 0xc0420c6000, 0x1000, 0x
1000, 0x0, 0x4567e0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:190 +0xf1
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc04244c280, 0xc0420c6000,
0x1000, 0x1000, 0x60, 0xc0421d9700, 0xc042015300)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/pipe.go:116 +0x63
net/http.(*persistConn).Read(0xc0424577a0, 0xc0420c6000, 0x1000, 0x1000, 0x12c8250, 0x0, 0xc04278fb80)
/opt/go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc04203db00)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc04203db00, 0x1, 0xc04203dda0, 0xc04278fc80, 0x0, 0xc0421ed320, 0x0)
/opt/go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc0424577a0)
/opt/go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1117 +0xa3c

goroutine 312 [semacquire]:
sync.runtime_notifyListWait(0xc04220a040, 0xc00000001a)
/opt/go/src/runtime/sema.go:297 +0x119
sync.(*Cond).Wait(0xc04220a030)
/opt/go/src/sync/cond.go:57 +0x90
io.(*pipe).read(0xc04220a000, 0xc04251a001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
/opt/go/src/io/pipe.go:47 +0x10b
io.(*PipeReader).Read(0xc0421ead20, 0xc04251a001, 0x5ff, 0x5ff, 0x2000000000004, 0xa, 0x1d)
/opt/go/src/io/pipe.go:130 +0x53
encoding/json.(*Decoder).refill(0xc0421ae1c0, 0x30a, 0x9)
/opt/go/src/encoding/json/stream.go:152 +0x110
encoding/json.(*Decoder).readValue(0xc0421ae1c0, 0x0, 0x0, 0x4107ef)
/opt/go/src/encoding/json/stream.go:128 +0x1d1
encoding/json.(*Decoder).Decode(0xc0421ae1c0, 0x10195c0, 0xc042516300, 0x0, 0x0)
/opt/go/src/encoding/json/stream.go:57 +0x8c
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats(0xc042224320, 0xc0424f83c0, 0x40, 0x
c04203dec0, 0x1, 0xc0423e7500, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1092 +0x40b
github.com/hashicorp/nomad/client/driver.(*DockerHandle).collectStats.func1(0xc0423c4840, 0xc0424f83c0, 0x40, 0xc04203de
c0, 0x1, 0xc0423e7500, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1348 +0x78
created by github.com/hashicorp/nomad/client/driver.(*DockerHandle).collectStats
/opt/gopath/src/github.com/hashicorp/nomad/client/driver/docker.go:1351 +0x125

goroutine 313 [chan receive]:
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc0424fac80, 0xc04217f710, 0x0, 0x
0, 0x0, 0xc000000000, 0x184e140, 0x18485a0, 0x4fc, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:167 +0x24d
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc0424fac80, 0xc04225e000, 0x1000, 0x
1000, 0x4fc, 0x0, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:190 +0xf1
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc04244c340, 0xc04225e000,
0x1000, 0x1000, 0x4fc, 0x0, 0x54ff09)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/pipe.go:116 +0x63
bufio.(*Reader).fill(0xc04243e120)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).ReadSlice(0xc04243e120, 0xc04243e10a, 0xc04219cfb0, 0x2, 0x2, 0x2, 0x0)
/opt/go/src/bufio/bufio.go:338 +0xc2
net/http/internal.readChunkLine(0xc04243e120, 0x42c34d, 0xc04243e120, 0x42f1df, 0x12c80d0, 0x2)
/opt/go/src/net/http/internal/chunked.go:122 +0x3b
net/http/internal.(*chunkedReader).beginChunk(0xc04219cf90)
/opt/go/src/net/http/internal/chunked.go:48 +0x39
net/http/internal.(*chunkedReader).Read(0xc04219cf90, 0xc0427b4000, 0x8000, 0x8000, 0xc0420fa000, 0x463cb7, 0x6d1d74)
/opt/go/src/net/http/internal/chunked.go:93 +0x12f
net/http.(*body).readLocked(0xc042446d40, 0xc0427b4000, 0x8000, 0x8000, 0x8000, 0x4f5, 0x0)
/opt/go/src/net/http/transfer.go:761 +0x68
net/http.(*body).Read(0xc042446d40, 0xc0427b4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/opt/go/src/net/http/transfer.go:753 +0x104
io.copyBuffer(0x184bbc0, 0xc0421ead28, 0x4b50b00, 0xc042446d40, 0xc0427b4000, 0x8000, 0x8000, 0x118c000, 0x1000000000000
00, 0x4b50b00)
/opt/go/src/io/io.go:390 +0x107
io.Copy(0x184bbc0, 0xc0421ead28, 0x4b50b00, 0xc042446d40, 0xc042446d40, 0xc042583c70, 0xc042583c70)
/opt/go/src/io/io.go:360 +0x6f
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.handleStreamResponse(0xc0424ba2d0, 0xc042583e58, 0xc
042446d40, 0x0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:630 +0x1fe
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).stream(0xc042224320, 0x126b328, 0x3, 0xc04
243e000, 0x5e, 0x10100, 0x0, 0x0, 0x0, 0x184bbc0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:607 +0x73b
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats.func2(0xc042224320, 0xc0424f83c0, 0x
40, 0xc04203dec0, 0x1, 0xc0423e7500, 0x0, 0x0, 0x0, 0x0, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1062 +0x1ea
created by github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1076 +0x232

goroutine 314 [select]:
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats.func3(0xc0424f83c0, 0x40, 0xc04203de
c0, 0x1, 0xc0423e7500, 0x0, 0x0, 0x0, 0x0, 0xc0421ead20, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1082 +0xf2
created by github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).Stats
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/container.go:1088 +0x2df

goroutine 315 [chan receive]:
github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).stream.func1(0x3f24910, 0xc0424f8900, 0x18
5a340, 0xc04244c340)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:564 +0x5b
created by github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient.(*Client).stream
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/fsouza/go-dockerclient/client.go:566 +0x427

goroutine 321 [chan receive]:
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc042410780, 0xc04229eff0, 0x0, 0x
0, 0x0, 0xc000000000, 0x184e140, 0x18485a0, 0x0, 0x40ff7c, ...)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:167 +0x24d
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc042410780, 0xc042272000, 0x1000, 0x
1000, 0x681e2f, 0xc04214ea00, 0x184c040)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/file.go:190 +0xf1
github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc04222f540, 0xc042272000,
0x1000, 0x1000, 0x60, 0xc04214e901, 0xc04222f4e0)
/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/Microsoft/go-winio/pipe.go:116 +0x63
net/http.(*persistConn).Read(0xc04231c000, 0xc042272000, 0x1000, 0x1000, 0x0, 0x0, 0xc04202db80)
/opt/go/src/net/http/transport.go:1316 +0x152
bufio.(*Reader).fill(0xc0421edda0)
/opt/go/src/bufio/bufio.go:97 +0x11e
bufio.(*Reader).Peek(0xc0421edda0, 0x1, 0xc042107920, 0xc04202dc80, 0xc04202dbe0, 0x4107ef, 0xc04214ea00)
/opt/go/src/bufio/bufio.go:129 +0x6e
net/http.(*persistConn).readLoop(0xc04231c000)
/opt/go/src/net/http/transport.go:1474 +0x19d
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1117 +0xa3c

goroutine 322 [select]:
net/http.(*persistConn).writeLoop(0xc04231c000)
/opt/go/src/net/http/transport.go:1704 +0x441
created by net/http.(*Transport).dialConn
/opt/go/src/net/http/transport.go:1118 +0xa61
PS C:>

@schmichael
Copy link
Member Author

windows_amd64.zip

The attached file is built with microsoft/go-winio#31 but sadly still panics. I've commented on the PR and will try to keep testing any updates on their end.

@cvandal
Copy link

cvandal commented Apr 28, 2017

hey @schmichael, I tried building my own version of 0.5.4 with the fix for #2193 but it still threw the same error. my go skills are quite limited. would it be possible to get a binary for 0.5.4 with only the windows memoryswap fix?

@schmichael
Copy link
Member Author

@cvandal Aha! You reminded me that a user suggested the go-winio bug is due to changes in Go 1.8.

I've cherry picked 8c35388 onto v0.5.4, pushed the branch, and built the following binaries using Go 1.7.5:

linux_amd64.zip
windows_386.zip
windows_amd64.zip

@Evertras
Copy link

Assuming this fixes the panic on Windows, do you foresee any issues running this in conjunction with a 0.5.6 cluster?

@cvandal
Copy link

cvandal commented Apr 30, 2017

@Evertras I have been running my Nomad Servers on 0.5.6 since release, and my clients on 0.5.4 without any issues so far.

@cvandal
Copy link

cvandal commented Apr 30, 2017

Thank you @schmichael! I will give this agent a try asap.

@schmichael
Copy link
Member Author

schmichael commented May 4, 2017

microsoft/go-winio#48 was merged which seems to have fixed the panic! Attached a binary for testing:

windows_amd64.zip

However I'm unable to start a simple redis job and get output like:

2017/05/03 23:36:48.931438 [DEBUG] driver.docker: failed to start container "17fea8cc5b23693e0acb572d995ddb75b32dae8b67762b6c590fe0eead343743" (attempt 1): API error
 (404): {"message":"container 17fea8cc5b23693e0acb572d995ddb75b32dae8b67762b6c590fe0eead343743 
 encountered an error during CreateContainer: failure in a Windows system call: Element not found. (0x490) extra info: {...}

I can replicate the error by running Docker directly, so it seems like a problem with my Windows, docker, and/or image?!

docker run redis:3.0-nanoserver

Would love some guidance from someone with more Windows experience! Seems like we're close!

I'm working in this branch: https://github.com/hashicorp/nomad/compare/b-2343-windows-panic

@cvandal
Copy link

cvandal commented May 4, 2017

hey @schmichael, i just rolled out the binary you provided above (0.6.0-dev) and so far so good! I also submitted a job to pull down redis:3.0-nanoserver and it's working fine for me.

@schmichael
Copy link
Member Author

@cvandal Fantastic! Any ideas what's wrong with my vm? Mind sharing your versions of Windows and Docker?

@cvandal
Copy link

cvandal commented May 4, 2017

I haven't come across that error myself, but I found a few reports of it on GitHub... According to issue moby/moby#32595 there was a recent Windows update that disabled IPv6 and by re-enabling it via this command: reg delete HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters /v DisabledComponents /f will fix it.

My OS, Docker, and Nomad versions are:
Microsoft Windows Server 2016 Datacenter 10.0.14393 Build 14393
17.03.1-ee-3
0.6.0dev

@schmichael schmichael modified the milestones: v0.6.0, near-term Jun 23, 2017
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants