Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure we Stop() everything when the main probe process exits #424

Closed
inercia opened this issue Sep 2, 2015 · 4 comments
Closed

Ensure we Stop() everything when the main probe process exits #424

inercia opened this issue Sep 2, 2015 · 4 comments
Assignees
@peterbourgon
Labels
tech-debt Unpleasantness that does (or may in future) affect development

Comments

@inercia
Copy link
Contributor

inercia commented Sep 2, 2015

No description provided.

@inercia inercia changed the title Ensure we Stop() everything when the main probe process exists Ensure we Stop() everything when the main probe process exits Sep 2, 2015
@peterbourgon
Copy link
Contributor

  • Which components aren't being stopped?
  • We should be stopping everything properly with defer foo.Stop() in func main.

@inercia
Copy link
Contributor Author

inercia commented Sep 2, 2015

The process has ben unresponsive to Ctrl-Cs a couple of times while capturing packets, so I guess there is something we are not Stop()ing (or maybe there is a problem in the pcap library)...

@inercia
Copy link
Contributor Author

inercia commented Sep 2, 2015

I don't know if it's worth filling as a different issue, but sometimes I get this error when Ctrl-Cing:

2015/09/02 18:21:34 conntrack error: XML syntax error on line 79608: unexpected end element </conntrack>
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4c0012]

goroutine 26 [running]:
github.com/weaveworks/scope/probe/endpoint.(*Conntracker).handleFlow(0xc82012dd40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/conntrack.go:190 +0x6c2
github.com/weaveworks/scope/probe/endpoint.(*Conntracker).run(0xc82012dd40, 0xc8200181e0, 0x4, 0x6)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/conntrack.go:153 +0xc24
created by github.com/weaveworks/scope/probe/endpoint.NewConntracker
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/conntrack.go:78 +0x14e

goroutine 1 [runnable]:
sync.runtime_Semacquire(0xc8200100f4)
        /usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0xc8200100f0)
        /usr/local/go/src/sync/mutex.go:82 +0x1c4
log.(*Logger).Output(0xc8200100f0, 0x2, 0xc8210be090, 0x9, 0x0, 0x0)
        /usr/local/go/src/log/log.go:147 +0x8b
log.Printf(0xb60980, 0x2, 0xc8206b7c20, 0x1, 0x1)
        /usr/local/go/src/log/log.go:289 +0x80
main.main()
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/main.go:199 +0x2a24

goroutine 17 [syscall, 27 minutes, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 18 [syscall]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 24 [select]:
main.staticResolver.loop(0xc82013e0c0, 0xc820efd700, 0xc820141ac0, 0x2, 0x2)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/resolver.go:72 +0x17a
created by main.newStaticResolver
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/resolver.go:40 +0xf1

goroutine 25 [runnable]:
syscall.Syscall6(0x3d, 0x1907, 0xc8215a77ec, 0x0, 0xc8220f6000, 0x0, 0x0, 0x1907, 0x0, 0x0)
        /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
syscall.wait4(0x1907, 0xc8215a77ec, 0x0, 0xc8220f6000, 0x90, 0x0, 0x0)
        /usr/local/go/src/syscall/zsyscall_linux_amd64.go:172 +0x72
syscall.Wait4(0x1907, 0xc8215a7834, 0x0, 0xc8220f6000, 0xc8220ce040, 0x0, 0x0)
        /usr/local/go/src/syscall/syscall_linux.go:256 +0x55
os.(*Process).wait(0xc8212fa0e0, 0xc82014d080, 0x0, 0x0)
        /usr/local/go/src/os/exec_unix.go:22 +0x105
os.(*Process).Wait(0xc8212fa0e0, 0x58e8f4, 0x0, 0x0)
        /usr/local/go/src/os/doc.go:45 +0x2d
os/exec.(*Cmd).Wait(0xc820156280, 0x0, 0x0)
        /usr/local/go/src/os/exec/exec.go:380 +0x211
github.com/weaveworks/scope/probe/endpoint.(*Conntracker).run.func1(0x7ff50c3bf328, 0xc82002a098)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/conntrack.go:124 +0x2e
github.com/weaveworks/scope/probe/endpoint.(*Conntracker).handleFlow(0xc82012d3c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/conntrack.go:190 +0x6c2
github.com/weaveworks/scope/probe/endpoint.(*Conntracker).run(0xc82012d3c0, 0xc8212d6540, 0x3, 0x3)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/conntrack.go:153 +0xc24
created by github.com/weaveworks/scope/probe/endpoint.NewConntracker
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/conntrack.go:78 +0x14e

goroutine 27 [IO wait]:
net.runtime_pollWait(0x7ff50d4391f0, 0x72, 0xc82000a1c0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820130300, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820130300, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201302a0, 0xc82015d400, 0x200, 0x200, 0x0, 0x7ff50d40b050, 0xc82000a1c0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82002a040, 0xc82015d400, 0x200, 0x200, 0x200, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net.(*UDPConn).readDNSResponse(0xc82002a040, 0xc820090880, 0x0, 0x0)
        /usr/local/go/src/net/dnsclient_unix.go:44 +0xa5
net.exchange(0xc8211717b0, 0xc, 0xc82192eac0, 0x1a, 0xc82117000c, 0x12a05f200, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/dnsclient_unix.go:147 +0x4d1
net.tryOneName(0xc820090a80, 0xc82192eac0, 0x1a, 0xc82117000c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/net/dnsclient_unix.go:176 +0x425
net.lookup(0xc82192eac0, 0x1a, 0xc82192000c, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/dnsclient_unix.go:309 +0x2e7
net.goLookupPTR(0xc820e45ce0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/dnsclient_unix.go:515 +0x167
net.lookupAddr(0xc820e45ce0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/lookup_unix.go:158 +0x125
net.LookupAddr(0xc820e45ce0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/lookup.go:168 +0x4c
github.com/weaveworks/scope/probe/endpoint.(*ReverseResolver).run(0xc82129bd80)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/resolver.go:56 +0xe7
created by github.com/weaveworks/scope/probe/endpoint.NewReverseResolver
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/resolver.go:30 +0x181

goroutine 28 [runnable]:
github.com/weaveworks/scope/report.NodeMetadata.Copy(0xc821844d80, 0xc821844db0, 0xc8210cb7d0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/report/topology.go:218 +0x187
github.com/weaveworks/scope/report.NodeMetadatas.Copy(0xc821844d50, 0xc821845110)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/report/topology.go:84 +0x170
github.com/weaveworks/scope/report.Topology.Copy(0xc821844d20, 0xc821844d50, 0x0, 0x0)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/report/topology.go:40 +0x45
github.com/weaveworks/scope/report.Topology.WithNode(0xc821844d20, 0xc821844d50, 0xc8210cbd10, 0xd, 0xc8218450b0, 0xc8218450e0, 0xc82250a8e0, 0x2, 0x2, 0x0, ...)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/report/topology.go:31 +0x18d
github.com/weaveworks/scope/probe/endpoint.(*Reporter).addConnection(0xc82012ddc0, 0xc8206bda48, 0xc8210cbb80, 0xc, 0xc8210cbc80, 0xe, 0xc801bbb70a, 0xc8200ffd50)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/reporter.go:172 +0x821
github.com/weaveworks/scope/probe/endpoint.(*Reporter).Report(0xc82012ddc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/endpoint/reporter.go:112 +0x279
main.main.func3(0xc820efd160, 0xc820efd190, 0xc82129c4e0, 0xc82012de00, 0xc8212d64b0, 0x3, 0x3, 0xc82129bdc0, 0x2, 0x2, ...)
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/main.go:186 +0x469
created by main.main
        /home/alvaro/Development/go/src/github.com/weaveworks/scope/probe/main.go:198 +0x2903

goroutine 29 [select, 27 minutes, locked to thread]:
runtime.gopark(0xc9e618, 0xc820025728, 0xb6cc08, 0x6, 0x435a18, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc820025728, 0x0, 0x18)
        /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc820025728)
        /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

@tomwilkie
Copy link
Contributor

Please file a separate issue for that.

@peterbourgon peterbourgon added the tech-debt Unpleasantness that does (or may in future) affect development label Sep 18, 2015
@peterbourgon peterbourgon mentioned this issue Sep 21, 2015
Merged
@tomwilkie tomwilkie mentioned this issue Sep 23, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@peterbourgon peterbourgon

Labels
tech-debt Unpleasantness that does (or may in future) affect development
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@peterbourgon @tomwilkie @inercia