Bitswap without DHT integration tests occasionally hanging

[whyrusleeping]

Logs here:
https://gist.github.com/whyrusleeping/8f5693e8ceea552a8b9c

[whyrusleeping]

Appears to maybe be a duplicate of #805

[whyrusleeping]

Found a data race:

whyrusleeping@idril ~/g/s/g/j/go-ipfs (master)> go test ./test/integration/ -v -run TestBitswapWithoutRouting -race
=== RUN TestBitswapWithoutRouting
==================
WARNING: DATA RACE
Write by goroutine 193:
  sync.raceWrite()
      /home/whyrusleeping/go/src/sync/race.go:41 +0x36
  sync.(*WaitGroup).Wait()
      /home/whyrusleeping/go/src/sync/waitgroup.go:125 +0x177
  github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup.(*contextGroup).closeLogic()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup/ctxgroup.go:202 +0xd7
  github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup.*contextGroup.(github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup.closeLogic)·fm()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup/ctxgroup.go:192 +0x34
  sync.(*Once).Do()
      /home/whyrusleeping/go/src/sync/once.go:44 +0xe6

Previous read by goroutine 115:
  sync.raceRead()
      /home/whyrusleeping/go/src/sync/race.go:37 +0x36
  sync.(*WaitGroup).Add()
      /home/whyrusleeping/go/src/sync/waitgroup.go:63 +0xc1
  github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup.(*contextGroup).AddChildFunc()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup/ctxgroup.go:168 +0x4e
  github.com/jbenet/go-ipfs/routing/dht.(*dhtQueryRunner).Run()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/routing/dht/query.go:110 +0x4cd
  github.com/jbenet/go-ipfs/routing/dht.(*dhtQuery).Run()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/routing/dht/query.go:59 +0xdd
  github.com/jbenet/go-ipfs/routing/dht.func·009()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/routing/dht/lookup.go:83 +0x11e

Goroutine 193 (running) created at:
  github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup.(*contextGroup).internalClose()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup/ctxgroup.go:192 +0xba
  github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup.(*contextGroup).closeOnContextDone()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-ctxgroup/ctxgroup.go:211 +0xad

Goroutine 115 (running) created at:
  github.com/jbenet/go-ipfs/routing/dht.(*IpfsDHT).GetClosestPeers()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/routing/dht/lookup.go:87 +0xc83
  github.com/jbenet/go-ipfs/routing/dht.(*IpfsDHT).Provide()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/routing/dht/routing.go:146 +0x451
  github.com/jbenet/go-ipfs/exchange/bitswap/network.(*impl).Provide()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/exchange/bitswap/network/ipfs_impl.go:138 +0xa4
  github.com/jbenet/go-ipfs/exchange/bitswap.(*bitswap).provideWorker()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/exchange/bitswap/workers.go:67 +0x374
  github.com/jbenet/go-ipfs/exchange/bitswap.func·011()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/exchange/bitswap/workers.go:32 +0x6f
  github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess.func·005()
      /home/whyrusleeping/gopkg/src/github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/impl-mutex.go:96 +0xa7
==================
--- PASS: TestBitswapWithoutRouting (0.60s)
PASS
ok      github.com/jbenet/go-ipfs/test/integration  0.614s

[whyrusleeping]

@jbenet im not very clear on the part of the codebase that race is in, do you have any insights?

[jbenet]

not sure-- i think it's dht query runner / ctxgroup mix. may be fixed by switching to goprocess or there may be an actual problem

[whyrusleeping]

Yeah, something in that race is why this test is hanging...

[whyrusleeping]

Fixed by #834