-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tests: Do not leave behind state goroutines #1349
Conversation
The head writer was not being closed which would leak the `handleRequests` go routine. This PR fixes it by closing the head writer.
The test TestPenultimateLogCorruption checks for corruption towards the end of the value log file. When the DB is opened for the second time, the go routines started by db.Open() were not being closed. This PR fixes it.
A bunch of tests would start go routines and never stop them. This PR fixes that. There should not be any goroutines left after all the tests are completed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 6 of 6 files at r1.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @jarifibrahim and @manishrjain)
db.go, line 381 at r1 (raw file):
// Perform cleanup.
db.go, line 387 at r1 (raw file):
db.bfCache.Close() db.orc.Stop() db.vlog.Close()
Not sure why we are doing clean up only for this error and not in case of other error like db.get(head). Also why are not closing compactions and memtable closer.
stream_writer.go, line 229 at r1 (raw file):
return err } headWriter.closer.SignalAndWait()
Fix this as per comment given in other PR.
value_test.go, line 804 at r1 (raw file):
require.NoError(t, err) require.NoError(t, db1.Close())
No need for this lin
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 2 of 6 files reviewed, 4 unresolved discussions (waiting on @ashish-goswami and @manishrjain)
db.go, line 381 at r1 (raw file):
Previously, ashish-goswami (Ashish Goswami) wrote…
// Perform cleanup.
Done.
db.go, line 387 at r1 (raw file):
Previously, ashish-goswami (Ashish Goswami) wrote…
Not sure why we are doing clean up only for this error and not in case of other error like db.get(head). Also why are not closing compactions and memtable closer.
This code is for db.Open(...)
. If open
failed, we should return nil
. But we should also stop all the goroutines that were started for compaction and other operations. Blindly returning nil here would mean that the compactions keep running in the background and will keep crashing.
The code here stops all the operations that were started before reaching this point.
stream_writer.go, line 229 at r1 (raw file):
Previously, ashish-goswami (Ashish Goswami) wrote…
Fix this as per comment given in other PR.
Done.
value_test.go, line 804 at r1 (raw file):
Previously, ashish-goswami (Ashish Goswami) wrote…
No need for this lin
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 1 of 5 files reviewed, 1 unresolved discussion (waiting on @ashish-goswami, @jarifibrahim, and @manishrjain)
db.go, line 387 at r1 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
This code is for
db.Open(...)
. Ifopen
failed, we should returnnil
. But we should also stop all the goroutines that were started for compaction and other operations. Blindly returning nil here would mean that the compactions keep running in the background and will keep crashing.The code here stops all the operations that were started before reaching this point.
Exactly. But I think we are not closing compactions and memtable closer. In db.close()
it is done as
db.stopMemoryFlush()
db.stopCompactions()
Also why we are not doing clean up in case db.get(headKey)
returns error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dismissed @ashish-goswami from a discussion.
Reviewable status: 1 of 5 files reviewed, 1 unresolved discussion (waiting on @ashish-goswami and @manishrjain)
db.go, line 387 at r1 (raw file):
Previously, ashish-goswami (Ashish Goswami) wrote…
Exactly. But I think we are not closing compactions and memtable closer. In
db.close()
it is done asdb.stopMemoryFlush() db.stopCompactions()
Also why we are not doing clean up in case
db.get(headKey)
returns error.
I see what you mean. I've fixed that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 5 of 6 files at r2.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @manishrjain)
This PR adds the following changes from **Master branch to the dgraph-maintenance branch**. ``` 25fd0ef Update ristretto to commit f66de99 (#1391) fac972f Update head while replaying value log (#1372) e5fd05a Rework DB.DropPrefix (#1381) adeb842 Fix assert in background compression and encryption. (#1366) ``` The **master and dgraph-maintenace branch** have following difference `+` (in green) means commit missing on dgraph-maintenance `-` (in red) means the commit exists. ```diff + 079f5ae DefaultOptions: Set KeepL0InMemory to false (#1345) + 7e19cac Add immudb to the project list (#1341) + da80eb9 Iterator: Always add key to txn.reads (#1328) + a7e239e StreamWriter: Close head writer (#1347) + 543f353 Fix build on golang tip (#1355) + fd89894 Compaction: Expired keys and delete markers are never purged (#1354) + 056d859 Support disabling conflict detection (#1344) + b762832 Tests: Do not leave behind state goroutines (#1349) + b2267c2 Restore: Account for value size as well (#1358) + 14386ac GC: Consider size of value while rewriting (#1357) + c45d966 Fix assert in background compression and encryption. (#1366) + dd332b0 Avoid panic in filltables() (#1365) + 3f4761d Force KeepL0InMemory to be true when InMemory is true (#1375) + d37ce36 Tests: Use t.Parallel in TestIteratePrefix tests (#1377) + 158d927 Remove second initialization of writech in Open (#1382) + 675efcd Increase default valueThreshold from 32B to 1KB (#1346) + 3042e37 pre allocate cache key for the block cache and the bloom filter cache (#1371) + e013bfd Rework DB.DropPrefix (#1381) + 509de73 Update head while replaying value log (#1372) + 09dfa66 Update ristretto to commit f66de99 (#1391) + 717b89c Enable cross-compiled 32bit tests on TravisCI (#1392) ``` <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/dgraph-io/badger/1398) <!-- Reviewable:end -->
A bunch of tests would start goroutines and never stop them. This PR fixes that. There should not be any goroutine left after all the tests are completed. This PR also adds a db.cleanup method which would be called in case badger.Open call fails. The cleanup method will stop all the goroutines started by Open call.
A bunch of tests would start goroutines and never stop them. This
PR fixes that. There should not be any goroutines left after all the
tests are completed.
Fixes BADGER-221
This change is