Turn titan into read-only on error during gc and version_change #42

JiayuZzz · 2019-07-14T07:05:18Z

Set background error if encountered a critical error. Now it's set for two cases
- BackgroundGC() error
- LogAndApply() error after generate new blob file
If background error is set, all following foreground write operations will be refused and return background error status

codecov-io · 2019-07-14T08:49:03Z

Codecov Report

Merging #42 into master will increase coverage by 0.2%.
The diff coverage is 92.85%.

@@            Coverage Diff            @@
##           master      #42     +/-   ##
=========================================
+ Coverage   84.12%   84.33%   +0.2%     
=========================================
  Files          43       43             
  Lines        2577     2617     +40     
=========================================
+ Hits         2168     2207     +39     
- Misses        409      410      +1

tabokie · 2019-07-15T08:54:14Z

src/db_impl_gc.cc

+  Status bg_err = s;
+  if (!db_options_.listeners.empty()) {
+    // TODO(@jiayu) : check if mutex_ is freeable for futrue use case
+    mutex_.Unlock();


I don't see why we could unlock here. Listener iterator needs syncing, and callback will mutate bg_err too.

bg_err is a local variable, won't be accessed concurrently.

I'm not sure if listeners would be modified, it seems no API to do that unless reopen db

tabokie · 2019-07-15T08:57:13Z

src/titan_db_test.cc

@@ -748,6 +814,9 @@ TEST_F(TitanDBTest, FallbackModeEncounterMissingBlobFile) {
  VerifyDB({{"bar", "v1"}});
 }

+TEST_F(TitanDBTest, BackgroundErrorHandling) { BackgroundErrorHandling(); }


Could we unfold the function here? Not general enough to be a test class method i think.

tabokie · 2019-07-15T08:57:44Z

src/db_impl.h

@@ -171,6 +221,9 @@ class TitanDBImpl : public TitanDB {
  EnvOptions env_options_;
  DBImpl* db_impl_;
  TitanDBOptions db_options_;
+  // Turn DB into read-only if background happened


tabokie · 2019-07-15T08:58:00Z

src/db_impl_gc.cc

+  mutex_.AssertHeld();
+  Status bg_err = s;
+  if (!db_options_.listeners.empty()) {
+    // TODO(@jiayu) : check if mutex_ is freeable for futrue use case


tabokie · 2019-07-15T08:58:39Z

src/db_impl.h

@@ -155,6 +196,15 @@ class TitanDBImpl : public TitanDB {
    return oldest_snapshot;
  }

+  Status SetBGError(const Status& s);


Should add mutex held hint. btw, google code style says we should write accessor as set_bg_error() and bg_error(), not sure if we follows.

I think the rocksdb convention is do SetBgError for non-trivial assessor and set_bg_error for trivial ones. And this one is non-trivial.

yiwu-arbug

Cool! The patch is nice and clean!

yiwu-arbug · 2019-07-17T03:20:20Z

src/db_impl.h

@@ -43,6 +43,47 @@ class TitanDBImpl : public TitanDB {

  Status CloseImpl();

+  using TitanDB::Put;


Move the following changes to db_impl.cc?

yiwu-arbug · 2019-07-17T03:22:28Z

src/db_impl.h

@@ -155,6 +196,15 @@ class TitanDBImpl : public TitanDB {
    return oldest_snapshot;
  }

+  Status SetBGError(const Status& s);


I think the rocksdb convention is do SetBgError for non-trivial assessor and set_bg_error for trivial ones. And this one is non-trivial.

yiwu-arbug · 2019-07-17T04:05:47Z

src/titan_db_test.cc

+  // BG error is restored by listener for first time
+  ASSERT_OK(db_->Put(WriteOptions(), key, val));
+  SetBGError(Status::IOError(""));
+  ASSERT_OK(db_->Get(ReadOptions(), key, &val));


Put a non-empty value and verify the value return from Get?

yiwu-arbug · 2019-07-17T04:08:34Z

@tabokie thanks for reviewing btw :)

JiayuZzz · 2019-07-17T09:06:46Z

@yiwu-arbug @tabokie All modified, thanks for reviewing.

yiwu-arbug

LGTM

Turn titan into read-only on error during gc and version_change (tikv#42)

JiayuZzz added 3 commits July 14, 2019 06:44

turn db into readonly if background error during gc/version_change

2b0a586

add tests

b8c42fa

fmt

4bafc79

JiayuZzz requested a review from yiwu-arbug July 15, 2019 02:20

tabokie reviewed Jul 15, 2019

View reviewed changes

tabokie changed the title ~~Turn titan into read-only if encountere error during gc and version_change~~ Turn titan into read-only on error during gc and version_change Jul 15, 2019

JiayuZzz added 2 commits July 15, 2019 11:24

unfold test functions

4b6d08b

mutex held hint

0006db0

yiwu-arbug reviewed Jul 17, 2019

View reviewed changes

move write operation implements to db_impl.cc

9a597dd

yiwu-arbug approved these changes Jul 18, 2019

View reviewed changes

yiwu-arbug merged commit 8ee2e79 into tikv:master Jul 18, 2019

GlitterIsMe added a commit to GlitterIsMe/titan that referenced this pull request Jul 20, 2019

Merge pull request #3 from pingcap/master

e7f04b4

Turn titan into read-only on error during gc and version_change (tikv#42)

lday0321 mentioned this pull request Aug 2, 2020

is HasBGError() strong enough when some error happened in LogAndApply #182

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Turn titan into read-only on error during gc and version_change #42

Turn titan into read-only on error during gc and version_change #42

JiayuZzz commented Jul 14, 2019

codecov-io commented Jul 14, 2019 •

edited

Loading

tabokie Jul 15, 2019

JiayuZzz Jul 15, 2019

JiayuZzz Jul 15, 2019

tabokie Jul 15, 2019 •

edited

Loading

JiayuZzz Jul 15, 2019

tabokie Jul 15, 2019

tabokie Jul 15, 2019

tabokie Jul 15, 2019 •

edited

Loading

yiwu-arbug Jul 17, 2019

yiwu-arbug left a comment

yiwu-arbug Jul 17, 2019

yiwu-arbug Jul 17, 2019

yiwu-arbug Jul 17, 2019

yiwu-arbug commented Jul 17, 2019

JiayuZzz commented Jul 17, 2019 •

edited

Loading

yiwu-arbug left a comment

		@@ -43,6 +43,47 @@ class TitanDBImpl : public TitanDB {

		Status CloseImpl();

		using TitanDB::Put;

Turn titan into read-only on error during gc and version_change #42

Turn titan into read-only on error during gc and version_change #42

Conversation

JiayuZzz commented Jul 14, 2019

codecov-io commented Jul 14, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tabokie Jul 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tabokie Jul 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yiwu-arbug left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yiwu-arbug commented Jul 17, 2019

JiayuZzz commented Jul 17, 2019 • edited Loading

yiwu-arbug left a comment

Choose a reason for hiding this comment

codecov-io commented Jul 14, 2019 •

edited

Loading

tabokie Jul 15, 2019 •

edited

Loading

tabokie Jul 15, 2019 •

edited

Loading

JiayuZzz commented Jul 17, 2019 •

edited

Loading