Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ui: add data distribution screen (aka replica matrix) #2

Closed
wants to merge 1,620 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
1620 commits
Select commit Hold shift + click to select a range
9ed03a0
Merge #26435
Jun 6, 2018
3c149f0
libroach: Obey read-only option in FileRegistry and DataKeyManager.
Jun 6, 2018
5dc88f6
Merge #26373
Jun 6, 2018
948ce12
util: explicitly enumerate encoding types
Jun 5, 2018
da2025f
storage: Disable campaign-on-wake when receiving raft messages
bdarnell Jun 5, 2018
20e8446
lint: Fix a file descriptor leak
bdarnell Jun 6, 2018
b09ab16
build: Pin go.uuid to the version currently in use
bdarnell Jun 6, 2018
099fc95
Merge #26468
Jun 6, 2018
e8911d4
Merge #26467
Jun 6, 2018
b0ab0d6
distsql: add NewFinalIterator to the rowIterator interface
asubiotto Jun 5, 2018
16f505a
libroach: format C++ code.
Jun 6, 2018
f6f6ee8
Merge #26441 #26463 #26469 #26470
Jun 6, 2018
35dd0a7
sql: support multiple SRFs in the same SELECT clause
knz Jun 6, 2018
81161b3
sql: add pg_catalog.pg_shdescription
BramGruneir Jun 6, 2018
6c70e54
Merge #26473
Jun 6, 2018
947c04f
Merge #26430 #26466
Jun 6, 2018
74a3a7a
sql, storage: fixed the spelling of deterministic
BramGruneir Jun 6, 2018
022befb
Merge #26223 #26474
Jun 6, 2018
6d1b0c4
distsql: remove unused ctx parameter from makeDiskRowContainer
asubiotto May 31, 2018
2ca0e0d
distsql: add diskBackedRowContainer
asubiotto Jun 1, 2018
341dfce
distsql: add diskBackedRowContainer to sortAllProcessor
asubiotto Jun 1, 2018
5d22bd3
sql: add pg_builtin `pg_is_xlog_replay_paused()`
BramGruneir Jun 6, 2018
efbc4b9
Merge #26462
Jun 6, 2018
743bb81
sql: fix small spelling error
BramGruneir Jun 6, 2018
e72da1b
Merge #26484
Jun 6, 2018
7a2f0e1
sql: various RSG improvements and new tests
maddyblue Jun 5, 2018
d316a68
opt/optbuilder: allow aggregation in order by
rytaft Jun 6, 2018
9800ea5
Merge #26458
Jun 6, 2018
5730e82
importccl: eliminate extra sample ch worker
dt Jun 6, 2018
7c4dee1
Merge #26490
Jun 6, 2018
74b0bf7
backupccl: support SHOW BACKUP [RANGES|FILES]
benesch Jun 5, 2018
b1c30ff
Merge #26450
Jun 6, 2018
5605f08
cli: support encrypted stores in debug commands.
Jun 6, 2018
80ab908
Merge #26477
Jun 6, 2018
d972ce1
Merge #26452
Jun 7, 2018
59b485b
cli/sql: make `\set` recognize assignments with `=`
knz Jun 7, 2018
5a9f24b
cli: make `cockroach demo` look&feel more like `cockroach sql`
knz Jun 7, 2018
6e4de47
cli: Add the --set flag to cochroach sql
chrisseto May 31, 2018
52df162
Merge #26287
Jun 7, 2018
ab535e5
storage: remove TestPut
tbg Jun 7, 2018
cdaa019
sql: simplify setNeededColumns
knz Jun 7, 2018
949dc21
sql: add a missing comment
knz Jun 7, 2018
b7c4848
sql: hoist an assignment
knz Jun 7, 2018
2804415
roachtest: fix kv use of encryption and enable.
Jun 7, 2018
4c43012
Merge #26502
Jun 7, 2018
f7f5419
Merge #26505
Jun 7, 2018
f2e379f
Merge #26357
Jun 7, 2018
8f43aa6
ui: sort system database last on databases page
couchand May 22, 2018
61c4a55
ui: remove unneeded excludedTableList
couchand May 23, 2018
50b3d68
ui: separate user & system databases with a divider
couchand May 23, 2018
557bab7
storage: remove last part of the TestTransferRaftLeadership test
windchan7 Jun 7, 2018
995bcf7
Merge #26480
Jun 7, 2018
46c1b4f
ui: declosurify event handlers on login page
couchand May 24, 2018
a9f74ae
ui: style login page
couchand May 24, 2018
b966f7c
ui: error state for login page
couchand May 29, 2018
79b9dc4
ui: add cockroach icon to login page
couchand May 29, 2018
4eaa236
ui: style navbar login indicator
couchand May 29, 2018
9f6f519
ui: add logout route
couchand May 29, 2018
f1ef286
ui: fix ts errors from previous changes
couchand Jun 4, 2018
973e516
ui: extract selector for sorting databases by type
couchand Jun 7, 2018
e235458
Merge #25817
Jun 7, 2018
6ad2093
sql: make a copy of the columns in explainNode
RaduBerinde Jun 7, 2018
bfa71a8
Merge #26507
Jun 7, 2018
8d20abc
ui, server: pass build tag to ui
couchand Jun 4, 2018
31f60eb
ui: update styling of login page
couchand Jun 4, 2018
6d064e0
ui: style insecure mode indicator
couchand Jun 4, 2018
ee1a743
ui: change copy on sign in button while waiting
couchand Jun 4, 2018
de8d6f1
ui: name route creators
couchand Jun 7, 2018
f1ff567
Merge #26491
Jun 7, 2018
0e0eaa2
workload/bank: improve splits UX
danhhz Jun 6, 2018
ebc9b0d
Merge #26511
Jun 7, 2018
c10e6d0
sql: add pg_catalog.pg_seclabel and pg_catalog.pg_shseclabel
BramGruneir Jun 7, 2018
993f7dd
distsqlrun: short circuit hash joins on empty side
Jun 7, 2018
29a54cf
sql: disable optimizer if force-lookup-joins flag set
RaduBerinde Jun 7, 2018
26fe882
Merge #26066
Jun 7, 2018
8b40a36
cli: brush up `debug check-store`
tbg Jun 7, 2018
7f39653
kv: stop the heartbeat loop on rollback errors
andreimatei Jun 6, 2018
690103b
libroach: don't initialize AESCipher::Decrypt
Jun 7, 2018
caa7989
Merge #26485 #26515
Jun 7, 2018
a6a7113
Merge #26479 #26516
Jun 7, 2018
16f75f2
ui: move all urls to a single file
couchand Jun 7, 2018
75b595b
roachtest: provide a timeout for tests
petermattis Jun 7, 2018
22554db
kv: fix issues around failed 1PC txns
andreimatei Jun 6, 2018
4dca928
ts: Properly read columnar format
May 30, 2018
0e84be8
Merge #26497
Jun 7, 2018
f1b36e5
kv: make transport.SendNext synchronous
jordanlewis Jun 1, 2018
f210d3e
Merge #26340 #26517 #26520 #26521
Jun 7, 2018
0586c83
Merge #25829
Jun 7, 2018
5258ccf
distsql: add stall time to InputStats
asubiotto Jun 6, 2018
f3aaa27
changefeedccl: rename hooks.go to changefeed_stmt.go
danhhz Jun 7, 2018
6219034
sql: add TODO for different conn close test
andreimatei Jun 7, 2018
aa2b931
storage: permit full leases in fatal logs from leasePostApply assertions
nvanbenschoten Jun 7, 2018
cdda7c0
opt: Fix panic when projecting non-covered column in sorted index join
andy-kimball Jun 8, 2018
301f934
changefeedccl: remove unnecessary test setup
danhhz Jun 8, 2018
49ad01b
changefeedccl: improve failed CREATE CHANGEFEED ux
danhhz Jun 7, 2018
3b3b326
kv: interrupt DistSender replica retry loop on ctx cancelation
andreimatei Jun 8, 2018
e6488f9
sql: support a limited subset of correlated SRFs
knz Jun 7, 2018
e2f7c33
opt: add EXPLAIN (OPT)
RaduBerinde Jun 8, 2018
50fb0f2
Merge #26461
Jun 8, 2018
f762b26
Merge #26536
Jun 8, 2018
4aedb39
roachtest: quiescence with dead node
tbg Jun 8, 2018
020ce2c
Merge #26542
Jun 8, 2018
748d058
storage: test lease request MaxRaftIndex handling
tbg Jun 7, 2018
79a7273
storage: don't spuriously retry lease proposals
tbg Jun 8, 2018
8864d22
Merge #26535
Jun 8, 2018
7254e37
opt: generalize LookupJoin operation
RaduBerinde Jun 6, 2018
451cb4e
opt: test updates
RaduBerinde Jun 6, 2018
87ef1b5
Merge #26446
Jun 8, 2018
da418b9
Merge #26486
Jun 8, 2018
5253acc
opt: add Index to LookupJoinDef
RaduBerinde Jun 9, 2018
8a437e5
opt: test updates
RaduBerinde Jun 9, 2018
f9b207e
opt: fixing some test TODOs
RaduBerinde Jun 9, 2018
ce2ed83
Merge #26509
Jun 9, 2018
7ece796
Merge #26566 #26567
Jun 9, 2018
4088cdd
distsqlrun: export distinct processor
Apr 19, 2018
4413d78
sql: auto-generate column names like in PostgreSQL
knz Jun 8, 2018
dde686a
Merge #26550
Jun 10, 2018
a1aca8d
opt: fix index-join ordering when moving sort around
RaduBerinde Jun 10, 2018
2f2a1df
Merge #26575
Jun 11, 2018
2987303
sql: clean up and stabilize the output of SHOW CONSTRAINTS
knz Jun 6, 2018
b065232
Merge #26478
Jun 11, 2018
508721b
storage/engine: RocksDB clear range hack
petermattis Jun 10, 2018
b225cc5
sql: Add support for ON CONFLICT DO NOTHING with no columns specified
emsal0 Jun 4, 2018
b3eddf8
Merge #26471
Jun 11, 2018
16da4ef
Merge #26465
Jun 11, 2018
91c1bf3
Merge #26576
Jun 11, 2018
96546dd
roachtest: Make election test work on release-2.0
a-robinson Jun 11, 2018
eb649af
sql: Remove excessive allocation in upsert conflict index enumeration
emsal0 Jun 11, 2018
74cfb8a
sql: add missing PostgreSQL built-in functions
knz Jun 11, 2018
ca05f45
ts: Add ability to write columnar data
Jun 11, 2018
3d72bd1
Merge #26586
Jun 11, 2018
c3fcf1a
Merge #26591
Jun 11, 2018
1e83a49
sql: remove a stale test comment
knz Jun 11, 2018
40c2b39
sql: normalize function names
knz Jun 11, 2018
20c6fe8
sql: be a bit more helpful to the user when a function is not found
knz Jun 11, 2018
9438e12
Merge #26588
Jun 11, 2018
ff23fcb
workload : fix histograms to take ramp into account
Jun 11, 2018
c1797f6
Merge #26589
Jun 11, 2018
02d71cf
importccl: add detail to mysql errors
dt May 25, 2018
2693c1d
sql: export InitTableDescriptor
dt May 22, 2018
76c992f
importccl: convert MySQL CREATE TABLE to cockroach
dt May 21, 2018
3582d8a
importccl: refactor mysql testdata creation
dt May 26, 2018
8e8ce54
Merge #26533
Jun 11, 2018
6366a35
kv: fix intent tracking
andreimatei Jun 11, 2018
1abcd1f
importccl: 'IMPORT TABLE foo FROM MYSQLDUMP bar'
dt May 26, 2018
7fa5485
importccl: multi-table `IMPORT MYSQLDUMP foo`
dt Jun 5, 2018
85eb2e5
Merge #26539
Jun 11, 2018
35a1def
rfc: change data capture
danhhz May 1, 2018
9ea1a40
Merge #25229
Jun 11, 2018
a9dfd98
internal/client: remove dead options
andreimatei Feb 24, 2018
8546492
backupccl: plug txn leak
andreimatei Feb 24, 2018
5fff8ba
client/txn: delete obsolete test
andreimatei Feb 24, 2018
cc6aa7b
internal/client: simplify test
andreimatei Feb 24, 2018
8d61ce9
internal/client, storage: migrate tests away from txn.Exec()
andreimatei Feb 24, 2018
22c26fd
internal/client: make txn.Exec() private
andreimatei Feb 24, 2018
4aa3d4f
sql: prevent txn leak in connExecutor
andreimatei Feb 24, 2018
edf7c72
kv: stop concerning the TxnCoordSender with abandoned txns
andreimatei Feb 25, 2018
dc34afc
kv: no more async abort for transactions
andreimatei May 11, 2018
b38728d
client: Make async rollback a stopper task
andreimatei May 17, 2018
f3e9650
client: remove a now-useless context fork in db.Txn()
andreimatei Jun 7, 2018
07a299f
Merge #23055 #26598
Jun 11, 2018
c74fb21
sql: use distsql for local distinct
Mar 16, 2018
f664a60
Merge #26579
Jun 12, 2018
e496459
Merge #26606
Jun 12, 2018
396ea7e
Merge #26503
Jun 12, 2018
4b995bd
storage: up verbosity on log message
tbg Jun 12, 2018
c3560ea
opt: make data-driven tests emit file info on own line
knz Jun 12, 2018
a364862
libroach: don't expose cryptopp includes in headers.
Jun 12, 2018
50416d0
Merge #26620
Jun 12, 2018
529f626
Merge #25873
Jun 12, 2018
54635d7
ui: extract sql box to standalone component
couchand Jun 12, 2018
7cc4975
Merge #26534
Jun 12, 2018
9182cc1
changefeedccl: add an initial Changefeed benchmark
danhhz Jun 8, 2018
bb991a7
changefeedccl: fix bug with early closed producer
danhhz Jun 8, 2018
51cd27e
Merge #26549
Jun 12, 2018
4738e40
changefeedccl: cache RowFetchers
danhhz Jun 11, 2018
2b87b8d
opt: fix catalog behavior for primary index
RaduBerinde Jun 12, 2018
897301f
Merge #26514
Jun 12, 2018
057cbf2
kv: Bail out early if context is canceled
bdarnell Jun 12, 2018
e7817ad
storage: Clear gc queue batch on failure
bdarnell Jun 12, 2018
6bc19ca
storage: Check for context cancellation in GC queue
bdarnell Jun 12, 2018
fbb00f6
Merge #26631
Jun 12, 2018
6639d36
server: add endpoint /_admin/v1/queries
Jan 19, 2018
dc15c47
sql, server: plumb through less-scrubbed queries to endpoint
Jun 11, 2018
6605993
ui: add "Queries" screen, showing recent queries & stats
Jan 19, 2018
8f326a9
ui: link to queries page from debug index
Jun 11, 2018
0249981
Merge #24485
Jun 12, 2018
2b24f64
roachtest: fix copy test
tbg Jun 12, 2018
350d68a
roachtest: disable quiescence test on release-2.0
tbg Jun 12, 2018
b81f1cc
Merge #26623
Jun 12, 2018
82d8956
ts: Enable Columnar Format with Cluster Version
Jun 12, 2018
ee9522c
Merge #26614
Jun 12, 2018
e79ee3a
ui: Add lease transfers to graph of range operations
a-robinson Jun 12, 2018
133b505
Merge #26487
Jun 12, 2018
e90f8d9
changefeedccl: reduce some allocations
danhhz Jun 11, 2018
dd8f79a
distsql: move tableReader stat collection setup out of Start
asubiotto Jun 12, 2018
66f08dc
Merge #26612
Jun 12, 2018
a130030
sql: drain result channel in distSQLWrapper
asubiotto Jun 12, 2018
857b3c9
Merge #26636 #26653 #26655
Jun 12, 2018
575b0b2
opt: Move listBuilder to new xfunc package
rytaft Jun 7, 2018
90f48d5
Merge #26164
Jun 12, 2018
45a2174
opt: move all custom functions to CustomFuncs structs
rytaft Jun 11, 2018
3b58c9f
opt: Move filter functions to xfunc package
rytaft Jun 11, 2018
386d857
opt: move some factory helper functions to xfunc
rytaft Jun 12, 2018
db49eb3
engine: disable mvcc perf stats collection on tracing
jordanlewis Jun 12, 2018
fdb924c
changefeedccl: skip flaky TestChangefeedPauseUnpause
danhhz Jun 12, 2018
d42f272
Merge #26578
Jun 12, 2018
5070f34
Merge #26660
Jun 12, 2018
9544af0
Merge #26601
Jun 12, 2018
d4697df
sql: only set PlaceholderInfo type if not already set
nvanbenschoten Jun 13, 2018
ff25bec
sql,compactor: rate limit clear range requests
benesch Jun 5, 2018
6e258f7
Merge #26449
Jun 13, 2018
46cd0b2
parser,changefeedccl: partial update of CHANGEFEED syntax to match rfc
danhhz Jun 6, 2018
7c5b549
Merge #26675
Jun 13, 2018
4708eef
jobs: factor out jobspb package
tbg Jun 13, 2018
5dafdcb
Merge #26664
Jun 13, 2018
0a3df5f
Merge #26622
Jun 13, 2018
d9cb7b2
sql/sem: simplify (*FuncExpr).evalArgs()
knz Jun 12, 2018
5947420
sql,opt: propagate composite types (labeled tuples)
knz Jun 12, 2018
5ca4c86
opt: Streamline logicalPropsBuilder
andy-kimball Jun 5, 2018
3b6c57e
opt: Move base table props generation to props package
andy-kimball Jun 9, 2018
0a51b15
opt: don't push limit through project when ordering on synthesized co…
RaduBerinde Jun 13, 2018
8bd51b6
sql: Check CHECK constraints on upsert when updating a conflicting row
emsal0 Jun 12, 2018
5280a1c
Merge #26570
Jun 13, 2018
a406b3b
Merge #26621 #26683
Jun 13, 2018
0506707
opt: ignore inverted indexes
RaduBerinde Jun 13, 2018
327faee
Merge #26642
Jun 13, 2018
30baaa1
Merge #26686
Jun 13, 2018
6881f30
libroach: make CryptoPP build with runtime AES-NI detection.
Jun 12, 2018
4e328d3
Merge #26663
Jun 13, 2018
aa6ac26
sql: fix the ordering of SRF projections
knz Jun 13, 2018
8f4ecb6
roachtest: use --sequential when generating store dumps
petermattis Jun 13, 2018
aac7366
server: use the right ctx when logging an error
andreimatei Jun 13, 2018
a7fab62
Merge #26706
Jun 13, 2018
90a874c
Merge #26670
Jun 13, 2018
80afdb8
cli: use the right context for logging on signal
andreimatei Jun 13, 2018
460472a
Merge #26703
Jun 14, 2018
3ed9485
sql: fix _pg_expandarray() and generate_subscripts()
knz Jun 14, 2018
4a97f07
sql: Support tuple column access and tuple stars
BramGruneir May 21, 2018
b4b0522
Merge #26628
Jun 14, 2018
12b3ab7
sql: Fix panic adding unique, non-indexable column
bobvawter Jun 13, 2018
9466628
Merge #26716
Jun 14, 2018
f4faa15
Merge #26684 #26705
Jun 14, 2018
b497dd0
Merge #26649
Jun 14, 2018
55ac430
opt: generate spans for IN filters
Jun 13, 2018
a49f75b
Merge #26637
Jun 14, 2018
b3f1d17
Merge #26708
Jun 14, 2018
f0c682e
Merge #26643
Jun 14, 2018
2a2f765
server: rename ReplicaMatrix endpoint to DataDistribution
May 31, 2018
721de4c
ui: add data distribution screen (aka replica matrix)
Dec 4, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
libroach: Obey read-only option in FileRegistry and DataKeyManager.
These two classes handle the File and Key registries persisted to local
disk. When the read-only option is set on the DB, it's important we not
modify on-disk state or we risk corrupting data for a running node.
marc committed Jun 6, 2018
commit 3c149f033add185d8d84f9452ad391ecf6546e46
16 changes: 9 additions & 7 deletions c-deps/libroach/ccl/db.cc
Original file line number Diff line number Diff line change
@@ -116,8 +116,8 @@ rocksdb::Status DBOpenHook(std::shared_ptr<rocksdb::Logger> info_log, const std:
env_mgr->TakeEnvOwnership(store_keyed_env);

// Initialize data key manager using the stored-keyed-env.
DataKeyManager* data_key_manager =
new DataKeyManager(store_keyed_env, db_dir, opts.data_key_rotation_period());
DataKeyManager* data_key_manager = new DataKeyManager(
store_keyed_env, db_dir, opts.data_key_rotation_period(), db_opts.read_only);
status = data_key_manager->LoadKeys();
if (!status.ok()) {
delete data_key_manager;
@@ -139,11 +139,13 @@ rocksdb::Status DBOpenHook(std::shared_ptr<rocksdb::Logger> info_log, const std:
std::unique_ptr<enginepbccl::KeyInfo> store_key = store_key_manager->CurrentKeyInfo();
assert(store_key != nullptr);

// Generate a new data key if needed by giving the active store key info to the data key
// manager.
status = data_key_manager->SetActiveStoreKey(std::move(store_key));
if (!status.ok()) {
return status;
if (!db_opts.read_only) {
// Generate a new data key if needed by giving the active store key info to the data key
// manager.
status = data_key_manager->SetActiveStoreKey(std::move(store_key));
if (!status.ok()) {
return status;
}
}

// Everything's ok: initialize a stats handler.
60 changes: 60 additions & 0 deletions c-deps/libroach/ccl/db_test.cc
Original file line number Diff line number Diff line change
@@ -83,3 +83,63 @@ TEST(LibroachCCL, DBOpen) {
DBClose(db);
}
}

TEST(LibroachCCL, ReadOnly) {
// We need a real directory.
TempDirHandler dir;
ASSERT_TRUE(dir.Init());

{
// Write/read a single key.
DBEngine* db;
DBOptions db_opts = defaultDBOptions();

EXPECT_STREQ(DBOpen(&db, ToDBSlice(dir.Path("")), db_opts).data, NULL);
EXPECT_STREQ(DBPut(db, ToDBKey("foo"), ToDBSlice("foo's value")).data, NULL);
DBString value;
EXPECT_STREQ(DBGet(db, ToDBKey("foo"), &value).data, NULL);
EXPECT_STREQ(ToString(value).c_str(), "foo's value");

DBClose(db);
}

{
// Re-open read-only without encryption options.
DBEngine* db;
DBOptions db_opts = defaultDBOptions();
db_opts.read_only = true;

EXPECT_STREQ(DBOpen(&db, ToDBSlice(dir.Path("")), db_opts).data, NULL);
// Read the previously-written key.
DBString ro_value;
EXPECT_STREQ(DBGet(db, ToDBKey("foo"), &ro_value).data, NULL);
EXPECT_STREQ(ToString(ro_value).c_str(), "foo's value");
// Try to write it again.
EXPECT_EQ(ToString(DBPut(db, ToDBKey("foo"), ToDBSlice("foo's value"))),
"Not implemented: Not supported operation in read only mode.");

DBClose(db);
}

{
// Re-open read-only with encryption options (plaintext-only).
DBEngine* db;
DBOptions db_opts = defaultDBOptions();
db_opts.read_only = true;
db_opts.use_file_registry = true;
auto extra_opts = MakePlaintextExtraOptions();
ASSERT_NE(extra_opts, "");
db_opts.extra_options = ToDBSlice(extra_opts);

EXPECT_STREQ(DBOpen(&db, ToDBSlice(dir.Path("")), db_opts).data, NULL);
// Read the previously-written key.
DBString ro_value;
EXPECT_STREQ(DBGet(db, ToDBKey("foo"), &ro_value).data, NULL);
EXPECT_STREQ(ToString(ro_value).c_str(), "foo's value");
// Try to write it again.
EXPECT_EQ(ToString(DBPut(db, ToDBKey("foo"), ToDBSlice("foo's value"))),
"Not implemented: Not supported operation in read only mode.");

DBClose(db);
}
}
6 changes: 4 additions & 2 deletions c-deps/libroach/ccl/encrypted_env_test.cc
Original file line number Diff line number Diff line change
@@ -24,7 +24,8 @@ TEST(EncryptedEnv, ConcurrentAccess) {
auto key_manager = new MemKeyManager(MakeAES128Key(env.get()));
auto stream = new CTRCipherStreamCreator(key_manager, enginepb::Data);

auto file_registry = std::unique_ptr<FileRegistry>(new FileRegistry(env.get(), "/"));
auto file_registry =
std::unique_ptr<FileRegistry>(new FileRegistry(env.get(), "/", false /* read-only */));
EXPECT_OK(file_registry->Load());

std::unique_ptr<rocksdb::Env> encrypted_env(
@@ -138,7 +139,8 @@ TEST(EncryptedEnv, FileOps) {
auto tmpdir = std::unique_ptr<TempDirHandler>(new TempDirHandler());
ASSERT_TRUE(tmpdir->Init());

auto file_registry = std::unique_ptr<FileRegistry>(new FileRegistry(env, tmpdir->Path("")));
auto file_registry =
std::unique_ptr<FileRegistry>(new FileRegistry(env, tmpdir->Path(""), false /* read-only */));
EXPECT_OK(file_registry->Load());

std::unique_ptr<rocksdb::Env> encrypted_env(
9 changes: 7 additions & 2 deletions c-deps/libroach/ccl/key_manager.cc
Original file line number Diff line number Diff line change
@@ -192,10 +192,11 @@ std::unique_ptr<enginepbccl::SecretKey> FileKeyManager::GetKey(const std::string
DataKeyManager::~DataKeyManager() {}

DataKeyManager::DataKeyManager(rocksdb::Env* env, const std::string& db_dir,
int64_t rotation_period)
int64_t rotation_period, bool read_only)
: env_(env),
registry_path_(db_dir + "/" + kKeyRegistryFilename),
rotation_period_(rotation_period) {}
rotation_period_(rotation_period),
read_only_(read_only) {}

rocksdb::Status DataKeyManager::LoadKeysHelper(enginepbccl::DataKeysRegistry* registry) {
rocksdb::Status status = env_->FileExists(registry_path_);
@@ -359,6 +360,10 @@ DataKeyManager::SetActiveStoreKey(std::unique_ptr<enginepbccl::KeyInfo> store_in

rocksdb::Status
DataKeyManager::PersistRegistryLocked(std::unique_ptr<enginepbccl::DataKeysRegistry> reg) {
if (read_only_) {
return rocksdb::Status::InvalidArgument("key manager is read-only, keys cannot be rotated");
}

// Validate before writing.
auto status = KeyManagerUtils::ValidateRegistry(reg.get());
if (!status.ok()) {
6 changes: 5 additions & 1 deletion c-deps/libroach/ccl/key_manager.h
Original file line number Diff line number Diff line change
@@ -111,7 +111,10 @@ class DataKeyManager : public KeyManager {
public:
// `env` is owned by the caller and should be an encrypted Env.
// `db_dir` is the rocksdb directory.
explicit DataKeyManager(rocksdb::Env* env, const std::string& db_dir, int64_t rotation_period);
// If read-only is true, the DataKeyManager can be used to lookup keys but cannot
// change any keys (eg: SetActiveStoreKey will fail).
explicit DataKeyManager(rocksdb::Env* env, const std::string& db_dir, int64_t rotation_period,
bool read_only);

virtual ~DataKeyManager();

@@ -137,6 +140,7 @@ class DataKeyManager : public KeyManager {
rocksdb::Env* env_;
std::string registry_path_;
int64_t rotation_period_;
bool read_only_;

// The registry is read-only and can only be swapped for another one, it cannot be mutated in
// place. mu_ must be held for any registry access.
41 changes: 31 additions & 10 deletions c-deps/libroach/ccl/key_manager_test.cc
Original file line number Diff line number Diff line change
@@ -266,23 +266,23 @@ TEST(DataKeyManager, LoadKeys) {

// Test a missing file first.
{
DataKeyManager dkm(env.get(), "", 0);
DataKeyManager dkm(env.get(), "", 0, false /* read-only */);
EXPECT_OK(dkm.LoadKeys());
ASSERT_EQ(dkm.CurrentKey(), nullptr);
}

// Now a file with random data.
{
ASSERT_OK(rocksdb::WriteStringToFile(env.get(), "blah blah", registry_path));
DataKeyManager dkm(env.get(), "", 0);
DataKeyManager dkm(env.get(), "", 0, false /* read-only */);
EXPECT_ERR(dkm.LoadKeys(), "failed to parse key registry " + registry_path);
ASSERT_OK(env->DeleteFile(registry_path));
}

// Empty file.
{
ASSERT_OK(rocksdb::WriteStringToFile(env.get(), "", registry_path));
DataKeyManager dkm(env.get(), "", 0);
DataKeyManager dkm(env.get(), "", 0, false /* read-only */);
EXPECT_OK(dkm.LoadKeys());
ASSERT_OK(env->DeleteFile(registry_path));
}
@@ -293,7 +293,7 @@ TEST(DataKeyManager, LoadKeys) {
std::string contents;
ASSERT_TRUE(registry.SerializeToString(&contents));
ASSERT_OK(rocksdb::WriteStringToFile(env.get(), contents, registry_path));
DataKeyManager dkm(env.get(), "", 0);
DataKeyManager dkm(env.get(), "", 0, false /* read-only */);
EXPECT_OK(dkm.LoadKeys());
ASSERT_OK(env->DeleteFile(registry_path));
}
@@ -307,7 +307,7 @@ TEST(DataKeyManager, LoadKeys) {
ASSERT_TRUE(registry.SerializeToString(&contents));
ASSERT_OK(rocksdb::WriteStringToFile(env.get(), contents, registry_path));

DataKeyManager dkm(env.get(), "", 0);
DataKeyManager dkm(env.get(), "", 0, false /* read-only */);
EXPECT_ERR(dkm.LoadKeys(), "active store key foobar not found");
ASSERT_OK(env->DeleteFile(registry_path));
}
@@ -321,7 +321,7 @@ TEST(DataKeyManager, LoadKeys) {
ASSERT_TRUE(registry.SerializeToString(&contents));
ASSERT_OK(rocksdb::WriteStringToFile(env.get(), contents, registry_path));

DataKeyManager dkm(env.get(), "", 0);
DataKeyManager dkm(env.get(), "", 0, false /* read-only */);
EXPECT_ERR(dkm.LoadKeys(), "active data key foobar not found");
ASSERT_OK(env->DeleteFile(registry_path));
}
@@ -342,7 +342,7 @@ TEST(DataKeyManager, LoadKeys) {
std::string contents;
ASSERT_TRUE(registry.SerializeToString(&contents));
ASSERT_OK(rocksdb::WriteStringToFile(env.get(), contents, registry_path));
DataKeyManager dkm(env.get(), "", 0);
DataKeyManager dkm(env.get(), "", 0, false /* read-only */);
EXPECT_OK(dkm.LoadKeys());

auto k = dkm.CurrentKey();
@@ -492,7 +492,7 @@ TEST(DataKeyManager, SetStoreKey) {
testKey("plain", "", enginepbccl::Plaintext, now, "data key manager", true, "plain"),
}};

DataKeyManager dkm(env.get(), "", 0);
DataKeyManager dkm(env.get(), "", 0, false /* read-only */);
ASSERT_OK(dkm.LoadKeys());

int test_num = 0;
@@ -532,7 +532,7 @@ TEST(DataKeyManager, SetStoreKey) {
}

// Initialize a new data key manager to load the file.
DataKeyManager tmp_dkm(env.get(), "", 0);
DataKeyManager tmp_dkm(env.get(), "", 0, false /* read-only */);
ASSERT_OK(tmp_dkm.LoadKeys());

if (status.ok()) {
@@ -599,7 +599,7 @@ TEST(DataKeyManager, RotateKey) {
},
};

DataKeyManager dkm(env.get(), "", 10 /* 10 second rotation period */);
DataKeyManager dkm(env.get(), "", 10 /* 10 second rotation period */, false /* read-only */);
ASSERT_OK(dkm.LoadKeys());

int test_num = 0;
@@ -617,4 +617,25 @@ TEST(DataKeyManager, RotateKey) {
ASSERT_NE(active_info, nullptr);
EXPECT_OK(compareNonRandomKeyInfo(active_info->info(), t.active_key));
}

{
// Now try a read-only data key manager.
DataKeyManager ro_dkm(env.get(), "", 10, true);
ASSERT_OK(ro_dkm.LoadKeys());
// Verify that the key matches the last test case.
auto tc = test_cases.back();
auto active_info = ro_dkm.CurrentKey();
ASSERT_NE(active_info, nullptr);
EXPECT_OK(compareNonRandomKeyInfo(active_info->info(), tc.active_key));

// Increase time, check rotation failure, and check key again.
env->SetCurrentTime(tc.current_time + 100);
auto store_info = std::unique_ptr<enginepbccl::KeyInfo>(
new enginepbccl::KeyInfo(keyInfoFromTestKey(tc.store_info)));
EXPECT_ERR(ro_dkm.SetActiveStoreKey(std::move(store_info)),
"key manager is read-only, keys cannot be rotated");
active_info = ro_dkm.CurrentKey();
ASSERT_NE(active_info, nullptr);
EXPECT_OK(compareNonRandomKeyInfo(active_info->info(), tc.active_key));
}
}
11 changes: 11 additions & 0 deletions c-deps/libroach/ccl/testutils.cc
Original file line number Diff line number Diff line change
@@ -8,10 +8,21 @@

#include "testutils.h"
#include <gtest/gtest.h>
#include "ccl/baseccl/encryption_options.pb.h"
#include "crypto_utils.h"

namespace testutils {

std::string MakePlaintextExtraOptions() {
cockroach::ccl::baseccl::EncryptionOptions opts;
opts.mutable_key_files()->set_current_key("plain");
opts.mutable_key_files()->set_old_key("plain");

std::string ret;
opts.SerializeToString(&ret);
return ret;
}

enginepbccl::SecretKey* MakeAES128Key(rocksdb::Env* env) {
int64_t now;
env->GetCurrentTime(&now);
3 changes: 3 additions & 0 deletions c-deps/libroach/ccl/testutils.h
Original file line number Diff line number Diff line change
@@ -17,6 +17,9 @@ namespace enginepbccl = cockroach::ccl::storageccl::engineccl::enginepbccl;

namespace testutils {

// Generate the DBOptions.extra_options string for plaintext keys.
std::string MakePlaintextExtraOptions();

// MakeAES<size>Key creates a SecretKeyObject with a key of the specified size.
// It needs an Env for the current time.
enginepbccl::SecretKey* MakeAES128Key(rocksdb::Env* env);
15 changes: 7 additions & 8 deletions c-deps/libroach/db.cc
Original file line number Diff line number Diff line change
@@ -78,10 +78,10 @@ ScopedStats::~ScopedStats() {
}
}

void BatchSSTablesForCompaction(const std::vector<rocksdb::SstFileMetaData> &sst,
void BatchSSTablesForCompaction(const std::vector<rocksdb::SstFileMetaData>& sst,
rocksdb::Slice start_key, rocksdb::Slice end_key,
uint64_t target_size, std::vector<rocksdb::Range> *ranges) {
int prev = -1; // index of the last compacted sst
uint64_t target_size, std::vector<rocksdb::Range>* ranges) {
int prev = -1; // index of the last compacted sst
uint64_t size = 0;
for (int i = 0; i < sst.size(); ++i) {
size += sst[i].size;
@@ -179,7 +179,8 @@ DBStatus DBOpen(DBEngine** db, DBSlice dir, DBOptions db_opts) {
}

// Create the file registry. It uses the base_env to access the registry file.
auto file_registry = std::unique_ptr<FileRegistry>(new FileRegistry(env_ctx->base_env, db_dir));
auto file_registry =
std::unique_ptr<FileRegistry>(new FileRegistry(env_ctx->base_env, db_dir, db_opts.read_only));

if (db_opts.use_file_registry) {
// We're using the file registry.
@@ -382,10 +383,8 @@ DBStatus DBCompactRange(DBEngine* db, DBSlice start, DBSlice end, bool force_bot
BatchSSTablesForCompaction(sst, start_key, end_key, target_size, &ranges);

for (auto r : ranges) {
rocksdb::Status status = db->rep->CompactRange(
options,
r.start.empty() ? nullptr : &r.start,
r.limit.empty() ? nullptr : &r.limit);
rocksdb::Status status = db->rep->CompactRange(options, r.start.empty() ? nullptr : &r.start,
r.limit.empty() ? nullptr : &r.limit);
if (!status.ok()) {
return ToDBStatus(status);
}
12 changes: 10 additions & 2 deletions c-deps/libroach/file_registry.cc
Original file line number Diff line number Diff line change
@@ -20,8 +20,11 @@ using namespace cockroach;

namespace cockroach {

FileRegistry::FileRegistry(rocksdb::Env* env, const std::string& db_dir)
: env_(env), db_dir_(db_dir), registry_path_(db_dir_ + "/" + kFileRegistryFilename) {}
FileRegistry::FileRegistry(rocksdb::Env* env, const std::string& db_dir, bool read_only)
: env_(env),
db_dir_(db_dir),
read_only_(read_only),
registry_path_(db_dir_ + "/" + kFileRegistryFilename) {}

rocksdb::Status FileRegistry::CheckNoRegistryFile() {
rocksdb::Status status = env_->FileExists(registry_path_);
@@ -233,6 +236,11 @@ rocksdb::Status FileRegistry::MaybeLinkEntry(const std::string& src, const std::
}

rocksdb::Status FileRegistry::PersistRegistryLocked(std::unique_ptr<enginepb::FileRegistry> reg) {
if (read_only_) {
return rocksdb::Status::InvalidArgument(
"file registry is read-only but registry modification attempted");
}

// Serialize and write to file.
std::string contents;
if (!reg->SerializeToString(&contents)) {
4 changes: 3 additions & 1 deletion c-deps/libroach/file_registry.h
Original file line number Diff line number Diff line change
@@ -34,9 +34,10 @@ static const std::string kFileRegistryFilename = "COCKROACHDB_REGISTRY";
// All paths given to FileRegistry should be absolute paths. It converts
// paths within `db_dir` to relative paths internally.
// Paths outside `db_dir` remain absolute.
// If read-only is set, the registry can be used for a read-only DB, but cannot be modified.
class FileRegistry {
public:
FileRegistry(rocksdb::Env* env, const std::string& db_dir);
FileRegistry(rocksdb::Env* env, const std::string& db_dir, bool read_only);
~FileRegistry() {}

// Returns OK if the registry file does not exist.
@@ -77,6 +78,7 @@ class FileRegistry {
private:
rocksdb::Env* env_;
std::string db_dir_;
bool read_only_;
std::string registry_path_;

// Write 'reg' to the registry file, and swap with registry_ if successful. mu_ is held.
48 changes: 45 additions & 3 deletions c-deps/libroach/file_registry_test.cc
Original file line number Diff line number Diff line change
@@ -13,9 +13,9 @@
// permissions and limitations under the License.

#include <vector>
#include "../fmt.h"
#include "../testutils.h"
#include "file_registry.h"
#include "fmt.h"
#include "testutils.h"

using namespace cockroach;

@@ -44,8 +44,50 @@ TEST(FileRegistry, TransformPath) {
for (auto t : test_cases) {
SCOPED_TRACE(fmt::StringPrintf("Testing #%d", test_num++));

FileRegistry reg(nullptr, t.db_dir);
FileRegistry reg(nullptr, t.db_dir, false /* read-only */);
auto out = reg.TransformPath(t.input);
EXPECT_EQ(t.output, out);
}
}

TEST(FileRegistry, FileOps) {
std::unique_ptr<rocksdb::Env> env(rocksdb::NewMemEnv(rocksdb::Env::Default()));

FileRegistry rw_reg(env.get(), "/", false /* read-only */);
ASSERT_OK(rw_reg.CheckNoRegistryFile());
ASSERT_OK(rw_reg.Load());

EXPECT_EQ(nullptr, rw_reg.GetFileEntry("/foo"));
auto entry = std::unique_ptr<enginepb::FileEntry>(new enginepb::FileEntry());
EXPECT_OK(rw_reg.SetFileEntry("/foo", std::move(entry)));
EXPECT_NE(nullptr, rw_reg.GetFileEntry("/foo"));

EXPECT_OK(rw_reg.MaybeLinkEntry("/foo", "/bar"));
EXPECT_NE(nullptr, rw_reg.GetFileEntry("/foo"));
EXPECT_NE(nullptr, rw_reg.GetFileEntry("/bar"));

EXPECT_OK(rw_reg.MaybeRenameEntry("/bar", "/baz"));
EXPECT_NE(nullptr, rw_reg.GetFileEntry("/foo"));
EXPECT_EQ(nullptr, rw_reg.GetFileEntry("/bar"));
EXPECT_NE(nullptr, rw_reg.GetFileEntry("/baz"));

EXPECT_OK(rw_reg.MaybeDeleteEntry("/baz"));
EXPECT_NE(nullptr, rw_reg.GetFileEntry("/foo"));
EXPECT_EQ(nullptr, rw_reg.GetFileEntry("/bar"));
EXPECT_EQ(nullptr, rw_reg.GetFileEntry("/baz"));

// Now try with a read-only registry.
FileRegistry ro_reg(env.get(), "/", true /* read-only */);
// We have a registry file.
EXPECT_ERR(ro_reg.CheckNoRegistryFile(), "registry file .* exists");
ASSERT_OK(ro_reg.Load());

EXPECT_NE(nullptr, ro_reg.GetFileEntry("/foo"));
EXPECT_EQ(nullptr, ro_reg.GetFileEntry("/bar"));
EXPECT_EQ(nullptr, ro_reg.GetFileEntry("/baz"));

// All mutable operations fail.
EXPECT_ERR(ro_reg.MaybeLinkEntry("/foo", "/bar"), "file registry is read-only .*");
EXPECT_ERR(ro_reg.MaybeRenameEntry("/foo", "/baz"), "file registry is read-only .*");
EXPECT_ERR(ro_reg.MaybeDeleteEntry("/foo"), "file registry is read-only .*");
}