Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[encryption] Manage file key by the file to encrypt itself #16

Merged
merged 9 commits into from
Sep 15, 2023

Conversation

acelyc111
Copy link
Member

@acelyc111 acelyc111 commented Aug 16, 2023

apache/incubator-pegasus#1575

After all encryption related patches been cherry-picked from
tikv and
merged, now we will improve the encrytion, including:

  • Fix action job build-linux-encrypted_env-no_compression-no_openssl
    to build binaries without openssl and compression libs correctly.
  • Fix action job build-linux-encrypted_env-openssl to export the
    ENCRYPTED_ENV enviroment variable correctly.
  • Don not skip tests which are skipped by TiKV.
  • Refactor AESCTRCipherStream and AESEncryptionProvider to support
    manage file key by the file itself, according to the design docs in
    Data at rest encryption.
  • Remove all KeyManager related codes.
  • Replace KeyManager tests by AES encryption tests.
  • Refactor encryption/encryption_test.cc and add more tests.
  • Make it possible to construct AESEncryptionProvider object via
    EncryptionProvider::CreateFromString() by registering a
    factory in "encryption" library.
    It's possible to construct an object by URI: AES, AES://test or
    AES:<instance_key>,<EncryptionMethod>.
  • ldb tool support to parse --fs_uri flags as the URI mentioned above.
  • Add tests to create AESEncryptionProvider object in
    CreateEncryptedEnvTest.CreateEncryptedFileSystem
  • db_bench support to run benchmark with encryption enabled, by adding
    new flags for db_bench, they are encryption_method and encryption_instance_key.
  • Move code from the exported header directory (i.e. include/rocksdb/encryption.h)
    to rocksdb internal (i.e. encryption/encryption.h), do not expose them to users.
  • Code format.

Review hint: #17 shows all the code changes
from the base branch (i.e. pegasus-kv:v8.3.2-pegasus), you can review it together to
make sure the request branch acelyc111:pk_enc_new doesn't have vice effect on the base.

Manual test:

// Generate some data.
./db_bench --encryption_method=AES128CTR --encryption_instance_key=test_instance_key  --num=10000

// Dump WAL OK
./tools/ldb --fs_uri="provider=AES; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES://test; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log

// Dump WAL failed. Pass bad provider parameters to --fs_uri, e.g.
./tools/ldb --fs_uri="provider=AES1:test_instance_key,1AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:bad_test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:test_instance_key,AES192CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log

// The same to other ldb tools.

@empiredan empiredan enabled auto-merge (squash) September 15, 2023 03:25
@acelyc111 acelyc111 disabled auto-merge September 15, 2023 03:48
@acelyc111 acelyc111 merged commit e9206ed into pegasus-kv:v8.3.2-pegasus-encrypt Sep 15, 2023
acelyc111 added a commit that referenced this pull request Sep 15, 2023
apache/incubator-pegasus#1575

After all encryption related patches been cherry-picked from
[tikv](https://github.com/tikv/rocksdb/commits/6.29.tikv) and
merged, now we will improve the encrytion, including:
- Fix action job `build-linux-encrypted_env-no_compression-no_openssl`
  to build binaries without openssl and compression libs correctly.
- Fix action job `build-linux-encrypted_env-openssl` to export the
  `ENCRYPTED_ENV` enviroment variable correctly.
- Don not skip tests which are skipped by TiKV.
- Refactor `AESCTRCipherStream` and `AESEncryptionProvider` to support
  manage file key by the file itself, according to the design docs in
[Data at rest
encryption](apache/incubator-pegasus#1575).
- Remove all KeyManager related codes.
- Replace KeyManager tests by AES encryption tests.
- Refactor encryption/encryption_test.cc and add more tests.
- Make it possible to construct AESEncryptionProvider object via
  `EncryptionProvider::CreateFromString()` by registering a
  factory in "encryption" library.
  It's possible to construct an object by URI: `AES`, `AES://test` or
  `AES:<instance_key>,<EncryptionMethod>`.
- `ldb` tool support to parse `--fs_uri` flags as the URI mentioned
above.
- Add tests to create AESEncryptionProvider object in
  `CreateEncryptedEnvTest.CreateEncryptedFileSystem`
- `db_bench` support to run benchmark with encryption enabled, by adding
new flags for `db_bench`, they are `encryption_method` and
`encryption_instance_key`.
- Move code from the exported header directory (i.e.
include/rocksdb/encryption.h)
to rocksdb internal (i.e. encryption/encryption.h), do not expose them
to users.
- Code format.

Review hint: #17 shows all the
code changes
from the base branch (i.e. `pegasus-kv:v8.3.2-pegasus`), you can review
it together to
make sure the request branch `acelyc111:pk_enc_new` doesn't have vice
effect on the base.

Manual test:
```
// Generate some data.
./db_bench --encryption_method=AES128CTR --encryption_instance_key=test_instance_key  --num=10000

// Dump WAL OK
./tools/ldb --fs_uri="provider=AES; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES://test; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log

// Dump WAL failed. Pass bad provider parameters to --fs_uri, e.g.
./tools/ldb --fs_uri="provider=AES1:test_instance_key,1AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:bad_test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:test_instance_key,AES192CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log

// The same to other ldb tools.

```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants