GH-29238 [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API #34616

tolleybot · 2023-03-17T19:52:29Z

Rationale for this change

The purpose of this pull request is to support modular encryption in the new Dataset API. See https://docs.google.com/document/d/13EysCNC6-Nu9wnJ8YpdzmD-aMLn4i2KXUJTNqIihy7A/edit# for supporting document.

What changes are included in this PR?

I made improvements to the C++ and Python code to enable the Dataset API to have per-file settings for each file saved. Previously, the Dataset API applied the same encryption properties to all saved files, but now I've updated the code to allow for greater flexibility. In the Python code, I've added support for the changes by updating the ParquetFormat class to accept DatasetEncryptionConfiguration and DatasetDecryptionConfiguration structures. With these changes, you can pass the format object to the write_dataset function, giving you the ability to set unique encryption properties for each file in your Dataset.

Are these changes tested?

Yes, unit tests are included. I have also included a python sample project.

Are there any user-facing changes?

Yes, as stated above the ParquetFormat class has optional parameters for DatasetEncryptionConfiguration and DatasetDecryptionConfiguration through setters and getters. The Dataset now has the option using this to set different file encryption properties per file

Closes: [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API #29238

github-actions · 2023-03-17T19:52:48Z

Closes: [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API #29238

github-actions · 2023-03-17T19:52:50Z

⚠️ GitHub issue #29238 has been automatically assigned in GitHub to PR creator.

github-actions · 2023-03-17T19:57:19Z

⚠️ GitHub issue #29238 has been automatically assigned in GitHub to PR creator.

github-actions · 2023-03-17T19:58:40Z

⚠️ GitHub issue #29238 has been automatically assigned in GitHub to PR creator.

mapleFU

Great work! Remember using clang-format to format cpp code

cpp/src/parquet/properties.h

wjones127

Thanks for starting this @tolleybot. I did a first pass looking through it, and it seems like we could use a bit more testing, both in C++ and in Python.

cpp/src/arrow/dataset/dataset_encryption_test.cc

python/examples/dataset/write_dataset_encrypted.py

cpp/src/arrow/dataset/dataset_encryption_test.cc

wjones127 · 2023-03-20T21:18:44Z

On the C++ formatting: If you install clang-tools 14 (must be that version) and set the path the binaries with CLANG_TOOLS_PATH , then when you reconfigure CMake it will create a format target and a lint target that can be used to check style. (You'll know it's configured right when the CMake configure output says Found ClangTools: /some/path/bin/clang-format-14)

For Python formatting, see these instructions: https://arrow.apache.org/docs/developers/python.html#coding-style

cpp/src/parquet/encryption/dataset_encryption_config.h

cpp/src/arrow/dataset/file_parquet.h

cpp/src/parquet/encryption/dataset_encryption_config.h

cpp/src/arrow/dataset/file_parquet.h

cpp/src/arrow/dataset/dataset_encryption_test.cc

jorisvandenbossche · 2023-10-10T15:14:56Z

@github-actions crossbow submit -g python

github-actions · 2023-10-10T15:18:08Z

Revision: d7a6b55

Submitted crossbow builds: ursacomputing/crossbow @ actions-f652f7cf6d

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.4.1
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.4.1
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

python/pyarrow/tests/test_dataset_encryption.py

jorisvandenbossche · 2023-10-10T17:01:10Z

@github-actions crossbow submit test-conda-python-3.10-pandas-latest

github-actions · 2023-10-10T17:03:50Z

Revision: b323457

Submitted crossbow builds: ursacomputing/crossbow @ actions-6e2d0464d5

Task	Status
test-conda-python-3.10-pandas-latest

anjakefala · 2023-10-10T19:26:26Z

I updated the branch with the latest main, so we can see if that helps with the unrelated failures.

ianmcook · 2023-10-10T19:29:03Z

@github-actions crossbow submit -g python

github-actions · 2023-10-10T19:32:15Z

Revision: ced2ed2

Submitted crossbow builds: ursacomputing/crossbow @ actions-6cee43f1c5

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

pitrou · 2023-10-10T19:42:08Z

@github-actions crossbow submit verifypython*

github-actions · 2023-10-10T19:45:11Z

Revision: ced2ed2

Submitted crossbow builds: ursacomputing/crossbow @ actions-379629a2a0

Task	Status
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-20.04-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

jorisvandenbossche · 2023-10-11T11:06:27Z

It seems that all triggered builds are green!

Thanks @tolleybot and all others involved to get this over the finish line

tolleybot · 2023-10-11T11:54:16Z

Thanks, everyone for all the reviews and work on this!

pitrou · 2023-10-11T12:01:31Z

Thank you @tolleybot for contributing this in the first place :-)

conbench-apache-arrow · 2023-10-14T04:44:05Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 0793432.

There were 7 benchmark results indicating a performance regression:

Commit Run on ursa-i9-9960x at 2023-10-12 08:25:25Z
- tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-02, scale_factor=1
- file-read (R) with compression=uncompressed, dataset=nyctaxi_2010-01, file_type=feather, language=R, output_type=table
and 5 more (see the report linked below)

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

jorisvandenbossche · 2023-10-14T08:12:51Z

FYI I checked the reports above and they are all flakes (the benchmarks reports are stable beyond this commit)

…tion in the new Dataset API (apache#34616) ### Rationale for this change The purpose of this pull request is to support modular encryption in the new Dataset API. See [https://docs.google.com/document/d/13EysCNC6-Nu9wnJ8YpdzmD-aMLn4i2KXUJTNqIihy7A/edit#](url) for supporting document. ### What changes are included in this PR? I made improvements to the C++ and Python code to enable the Dataset API to have per-file settings for each file saved. Previously, the Dataset API applied the same encryption properties to all saved files, but now I've updated the code to allow for greater flexibility. In the Python code, I've added support for the changes by updating the ParquetFormat class to accept DatasetEncryptionConfiguration and DatasetDecryptionConfiguration structures. With these changes, you can pass the format object to the write_dataset function, giving you the ability to set unique encryption properties for each file in your Dataset. ### Are these changes tested? Yes, unit tests are included. I have also included a python sample project. ### Are there any user-facing changes? Yes, as stated above the ParquetFormat class has optional parameters for DatasetEncryptionConfiguration and DatasetDecryptionConfiguration through setters and getters. The Dataset now has the option using this to set different file encryption properties per file * Closes: apache#29238 Lead-authored-by: Don <[email protected]> Co-authored-by: Donald Tolley <[email protected]> Co-authored-by: Joris Van den Bossche <[email protected]> Co-authored-by: anjakefala <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Weston Pace <[email protected]> Co-authored-by: Gang Wu <[email protected]> Co-authored-by: scoder <[email protected]> Co-authored-by: Will Jones <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

github-actions bot added Component: C++ Component: Parquet Component: Python awaiting review Awaiting review labels Mar 17, 2023

mapleFU reviewed Mar 17, 2023

View reviewed changes

cpp/src/parquet/properties.h Outdated Show resolved Hide resolved

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Mar 17, 2023

tolleybot marked this pull request as ready for review March 17, 2023 21:41

tolleybot requested review from AlenkaF, wjones127 and westonpace as code owners March 17, 2023 21:41

mapleFU reviewed Mar 19, 2023

View reviewed changes

cpp/src/parquet/properties.h Outdated Show resolved Hide resolved

wjones127 requested changes Mar 20, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Mar 20, 2023

wgtmac reviewed Mar 21, 2023

View reviewed changes

cpp/src/parquet/encryption/dataset_encryption_config.h Outdated Show resolved Hide resolved

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 22, 2023

westonpace reviewed Mar 23, 2023

View reviewed changes

cpp/src/arrow/dataset/file_parquet.h Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Mar 23, 2023

wgtmac reviewed Mar 29, 2023

View reviewed changes

cpp/src/parquet/encryption/dataset_encryption_config.h Outdated Show resolved Hide resolved

cpp/src/arrow/dataset/file_parquet.h Outdated Show resolved Hide resolved

wgtmac reviewed Mar 29, 2023

View reviewed changes

cpp/src/arrow/dataset/dataset_encryption_test.cc Outdated Show resolved Hide resolved

cpp/src/arrow/dataset/dataset_encryption_test.cc Outdated Show resolved Hide resolved

cpp/src/arrow/dataset/dataset_encryption_test.cc Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Run linter

22033b7

jorisvandenbossche reviewed Oct 10, 2023

View reviewed changes

python/pyarrow/tests/test_dataset_encryption.py Outdated Show resolved Hide resolved

Update python/pyarrow/tests/test_dataset_encryption.py

b323457

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 10, 2023

Merge branch 'main' into dataset_encryption

ced2ed2

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 10, 2023

apache deleted a comment from github-actions bot Oct 10, 2023

jorisvandenbossche merged commit 0793432 into apache:main Oct 11, 2023
33 checks passed

jorisvandenbossche removed the awaiting change review Awaiting change review label Oct 11, 2023

jorisvandenbossche mentioned this pull request Dec 1, 2023

[Python] Minimal pyarrow installation should not require libparquet #39006

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-29238 [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API #34616

GH-29238 [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API #34616

tolleybot commented Mar 17, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

mapleFU left a comment

wjones127 left a comment

wjones127 commented Mar 20, 2023

jorisvandenbossche commented Oct 10, 2023

This comment has been minimized.

github-actions bot commented Oct 10, 2023

jorisvandenbossche commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

anjakefala commented Oct 10, 2023

ianmcook commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

pitrou commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

jorisvandenbossche commented Oct 11, 2023

tolleybot commented Oct 11, 2023

pitrou commented Oct 11, 2023

conbench-apache-arrow bot commented Oct 14, 2023

jorisvandenbossche commented Oct 14, 2023

GH-29238 [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API #34616

GH-29238 [C++][Dataset][Parquet] Support parquet modular encryption in the new Dataset API #34616

Conversation

tolleybot commented Mar 17, 2023 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

mapleFU left a comment

Choose a reason for hiding this comment

wjones127 left a comment

Choose a reason for hiding this comment

wjones127 commented Mar 20, 2023

jorisvandenbossche commented Oct 10, 2023

This comment has been minimized.

github-actions bot commented Oct 10, 2023

jorisvandenbossche commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

anjakefala commented Oct 10, 2023

ianmcook commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

pitrou commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

jorisvandenbossche commented Oct 11, 2023

tolleybot commented Oct 11, 2023

pitrou commented Oct 11, 2023

conbench-apache-arrow bot commented Oct 14, 2023

jorisvandenbossche commented Oct 14, 2023

tolleybot commented Mar 17, 2023 •

edited by github-actions bot

Loading