Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support to read StructArray from parquet #547

Merged
merged 6 commits into from
Nov 17, 2021
Merged

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Oct 22, 2021

This PR adds support to read (nested) StructArray from parquet.

Integration test against pyarrow included.

@jorgecarleitao jorgecarleitao added the feature A new feature label Oct 22, 2021
@jorgecarleitao jorgecarleitao marked this pull request as ready for review October 24, 2021 06:30
@jorgecarleitao jorgecarleitao marked this pull request as draft October 24, 2021 06:40
@jorgecarleitao jorgecarleitao marked this pull request as ready for review October 29, 2021 15:36
@codecov
Copy link

codecov bot commented Oct 29, 2021

Codecov Report

Merging #547 (2a48bbf) into main (73ee16d) will increase coverage by 0.07%.
The diff coverage is 76.54%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #547      +/-   ##
==========================================
+ Coverage   79.37%   79.45%   +0.07%     
==========================================
  Files         401      403       +2     
  Lines       24966    25016      +50     
==========================================
+ Hits        19817    19876      +59     
+ Misses       5149     5140       -9     
Impacted Files Coverage Δ
src/io/parquet/write/binary/basic.rs 69.31% <33.33%> (ø)
src/io/parquet/read/binary/mod.rs 37.14% <37.14%> (ø)
src/io/parquet/read/primitive/mod.rs 41.02% <58.82%> (-10.04%) ⬇️
src/io/parquet/read/record_batch.rs 79.74% <64.28%> (-1.51%) ⬇️
src/io/parquet/read/nested_utils.rs 77.45% <70.00%> (-5.55%) ⬇️
src/io/parquet/read/boolean/mod.rs 73.68% <73.68%> (ø)
src/io/parquet/read/mod.rs 43.41% <85.71%> (+11.80%) ⬆️
src/io/parquet/read/binary/basic.rs 67.72% <100.00%> (+4.33%) ⬆️
src/io/parquet/read/binary/nested.rs 50.00% <100.00%> (-3.17%) ⬇️
src/io/parquet/read/boolean/basic.rs 67.34% <100.00%> (-5.54%) ⬇️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 73ee16d...2a48bbf. Read the comment docs.

@jorgecarleitao jorgecarleitao force-pushed the parquet_struct branch 2 times, most recently from 22a187d to 711a822 Compare October 29, 2021 18:30
@jorgecarleitao jorgecarleitao merged commit 9d4107c into main Nov 17, 2021
@jorgecarleitao jorgecarleitao deleted the parquet_struct branch November 17, 2021 05:07
@jorgecarleitao jorgecarleitao changed the title Added support to read struct arrays from parquet Added support to read structArray from parquet Nov 17, 2021
@jorgecarleitao jorgecarleitao changed the title Added support to read structArray from parquet Added support to read StructArray from parquet Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant