Code

PARQUET-2430: Add parquet joiner v2#1335

Code

Merged

wgtmac merged 67 commits intoapache:masterapache/parquet-java:masterfrom MaxNevermind:add-parquet-joiner-v2MaxNevermind/parquet-mr:add-parquet-joiner-v2Copy head branch name to clipboardSep 19, 2024

+865-318

Commits

Commits on Jan 28, 2024

• add initial ParquetJoiner implementation

  

  maxim_konstantinov

  committed
  f5144b2

Commits on Feb 1, 2024

• add initial ParquetJoiner implementation

  

  maxim_konstantinov

  committed
  01a08dd

Commits on Feb 12, 2024

• Merge remote-tracking branch 'origin/master' into add-parquet-joiner

  

  maxim_konstantinov

  committed
  28c987c

Commits on Feb 17, 2024

• refactor ParquetJoiner implementation

  

  maxim_konstantinov

  committed
  7ae3505

Commits on Feb 20, 2024

• extend the main test for multiple files on the right

  

  maxim_konstantinov

  committed
  05eb22a

Commits on Feb 22, 2024

• extend the main test for multiple files on the right

  

  maxim_konstantinov

  committed
  6bb950d

• Merge branch 'master' into add-parquet-joiner

  

  maxim_konstantinov

  committed
  87b923c

Commits on Feb 23, 2024

• converge join logic, crate a draft of options and rewriter

  

  maxim_konstantinov

  committed
  f9536c3

Commits on Feb 27, 2024

• move ParquetJoinTest logic to ParquetRewriterTest

  

  maxim_konstantinov

  committed
  d7f11d9

Commits on Mar 1, 2024

• improve Parquet stitching test

  

  maxim_konstantinov

  committed
  e8e7ffe

Commits on Mar 6, 2024

• remove custom ParquetRewriter constructor

  

  maxim_konstantinov

  committed
  3ee946c

• remove custom ParquetRewriter constructor

  

  maxim_konstantinov

  committed
  fd409c4

Commits on Mar 12, 2024

• refactor ParquetRewriter

  

  maxim_konstantinov

  committed
  5a98219

Commits on Mar 14, 2024

• apply spotless and address PR comments

  

  maxim_konstantinov

  committed
  7b2fd1a

Commits on Mar 15, 2024

• move extra column writing into processBlocksFromReader

  

  maxim_konstantinov

  committed
  8da8291

Commits on Mar 16, 2024

• add getInputFiles back

  

  maxim_konstantinov

  committed
  68e41ba

• Merge remote-tracking branch 'fork/master' into add-parquet-joiner

  

  maxim_konstantinov

  committed
  98b9b23

• fix extra ParquetRewriter constructor so tests can pass

  

  maxim_konstantinov

  committed
  6d2c222

Commits on Mar 20, 2024

• remove not needed TODOs

  

  maxim_konstantinov

  committed
  883e935

Commits on Mar 24, 2024

• address PR comments

  

  maxim_konstantinov

  committed
  8ef36b5

Commits on Apr 11, 2024

• Merge remote-tracking branch 'origin/master' into add-parquet-joiner

  

  maxim_konstantinov

  committed
  79cc2b8

• rename inputFilesR to inputFilesToJoin

  

  maxim_konstantinov

  committed
  0bbf72f

• rename inputFilesR to inputFilesToJoinColumns

  

  maxim_konstantinov

  committed
  ca53bff

• add getParquetInputFiles listing to the rewrite start logging

  

  maxim_konstantinov

  committed
  1e7998a

Commits on Apr 28, 2024

• redesign file joiner in ParquetRewriter

  

  maxim_konstantinov

  committed
  2ee9b40

• Merge remote-tracking branch 'origin/master' into add-parquet-joiner-v2

  

  maxim_konstantinov

  committed
  fc32dfd

• redesign file joiner in ParquetRewriter

  

  maxim_konstantinov

  committed
  db52c85

• redesign file joiner in ParquetRewriter

  

  maxim_konstantinov

  committed
  9057e91

• redesign file joiner in ParquetRewriter

  

  maxim_konstantinov

  committed
  5b055c0

• uncomment some code

  

  maxim_konstantinov

  committed
  b70f88f

Commits on May 4, 2024

• fix ParquetRewriter joiner test

  

  maxim_konstantinov

  committed
  270126b

Commits on Jul 25, 2024

• Merge remote-tracking branch 'refs/remotes/origin/master' into add-parquet-joiner-v2

  Show description for 008cb40
  

  maxim_konstantinov

  committed
  008cb40

• add initial ParquetJoiner implementation

  

  maxim_konstantinov

  committed
  0dc1793

Commits on Jul 31, 2024

• add initial ParquetJoiner implementation

  

  maxim_konstantinov

  committed
  a53d108

Commits on Aug 6, 2024

• typo

  

  maxim_konstantinov

  committed
  4da0b85

• typo

  

  maxim_konstantinov

  committed
  c5c7b38

• typo

  

  maxim_konstantinov

  committed
  92c95db

• typo

  

  maxim_konstantinov

  committed
  86f7a4c

• docs

  

  maxim_konstantinov

  committed
  73a4af4

Commits on Aug 7, 2024

• typo

  

  maxim_konstantinov

  committed
  18feef4

• add getExtraMetadata()

  

  maxim_konstantinov

  committed
  b24bffa

• extract ensureRowCount()

  

  maxim_konstantinov

  committed
  21a5926

• typo

  

  maxim_konstantinov

  committed
  c521a95

• typo

  

  maxim_konstantinov

  committed
  1ea6755

• add logging into getSchema()

  

  maxim_konstantinov

  committed
  f2e01a2

• typo

  

  maxim_konstantinov

  committed
  d393125

• add closing of input files readers

  

  maxim_konstantinov

  committed
  64d3bb2

• fix RewriteOptions builder for inputFilesToJoin

  

  maxim_konstantinov

  committed
  f50666a

• Merge remote-tracking branch 'refs/remotes/origin/master' into add-parquet-joiner-v2

  

  maxim_konstantinov

  committed
  d306336

• fix ParquetRewriter constructor

  

  maxim_konstantinov

  committed
  ae9589d

Commits on Aug 14, 2024

• extend tests for ParquetRewriter

  

  maxim_konstantinov

  committed
  9157960

• spotless

  

  maxim_konstantinov

  committed
  3b722e4

Commits on Aug 15, 2024

• refactor ParquetRewriterTest

  

  maxim_konstantinov

  committed
  bdba14c

Commits on Aug 16, 2024

• add tests into ParquetRewriterTest

  

  maxim_konstantinov

  committed
  a89eba6

Commits on Aug 26, 2024

• Merge remote-tracking branch 'origin/master' into add-parquet-joiner-v2

  

  maxim_konstantinov

  committed
  57432ee

• extend tests in ParquetRewriterTest for joiner part

  

  maxim_konstantinov

  committed
  f674bcf

Commits on Aug 27, 2024

• add testMergeFilesToJoinWithDifferentRowCount test into ParquetRewriterTest

  

  maxim_konstantinov

  committed
  8514f39

Commits on Aug 29, 2024

• Merge remote-tracking branch 'origin/master' into add-parquet-joiner-v2

  

  maxim_konstantinov

  committed
  0aaf963

• add testOneInputFileManyInputFilesToJoin with and without JoinColumnsOverwrite

  

  maxim_konstantinov

  committed
  4340c42

Commits on Aug 31, 2024

• Merge remote-tracking branch 'origin/master' into add-parquet-joiner-v2

  Show description for e475648
  

  maxim_konstantinov

  committed
  e475648

• add encrypt validation into ParquetRewriterTest's testOneInputFileManyInputFilesToJoin

  

  maxim_konstantinov

  committed
  bb42979

Commits on Sep 8, 2024

• refactor ParquetRewriter slightly to address PR comments

  

  maxim_konstantinov

  committed
  5b97a4c

Commits on Sep 9, 2024

• add javadoc to ParquetRewriter

  

  maxim_konstantinov

  committed
  27ba73b

Commits on Sep 10, 2024

• add javadoc to ParquetRewriter

  

  maxim_konstantinov

  committed
  07f1e74

Commits on Sep 13, 2024

• fix javadoc in ParquetRewriter to comply with Maven javadoc plugin

  

  maxim_konstantinov

  committed
  e96c022

• fix javadoc in ParquetRewriter to comply with Maven javadoc plugin

  

  maxim_konstantinov

  committed
  d1c1d76

• fix javadoc in ParquetRewriter to comply with Maven javadoc plugin

  

  maxim_konstantinov

  committed
  9de20d7