Code

use row encoding for SortExec#5292

Code

Closed

jaylmiller wants to merge 44 commits intoapache:mainapache/datafusion:mainfrom jaylmiller:masterjaylmiller/arrow-datafusion:masterCopy head branch name to clipboard

+1,356-209

Commits

Commits on Feb 10, 2023

• modify sort_batch to use arrow row format for multi-column sorts

  jaylmiller

  committed
  ec44910

  Copy full SHA for ec44910

Commits on Feb 11, 2023

• use row encoding for in memory partial sorting within SortExec

  jaylmiller

  committed
  c7c43e4

  Copy full SHA for c7c43e4

Commits on Feb 12, 2023

• Revert preserving row encoding changes

  jaylmiller

  committed
  2e1c143

  Copy full SHA for 2e1c143

Commits on Feb 13, 2023

• add bench for SortExec

  jaylmiller

  committed
  2ab03ee

  Copy full SHA for 2ab03ee

• fix preserve partitioning case to run every partition instead of just first

  jaylmiller

  committed
  11be061

  Copy full SHA for 11be061

Commits on Feb 14, 2023

• rough draft: sorting works.

  Show description for 730a89c
  jaylmiller

  committed
  730a89c

  Copy full SHA for 730a89c

• Merge branch 'apache:master' into master

  jaylmiller

  authored
  4b1ea08

  Copy full SHA for 4b1ea08

• Merge branch 'sort-preserve-row-encoding'

  jaylmiller

  committed
  a64705f

  Copy full SHA for a64705f

• remove some todos

  jaylmiller

  committed
  7aaddb5

  Copy full SHA for 7aaddb5

• fix clippy warnings

  jaylmiller

  committed
  96a2e15

  Copy full SHA for 96a2e15

• checkpointing

  jaylmiller

  committed
  a3c632c

  Copy full SHA for a3c632c

• SortPreservingMergeStream emits row encodings when used from SortExec

  jaylmiller

  committed
  c51b23c

  Copy full SHA for c51b23c

Commits on Feb 15, 2023

• spill logic working (w/ a temporary serialization format solution)

  jaylmiller

  committed
  0f7bfc3

  Copy full SHA for 0f7bfc3

• Merge branch 'sort-exec'

  jaylmiller

  committed
  cdf72d8

  Copy full SHA for cdf72d8

• add row encoding sizes to spill calculations.

  Show description for e313134
  jaylmiller

  committed
  e313134

  Copy full SHA for e313134

• clean comments and small todos

  jaylmiller

  committed
  8b2450b

  Copy full SHA for 8b2450b

Commits on Feb 16, 2023

• cleanup SortedStream types

  Show description for 085a871
  jaylmiller

  committed
  085a871

  Copy full SHA for 085a871

• add SortExec input case to each merge bench case

  Show description for 4196a25
  jaylmiller

  committed
  4196a25

  Copy full SHA for 4196a25

• row serialization format

  jaylmiller

  committed
  d4f5c10

  Copy full SHA for d4f5c10

Commits on Feb 17, 2023

• RowBatch construction re-use row ref offsets instead of just appending

  jaylmiller

  committed
  33c611c

  Copy full SHA for 33c611c

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  331d205

  Copy full SHA for 331d205

Commits on Feb 18, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  cba6d30

  Copy full SHA for cba6d30

• dont need to keep array refs if we use rows

  jaylmiller

  committed
  1513f9a

  Copy full SHA for 1513f9a

• dont use channel for SortedSizedStream in sort (emit tuple)

  jaylmiller

  committed
  2ebcbc7

  Copy full SHA for 2ebcbc7

• add unit test for edge case where we skip spilling the row data

  jaylmiller

  committed
  7354952

  Copy full SHA for 7354952

• fix sort bench to actually use full data set in non-preserve partition case

  jaylmiller

  committed
  e6fe175

  Copy full SHA for e6fe175

Commits on Feb 20, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  committed
  353815b

  Copy full SHA for 353815b

• clippy

  jaylmiller

  committed
  ec49492

  Copy full SHA for ec49492

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  60b8e6f

  Copy full SHA for 60b8e6f

Commits on Feb 21, 2023

• add data skewed to first partition case for the tuple sorts in bench

  jaylmiller

  committed
  e430470

  Copy full SHA for e430470

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  a55d34e

  Copy full SHA for a55d34e

• clippy err

  jaylmiller

  committed
  279c6f5

  Copy full SHA for 279c6f5

Commits on Feb 23, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  d42f380

  Copy full SHA for d42f380

Commits on Mar 2, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  eeb1e9c

  Copy full SHA for eeb1e9c

Commits on Mar 3, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  ba08237

  Copy full SHA for ba08237

Commits on Mar 4, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  e80578b

  Copy full SHA for e80578b

• use mergesort in the merge step of sort exec

  jaylmiller

  committed
  b82545e

  Copy full SHA for b82545e

• remove experimental bench cases

  jaylmiller

  committed
  c685a0d

  Copy full SHA for c685a0d

• move gating logic outside of sort_batch

  jaylmiller

  committed
  08b3fe5

  Copy full SHA for 08b3fe5

• dont use row encoding on single batch code path

  jaylmiller

  committed
  790546f

  Copy full SHA for 790546f

Commits on Mar 6, 2023

• batch insertion order fix

  jaylmiller

  committed
  d13912b

  Copy full SHA for d13912b

Commits on Mar 11, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  fe79275

  Copy full SHA for fe79275

Commits on Mar 14, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  0a37892

  Copy full SHA for 0a37892

Commits on Mar 17, 2023

• Merge branch 'apache:main' into master

  jaylmiller

  authored
  d202638

  Copy full SHA for d202638