GH-34138: [C++][Parquet] Fix parsing stats from min_value/max_value #34112

wgtmac · 2023-02-10T03:57:23Z

Rationale for this change

The code below does not read from stats.min_value/max_value at all.

// Extracts encoded statistics from V1 and V2 data page headers
template <typename H>
EncodedStatistics ExtractStatsFromHeader(const H& header) {
  EncodedStatistics page_statistics;
  if (!header.__isset.statistics) {
    return page_statistics;
  }
  const format::Statistics& stats = header.statistics;
  if (stats.__isset.max) {
    page_statistics.set_max(stats.max);
  }
  if (stats.__isset.min) {
    page_statistics.set_min(stats.min);
  }
  if (stats.__isset.null_count) {
    page_statistics.set_null_count(stats.null_count);
  }
  if (stats.__isset.distinct_count) {
    page_statistics.set_distinct_count(stats.distinct_count);
  }
  return page_statistics;
}

What changes are included in this PR?

Do similar thing from parquet-mr to check and read min_value/max_value from thrift stats.

Are these changes tested?

Some test cases fail after the fix. Fixed them to make sure it is covered.

Are there any user-facing changes?

No.

Closes: [C++][Parquet] Fix parsing stats from min_value/max_value #34138

…alue

github-actions · 2023-02-10T03:57:46Z

Closes: [C++][Parquet] Support min_value and max_value Statistics #14870

github-actions · 2023-02-10T03:57:48Z

⚠️ GitHub issue #14870 has been automatically assigned in GitHub to PR creator.

wgtmac · 2023-02-10T05:30:23Z

@pitrou @wjones127 @westonpace Could you please take a look?

mapleFU

Rest LGTM

mapleFU · 2023-02-10T06:34:03Z

cpp/src/parquet/column_reader.cc

@@ -211,10 +211,15 @@ EncodedStatistics ExtractStatsFromHeader(const H& header) {
    return page_statistics;
  }
  const format::Statistics& stats = header.statistics;
-  if (stats.__isset.max) {
+  // Use the new V2 min-max statistics over the former one if it is filled


Previously, page_statistics will handle min-max separately. This patch changes it to once have all min-max, otherwise, cannot use min-max

Yes, page stats without either min or max is corrupted and cannot be used. Check parquet-mr for detail: https://github.com/apache/parquet-mr/blob/5290bd5e0ee5dc30db0576e2bfc6eea335c465cf/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L797

Although in parquet.thrift, min-max can exist only one. But I think handling it like this is ok

So we revert back to previous mode that only has min or max is ok?

wjones127

This looks good. @wgtmac before we merge, could you create a new issue for this bug fix? Also, if you could create issues for the TODOs as well, that would be appreciated.

westonpace · 2023-02-10T20:09:16Z

Does this mean the following (testing my understanding)?

If min is set but max is not set then we should ignore both min and max

wjones127 · 2023-02-10T21:17:28Z

Does this mean the following (testing my understanding)?

If min is set but max is not set then we should ignore both min and max

Yes, it does mean we will. Do you foresee that as an issue? It sounds like Java implementation takes the same approach.

github-actions · 2023-02-11T03:16:38Z

Closes: [C++][Parquet] Fix parsing stats from min_value/max_value #34138

github-actions · 2023-02-11T03:16:41Z

⚠️ GitHub issue #34138 has been automatically assigned in GitHub to PR creator.

wgtmac · 2023-02-11T03:29:46Z

Does this mean the following (testing my understanding)?

If min is set but max is not set then we should ignore both min and max

When only one of min and max exists, it usually happens when a binary value has an extreme length or a floating value has NaN. In this case, the stats provide little value and make it tricker to use.

wgtmac · 2023-02-11T03:30:30Z

This looks good. @wgtmac before we merge, could you create a new issue for this bug fix? Also, if you could create issues for the TODOs as well, that would be appreciated.

I have created a new issue for this PR and updated the title.

A separate issue has been created for the TODO items: #34139

Thanks for the review! @wjones127

mapleFU

Rest LGTM

mapleFU · 2023-02-13T08:18:07Z

cpp/src/parquet/column_reader.cc

+    page_statistics.set_max(stats.max_value);
+    page_statistics.set_min(stats.min_value);
+  } else if (stats.__isset.max && stats.__isset.min) {
+    // TODO: check created_by to see if it is corrupted for some types.


I've no problem here, but just curious, does this code meaning the parquet-mr's CorruptStatistics.shouldIgnoreStatistics? (It's really trickey...)

wgtmac · 2023-02-13T14:43:58Z

Gentle ping @wjones127 @pitrou

Once this gets merged, I will rebase #34107 which is blocked by it.

emkornfield · 2023-02-15T21:10:28Z

CC @fatemehp

westonpace · 2023-02-16T20:41:20Z

Yes, it does mean we will. Do you foresee that as an issue? It sounds like Java implementation takes the same approach.

In datasets, for row group statistics, we recently added a check that was roughly...

if (is_nan(min) && is_nan(max)) {
  // Ignore statistics
} else if (is_nan(min)) {
  // Assume x <= max
} else if(is_nan(max)) {
  // Assume x >= min
} else {
  // Assume min <= x <= max
}

In other words, if one of min or max is NaN then we still use the other side of the equality. I think my primary concern is to validate that is a safe assumption. In other words, I want to make sure we aren't using garbage data in our handling of row groups.

wjones127 · 2023-02-16T21:59:08Z

@westonpace that makes sense.

When only one of min and max exists, it usually happens when a binary value has an extreme length or a floating value has NaN. In this case, the stats provide little value and make it tricker to use.

It seems like we do have handling for these two cases. See Weston's message for NaN handling and max_statistics_size on WriterProperties. Based on that, I'd actually prefer we keep the ability to parse just the min or max if only one is available.

wjones127 · 2023-02-16T22:00:42Z

cpp/src/parquet/column_reader.cc

+  if (stats.__isset.max_value && stats.__isset.min_value) {
+    // TODO: check if the column_order is TYPE_DEFINED_ORDER.
+    page_statistics.set_max(stats.max_value);
+    page_statistics.set_min(stats.min_value);


Suggested change

if (stats.__isset.max_value && stats.__isset.min_value) {

// TODO: check if the column_order is TYPE_DEFINED_ORDER.

page_statistics.set_max(stats.max_value);

page_statistics.set_min(stats.min_value);

if (stats.__isset.max_value || stats.__isset.min_value) {

// TODO: check if the column_order is TYPE_DEFINED_ORDER.

if (stats.__isset.max_value) {

page_statistics.set_max(stats.max_value);

}

if (stats.__isset.min_value) {

page_statistics.set_min(stats.min_value);

}

Fixed. Please take a look again. Thanks @wjones127 !

westonpace · 2023-02-17T01:54:02Z

It seems like we do have handling for these two cases.

Just for clarity, the PR I linked (and my thoughts) were about how we currently handle row group statistics. I'm not sure if the rules are identical for page statistics. I mainly wanted to make sure my understanding of the row group statistics wasn't invalid.

wgtmac · 2023-02-17T02:09:38Z

It seems like we do have handling for these two cases.

Just for clarity, the PR I linked (and my thoughts) were about how we currently handle row group statistics. I'm not sure if the rules are identical for page statistics. I mainly wanted to make sure my understanding of the row group statistics wasn't invalid.

IIUC, row group statistics are aggregated from page statistics so they should share the same rules. The parquet thrift message definition does allow only one side of min or max exist:

/**
 * Statistics per row group and per page
 * All fields are optional.
 */
struct Statistics {
   /**
    * DEPRECATED: min and max value of the column. Use min_value and max_value.
    *
    * Values are encoded using PLAIN encoding, except that variable-length byte
    * arrays do not include a length prefix.
    *
    * These fields encode min and max values determined by signed comparison
    * only. New files should use the correct order for a column's logical type
    * and store the values in the min_value and max_value fields.
    *
    * To support older readers, these may be set when the column order is
    * signed.
    */
   1: optional binary max;
   2: optional binary min;
   /** count of null value in the column */
   3: optional i64 null_count;
   /** count of distinct values occurring */
   4: optional i64 distinct_count;
   /**
    * Min and max values for the column, determined by its ColumnOrder.
    *
    * Values are encoded using PLAIN encoding, except that variable-length byte
    * arrays do not include a length prefix.
    */
   5: optional binary max_value;
   6: optional binary min_value;
}

On the other side, the story of page index is different. The column index definition does require existence of both min and max values if it is not a null page:

/**
 * Description for ColumnIndex.
 * Each <array-field>[i] refers to the page at OffsetIndex.page_locations[i]
 */
struct ColumnIndex {
  /**
   * A list of Boolean values to determine the validity of the corresponding
   * min and max values. If true, a page contains only null values, and writers
   * have to set the corresponding entries in min_values and max_values to
   * byte[0], so that all lists have the same length. If false, the
   * corresponding entries in min_values and max_values must be valid.
   */
  1: required list<bool> null_pages

  /**
   * Two lists containing lower and upper bounds for the values of each page
   * determined by the ColumnOrder of the column. These may be the actual
   * minimum and maximum values found on a page, but can also be (more compact)
   * values that do not exist on a page. For example, instead of storing ""Blart
   * Versenwald III", a writer may set min_values[i]="B", max_values[i]="C".
   * Such more compact values must still be valid values within the column's
   * logical type. Readers must make sure that list entries are populated before
   * using them by inspecting null_pages.
   */
  2: required list<binary> min_values
  3: required list<binary> max_values

  /**
   * Stores whether both min_values and max_values are ordered and if so, in
   * which direction. This allows readers to perform binary searches in both
   * lists. Readers cannot assume that max_values[i] <= min_values[i+1], even
   * if the lists are ordered.
   */
  4: required BoundaryOrder boundary_order

  /** A list containing the number of null values for each page **/
  5: optional list<i64> null_counts
}

So I am fine with parsing only one side min or max values from page/row group statistics. @westonpace @wjones127

wgtmac · 2023-02-17T04:32:47Z

The CI build is failed due to a recent branch rename and will be fixed by #34218

wgtmac · 2023-02-17T04:33:23Z

@westonpace Could you please take another pass?

…alue (apache#34112) ### Rationale for this change The code below does not read from stats.min_value/max_value at all. ```cpp // Extracts encoded statistics from V1 and V2 data page headers template <typename H> EncodedStatistics ExtractStatsFromHeader(const H& header) { EncodedStatistics page_statistics; if (!header.__isset.statistics) { return page_statistics; } const format::Statistics& stats = header.statistics; if (stats.__isset.max) { page_statistics.set_max(stats.max); } if (stats.__isset.min) { page_statistics.set_min(stats.min); } if (stats.__isset.null_count) { page_statistics.set_null_count(stats.null_count); } if (stats.__isset.distinct_count) { page_statistics.set_distinct_count(stats.distinct_count); } return page_statistics; } ``` ### What changes are included in this PR? Do similar thing from parquet-mr to check and read min_value/max_value from thrift stats. ### Are these changes tested? Some test cases fail after the fix. Fixed them to make sure it is covered. ### Are there any user-facing changes? No. * Closes: apache#34138 Authored-by: Gang Wu <[email protected]> Signed-off-by: Will Jones <[email protected]>

ursabot · 2023-02-17T21:24:14Z

Benchmark runs are scheduled for baseline = 1264e40 and contender = 8e5e438. 8e5e438 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.34% ⬆️0.03%] test-mac-arm
[Finished ⬇️1.02% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.54% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 8e5e438d ec2-t3-xlarge-us-east-2
[Failed] 8e5e438d test-mac-arm
[Finished] 8e5e438d ursa-i9-9960x
[Finished] 8e5e438d ursa-thinkcentre-m75q
[Finished] 1264e409 ec2-t3-xlarge-us-east-2
[Finished] 1264e409 test-mac-arm
[Finished] 1264e409 ursa-i9-9960x
[Finished] 1264e409 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

…alue (apache#34112) ### Rationale for this change The code below does not read from stats.min_value/max_value at all. ```cpp // Extracts encoded statistics from V1 and V2 data page headers template <typename H> EncodedStatistics ExtractStatsFromHeader(const H& header) { EncodedStatistics page_statistics; if (!header.__isset.statistics) { return page_statistics; } const format::Statistics& stats = header.statistics; if (stats.__isset.max) { page_statistics.set_max(stats.max); } if (stats.__isset.min) { page_statistics.set_min(stats.min); } if (stats.__isset.null_count) { page_statistics.set_null_count(stats.null_count); } if (stats.__isset.distinct_count) { page_statistics.set_distinct_count(stats.distinct_count); } return page_statistics; } ``` ### What changes are included in this PR? Do similar thing from parquet-mr to check and read min_value/max_value from thrift stats. ### Are these changes tested? Some test cases fail after the fix. Fixed them to make sure it is covered. ### Are there any user-facing changes? No. * Closes: apache#34138 Authored-by: Gang Wu <[email protected]> Signed-off-by: Will Jones <[email protected]>

apacheGH-14870: [C++][Parquet] Fix parsing stats from min_value/max_v…

f5c8e79

…alue

wgtmac requested a review from wjones127 as a code owner February 10, 2023 03:57

github-actions bot added Component: C++ Component: Parquet labels Feb 10, 2023

wgtmac mentioned this pull request Feb 10, 2023

GH-34106: [C++][Parquet] Fix updating page stats for WriteArrowDictionary #34107

Merged

mapleFU reviewed Feb 10, 2023

View reviewed changes

wjones127 approved these changes Feb 10, 2023

View reviewed changes

wgtmac changed the title ~~GH-14870: [C++][Parquet] Fix parsing stats from min_value/max_value~~ GH-34138: [C++][Parquet] Fix parsing stats from min_value/max_value Feb 11, 2023

wgtmac mentioned this pull request Feb 11, 2023

[C++][Parquet] Ignore corrupted or invalid statistics #34139

Open

mapleFU approved these changes Feb 13, 2023

View reviewed changes

jorisvandenbossche mentioned this pull request Feb 15, 2023

[Python] Expose more metadata in pyarrow.parquet.ParquetFile.metadata #34180

Open

wjones127 requested changes Feb 16, 2023

View reviewed changes

support parsing only one side of min/max

ccdcbbb

wgtmac requested a review from wjones127 February 17, 2023 02:17

wjones127 approved these changes Feb 17, 2023

View reviewed changes

wjones127 merged commit 8e5e438 into apache:main Feb 17, 2023

wgtmac mentioned this pull request Jul 23, 2024

[C++][Parquet] min-max Statistics doesn't work well when one of min-max being truncated #43382

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-34138: [C++][Parquet] Fix parsing stats from min_value/max_value #34112

GH-34138: [C++][Parquet] Fix parsing stats from min_value/max_value #34112

wgtmac commented Feb 10, 2023 •

edited

Loading

github-actions bot commented Feb 10, 2023

github-actions bot commented Feb 10, 2023

wgtmac commented Feb 10, 2023

mapleFU left a comment

mapleFU Feb 10, 2023

wgtmac Feb 10, 2023

mapleFU Feb 10, 2023

mapleFU Feb 17, 2023

wjones127 left a comment

westonpace commented Feb 10, 2023

wjones127 commented Feb 10, 2023

github-actions bot commented Feb 11, 2023

github-actions bot commented Feb 11, 2023

wgtmac commented Feb 11, 2023

wgtmac commented Feb 11, 2023

mapleFU left a comment

mapleFU Feb 13, 2023

wgtmac commented Feb 13, 2023

emkornfield commented Feb 15, 2023

westonpace commented Feb 16, 2023

wjones127 commented Feb 16, 2023

wjones127 Feb 16, 2023

wgtmac Feb 17, 2023

westonpace commented Feb 17, 2023

wgtmac commented Feb 17, 2023

wgtmac commented Feb 17, 2023 •

edited

Loading

wgtmac commented Feb 17, 2023

ursabot commented Feb 17, 2023

GH-34138: [C++][Parquet] Fix parsing stats from min_value/max_value #34112

GH-34138: [C++][Parquet] Fix parsing stats from min_value/max_value #34112

Conversation

wgtmac commented Feb 10, 2023 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Feb 10, 2023

github-actions bot commented Feb 10, 2023

wgtmac commented Feb 10, 2023

mapleFU left a comment

Choose a reason for hiding this comment

mapleFU Feb 10, 2023

Choose a reason for hiding this comment

wgtmac Feb 10, 2023

Choose a reason for hiding this comment

mapleFU Feb 10, 2023

Choose a reason for hiding this comment

mapleFU Feb 17, 2023

Choose a reason for hiding this comment

wjones127 left a comment

Choose a reason for hiding this comment

westonpace commented Feb 10, 2023

wjones127 commented Feb 10, 2023

github-actions bot commented Feb 11, 2023

github-actions bot commented Feb 11, 2023

wgtmac commented Feb 11, 2023

wgtmac commented Feb 11, 2023

mapleFU left a comment

Choose a reason for hiding this comment

mapleFU Feb 13, 2023

Choose a reason for hiding this comment

wgtmac commented Feb 13, 2023

emkornfield commented Feb 15, 2023

westonpace commented Feb 16, 2023

wjones127 commented Feb 16, 2023

wjones127 Feb 16, 2023

Choose a reason for hiding this comment

wgtmac Feb 17, 2023

Choose a reason for hiding this comment

westonpace commented Feb 17, 2023

wgtmac commented Feb 17, 2023

wgtmac commented Feb 17, 2023 • edited Loading

wgtmac commented Feb 17, 2023

ursabot commented Feb 17, 2023

wgtmac commented Feb 10, 2023 •

edited

Loading

wgtmac commented Feb 17, 2023 •

edited

Loading