Allow forcing delta lake to recalculate all statistics #16634

homar · 2023-03-20T10:48:17Z

Description

Release notes

(x) Release notes are required, with the following suggested text:

# Delat Lake
* Add support for recalculating all statistics in `ANALYZE` statement. ({issue}`15968`)

pajaks · 2023-03-22T14:04:56Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/AnalyzeHandle.java


    @JsonCreator
    public AnalyzeHandle(
            @JsonProperty("initialAnalyze") boolean initialAnalyze,
            @JsonProperty("startTime") Optional<Instant> filesModifiedAfter,
-            @JsonProperty("columns") Optional<Set<String>> columns)
+            @JsonProperty("columns") Optional<Set<String>> columns,
+            @JsonProperty("recalculateAllStatistics") Optional<Boolean> recalculateAllStatistics)


Can we use initialAnalyze instead of adding new field?

Could we have an enum mode with INITIAL, INCREMENTAL, and FULL_REFRESH options?

pajaks · 2023-03-22T14:17:05Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeSplitManager.java

@@ -139,7 +139,8 @@ private Stream<DeltaLakeSplit> getSplits(
        // per file.
        boolean splittable = tableHandle.getWriteType().isEmpty();
        AtomicInteger remainingInitialSplits = new AtomicInteger(maxInitialSplits);
-        Optional<Instant> filesModifiedAfter = tableHandle.getAnalyzeHandle().flatMap(AnalyzeHandle::getFilesModifiedAfter);
+        Optional<Instant> recalculateAllStatistics = tableHandle.getAnalyzeHandle().flatMap(AnalyzeHandle::getFilesModifiedAfter);


Misleading name change

pajaks · 2023-03-22T14:27:19Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeSplitManager.java

@@ -165,7 +166,7 @@ private Stream<DeltaLakeSplit> getSplits(
                        return Stream.empty();
                    }

-                    if (filesModifiedAfter.isPresent() && addAction.getModificationTime() <= filesModifiedAfter.get().toEpochMilli()) {
+                    if ((!ignoreOldStats.orElse(false)) && recalculateAllStatistics.isPresent() && addAction.getModificationTime() <= recalculateAllStatistics.get().toEpochMilli()) {


There should be no need to check ignoreOldStats. filesModifiedAfter should is empty for recalculation.

Yeah. This part doesn't seem right, besides the parenthesis around the first item being redundant.

(!ignoreOldStats.orElse(false)) && recalculateAllStatistics.isPresent()

alexjo2144 · 2023-04-10T18:00:03Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

@@ -2108,8 +2109,12 @@ public ConnectorAnalyzeMetadata getStatisticsCollectionMetadata(ConnectorSession
        MetadataEntry metadata = handle.getMetadataEntry();

        Optional<Instant> filesModifiedAfterFromProperties = getFilesModifiedAfterProperty(analyzeProperties);
+        Optional<Boolean> ignoreOldStats = getRecalculateAllStatisticsProperty(analyzeProperties);


Let's name this boolean consistently throughout, rather than recalculateAll some and ignoreOld elsewhere. Personally I prefer affirmative names like recalculateAll or forceFullRefresh.

alexjo2144 · 2023-04-10T18:00:33Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/AnalyzeHandle.java

@@ -28,17 +28,20 @@
    private final boolean initialAnalyze;
    private final Optional<Instant> filesModifiedAfter;
    private final Optional<Set<String>> columns;
+    private final Optional<Boolean> recalculateAllStatistics;


Does this need to be Optional? Looks like it just always defaults to false?

alexjo2144 · 2023-04-10T18:01:18Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/AnalyzeHandle.java


    @JsonCreator
    public AnalyzeHandle(
            @JsonProperty("initialAnalyze") boolean initialAnalyze,
            @JsonProperty("startTime") Optional<Instant> filesModifiedAfter,
-            @JsonProperty("columns") Optional<Set<String>> columns)
+            @JsonProperty("columns") Optional<Set<String>> columns,
+            @JsonProperty("recalculateAllStatistics") Optional<Boolean> recalculateAllStatistics)


Could we have an enum mode with INITIAL, INCREMENTAL, and FULL_REFRESH options?

alexjo2144 · 2023-04-10T18:06:17Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeSplitManager.java

@@ -165,7 +166,7 @@ private Stream<DeltaLakeSplit> getSplits(
                        return Stream.empty();
                    }

-                    if (filesModifiedAfter.isPresent() && addAction.getModificationTime() <= filesModifiedAfter.get().toEpochMilli()) {
+                    if ((!ignoreOldStats.orElse(false)) && recalculateAllStatistics.isPresent() && addAction.getModificationTime() <= recalculateAllStatistics.get().toEpochMilli()) {


Yeah. This part doesn't seem right, besides the parenthesis around the first item being redundant.

(!ignoreOldStats.orElse(false)) && recalculateAllStatistics.isPresent()

findinpath · 2023-04-11T16:41:15Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/AnalyzeHandle.java

            @JsonProperty("startTime") Optional<Instant> filesModifiedAfter,
            @JsonProperty("columns") Optional<Set<String>> columns)
    {
-        this.initialAnalyze = initialAnalyze;
+        this.analyzeType = analyzeType;


plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/AnalyzeHandle.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeSplitManager.java

findinpath · 2023-04-11T16:59:44Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeSplitManager.java

@@ -160,8 +161,10 @@ private Stream<DeltaLakeSplit> getSplits(

        return validDataFiles.stream()
                .flatMap(addAction -> {


The following check doesn't need to be computed for every element from the stream:

tableHandle.getAnalyzeHandle().isPresent() && !(tableHandle.getAnalyzeHandle().get().getAnalyzeType() == FULL_REFRESH)

findinpath · 2023-04-11T17:15:42Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAnalyze.java

+                "('comment', 3764.0, 50.0, 0.0, null, null, null)," +
+                "('name', 379.0, 50.0, 0.0, null, null, null)," +
+                "(null, null, null, null, 50.0, null, null)";
+        assertUpdate(format("ANALYZE %s WITH(recalculate_all_statistics = true)", tableName));


While testing, I tried the combination

ANALYZE %s WITH(recalculate_all_statistics = true, columns = ARRAY['nationkey', 'regionkey']

Is it a bit confusing to you as well to use "recalculate_all_statistics" , but actually do it only for a specific set of columns?
Just thinking out loud whether the name chosen for the new ANALYZE property is sane.

ok i will try to rename

findinpath · 2023-04-11T17:18:00Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

@@ -2284,7 +2291,14 @@ private void updateTableStatistics(
            Optional<Instant> maxFileModificationTime,
            Collection<ComputedStatistics> computedStatistics)
    {
-        Optional<ExtendedStatistics> oldStatistics = statisticsAccess.readExtendedStatistics(session, location);
+        boolean recalculateAllStatistics = analyzeHandle.map(AnalyzeHandle::getAnalyzeType)


Suggested change

boolean recalculateAllStatistics = analyzeHandle.map(AnalyzeHandle::getAnalyzeType)

boolean recalculateAllStatistics = analyzeHandle

.map(AnalyzeHandle::getAnalyzeType)

.map(analyzeType -> analyzeType == FULL_REFRESH)

.orElse(false);

I agree with map on a new line the rest is similar

findinpath · 2023-04-11T17:25:02Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

@@ -2125,8 +2128,12 @@ public ConnectorAnalyzeMetadata getStatisticsCollectionMetadata(ConnectorSession
        MetadataEntry metadata = handle.getMetadataEntry();

        Optional<Instant> filesModifiedAfterFromProperties = getFilesModifiedAfterProperty(analyzeProperties);
+        Optional<Boolean> recalculateAll = getRecalculateAllStatisticsProperty(analyzeProperties);


recalculateAll -> recalculateAllStatistics

renamed anyway

findinpath · 2023-04-11T17:26:39Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java


-        Optional<ExtendedStatistics> statistics = statisticsAccess.readExtendedStatistics(session, handle.getLocation());
+        Optional<ExtendedStatistics> statistics = Optional.empty();
+        if (!recalculateAll.orElse(false)) {


!foo.orElse(false) takes a bit too much time to reason as a reader of the code

pajaks · 2023-04-13T08:47:42Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

@@ -2284,7 +2291,14 @@ private void updateTableStatistics(
            Optional<Instant> maxFileModificationTime,
            Collection<ComputedStatistics> computedStatistics)
    {
-        Optional<ExtendedStatistics> oldStatistics = statisticsAccess.readExtendedStatistics(session, location);
+        boolean recalculateAllStatistics = analyzeHandle.map(AnalyzeHandle::getAnalyzeType)


Do we need to convert AnalyzeType to boolean? Maybe we could check if it's FULL_REFRESH in following if?

we could check if there but we still would have to get rid of optional around AnalyzeHandle so it also won't be nice, and in the end fo the IF you need a boolean

pajaks · 2023-04-13T08:57:28Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAnalyze.java

@@ -357,19 +357,20 @@ public void testAnalyzeSomeColumns()
                        "('name', null, null, 0.0, null, null, null)," +
                        "(null, null, null, null, 50.0, null, null)");

+        String expectedFullStats = "VALUES " +


Please add tests when:

statsOnWrite is disabled and when initial data is inserted

statsOnWrite it's enabled and some data is added

SHOW STATS before and after ANALYZE should show only stats from second data addition

ANALYZE with recalucation should update statistics to have all data.

Maybe also test that modification using UPDATE and DELTE can now be reflected in STATS when using recalculation.

1 and 2 are not really related, why would I test if statsOnWrite works well in this PR, and situations when there are disabled and enabled are covered in TestDeltaLakeAnalyze
3 is covered in several places in TestDeltaLakeAnalyze
4 is covered by the piece of test code I added

Maybe also test that modification using UPDATE and DELTE can now be reflected in STATS when using recalculation.

ok

The 1-4 are steps for one test case and covers situation which is described in ticket which this PR resolves.

hmm i must have missread the description

findinpath · 2023-04-17T04:44:38Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java

@@ -39,6 +40,7 @@
 {
    public static final String FILES_MODIFIED_AFTER = "files_modified_after";
    public static final String COLUMNS_PROPERTY = "columns";
+    public static final String FORCE_RECALCULATE_STATISTICS = "force_recalculate_statistics";


What about using full_refresh ?

wouldn't it have the same problem as you mentioned with recalculate_all_stats ?

I'm assuming you were referring to #16634 (comment)

I find that full_refresh is appropriate even though the user may choose to analyze only specific columns.
While looking for alternatives, I came over fullscan on Transact-SQL.
However, I find full_refresh better suited for the purpose of the functionality you are exposing.

ok, no problem

ok let's go with full_refresh

findinpath · 2023-04-17T04:46:34Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java

@@ -39,6 +40,7 @@
 {
    public static final String FILES_MODIFIED_AFTER = "files_modified_after";
    public static final String COLUMNS_PROPERTY = "columns";
+    public static final String FORCE_RECALCULATE_STATISTICS = "force_recalculate_statistics";


Pls create a follow-up docs task if you don't intend now to document this new option.

i forgot about docs, i will add them here

findinpath · 2023-04-17T04:47:49Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java


-        Optional<ExtendedStatistics> statistics = statisticsAccess.readExtendedStatistics(session, handle.getLocation());
+        Optional<ExtendedStatistics> statistics = Optional.empty();
+        if (!forceRecalculate) {


Shouldn't we drop initially the existing stats in case we do a full refresh?
In the unlikely case that ANALYZE statement fails, the stats for the table will still be present.

I don't know, but is it bad to have previous stats there when ANALYZE fails?

cc @findepi

findinpath · 2023-04-17T04:54:06Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAnalyze.java

+                        "('name', 177.0, 25.0, 0.0, null, null, null)," +
+                        "(null, null, null, null, 25.0, null, null)");
+
+        assertUpdate("DELETE FROM " + tableName + " WHERE nationkey = 1", 1);


It is not clear from the test why the table stats are not updated when doing UPDATE / DELETE. Could you please add a comment in this regard?

They are updated but not correctly, there is a difference in expected values, I am not 100% sure about the reason of this.

NDV is updated because of normalization (for example cannot be higher than row count). With different data it should not change.

trino/core/trino-main/src/main/java/io/trino/cost/StatsNormalizer.java

Line 139 in dc7a9fd

if (distinctValuesCount > outputRowCount) {

After update size and NDV has strange values because of estimation based on null count. If changed value would be non-null, size and NDV would not change.

trino/core/trino-main/src/main/java/io/trino/sql/rewrite/ShowStatsRewrite.java

Line 182 in dc7a9fd

rowValues.add(toDoubleLiteral(symbolStatistics.getAverageRowSize() * planNodeStatsEstimate.getOutputRowCount() * (1 - symbolStatistics.getNullsFraction())));

yes, but I wanted them to change so I can show the diff later, i hope that it makes sense

pajaks · 2023-04-17T09:23:54Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAnalyze.java

+                "VALUES " +
+                        "('nationkey', null, 24.0, 0.0, null, 0, 24)," +
+                        "('regionkey', null, 5.0, 0.0, null, 0, 4)," +
+                        "('comment', 3638.0, 24.0, 0.0, null, null, null)," +


Size metric after UPDATE and ANALYZE gets totally off. I believe it's because we collect statistic on rewritten data and treats them as new entries.
But it's not your changes that causes that.

yep, exactly, i jsut added it here to show the diff after running analyze

I created separate task to tackle it: #17096

pajaks · 2023-04-20T13:07:29Z

docs/src/main/sphinx/connector/delta-lake.rst

@@ -757,6 +757,11 @@ To collect statistics for a table, execute the following statement::

  ANALYZE table_schema.table_name;

+Because Delta Lake connector can also update stats on writes they can drift away from


Drifting from proper values are because we don't update statistics in some situations not the opposite.

pajaks · 2023-04-20T13:15:17Z

docs/src/main/sphinx/connector/delta-lake.rst

@@ -757,6 +757,11 @@ To collect statistics for a table, execute the following statement::

  ANALYZE table_schema.table_name;

+Because Delta Lake connector can also update stats on writes they can drift away from


Suggested change

Because Delta Lake connector can also update stats on writes they can drift away from

Because Delta Lake connector can't update stats on all writes they can drift away from

I will follow Marius advise and delete explanation

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeSplitManager.java

docs/src/main/sphinx/connector/delta-lake.rst

findinpath · 2023-04-21T03:43:08Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java


-        Optional<ExtendedStatistics> statistics = statisticsAccess.readExtendedStatistics(session, handle.getLocation());
+        Optional<ExtendedStatistics> statistics = Optional.empty();
+        if (!forceRecalculate) {


cc @findepi

findinpath · 2023-04-21T03:46:09Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java

@@ -39,6 +40,7 @@
 {
    public static final String FILES_MODIFIED_AFTER = "files_modified_after";
    public static final String COLUMNS_PROPERTY = "columns";
+    public static final String FORCE_RECALCULATE_STATISTICS = "force_recalculate_statistics";


I'm assuming you were referring to #16634 (comment)

I find that full_refresh is appropriate even though the user may choose to analyze only specific columns.
While looking for alternatives, I came over fullscan on Transact-SQL.
However, I find full_refresh better suited for the purpose of the functionality you are exposing.

findinpath · 2023-04-21T03:51:11Z

docs/src/main/sphinx/connector/delta-lake.rst

@@ -757,6 +757,11 @@ To collect statistics for a table, execute the following statement::

  ANALYZE table_schema.table_name;

+Because Delta Lake connector can also update stats on writes they can drift away from
+proper values. To recalculate all stats for the table use additional parameter ``force_recalculate_statistics``.


To compute the table statistics from scratch, dismissing the existing table statistics, use the parameter ...

Let's leave aside the explanations about stats drifting away from the reality of the data.
There are multiple reasons why the stats may drift from reality.

findinpath · 2023-04-21T04:01:53Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAnalyze.java

+                "('comment', 3764.0, 50.0, 0.0, null, null, null)," +
+                "('name', 379.0, 50.0, 0.0, null, null, null)," +
+                "(null, null, null, null, 50.0, null, null)";
+        assertUpdate(format("ANALYZE %s WITH(force_recalculate_statistics = true)", tableName));


Let's actually showcase that recalculate_statistics does work with the columns which, when used previously without this option in the test, was causing a statement failure.

ANALYZE %s WITH(force_recalculate_statistics = true, columns = ARRAY['nationkey', 'regionkey', 'name'])

This can be a separate test case, here i want to show that using full_refresh or however we will name it results in exactly the same stats as you get after dropping existing stats

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java

findinpath · 2023-04-25T08:31:47Z

docs/src/main/sphinx/connector/delta-lake.rst

@@ -757,6 +757,10 @@ To collect statistics for a table, execute the following statement::

  ANALYZE table_schema.table_name;

+To recalculate all stats for the table use additional parameter ``full_refresh``.


Suggested change

To recalculate all stats for the table use additional parameter ``full_refresh``.

Use additional parameter ``full_refresh`` to recalculate from scratch the statistics for the table.

Can be a follow-up

findinpath · 2023-04-25T08:32:56Z

@ebyhr could you please have a look?

findinpath

LGTM except maybe the naming

Not all the statistics are recomputed.
What is actually being computed from scratch now with the new functionality are the extended stats (the file stats stay the same).

There is probably a better parameter name for ANALYZE in this context.

homar · 2023-04-26T10:04:20Z

LGTM except maybe the naming

Not all the statistics are recomputed. What is actually being computed from scratch now with the new functionality are the extended stats (the file stats stay the same).

There is probably a better parameter name for ANALYZE in this context.

So please suggest something, i followed your last suggestion ;)

ebyhr · 2023-05-09T00:55:51Z

Rebased on master to resolve conflicts.

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/AnalyzeHandle.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java

docs/src/main/sphinx/connector/delta-lake.rst

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAnalyze.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java

ebyhr · 2023-06-12T11:13:57Z

/test-with-secrets sha=5532e071209fbefb3111fb25527db54fbd154a9e

github-actions · 2023-06-12T13:46:39Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/5242991620

hashhar · 2023-06-12T15:56:54Z

/test-with-secrets sha=e7f32e7205adaddf942cc22a2ca10bd85f03883f

https://github.com/trinodb/trino/actions/runs/5247027640

docs/src/main/sphinx/connector/delta-lake.rst

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java

cla-bot bot added the cla-signed label Mar 20, 2023

github-actions bot added the delta-lake Delta Lake connector label Mar 20, 2023

homar requested a review from findepi March 20, 2023 14:06

findepi requested review from findinpath and alexjo2144 and removed request for findepi March 21, 2023 09:32

pajaks reviewed Mar 22, 2023

View reviewed changes

ebyhr assigned homar Apr 9, 2023

alexjo2144 reviewed Apr 10, 2023

View reviewed changes

homar requested review from pajaks and alexjo2144 April 11, 2023 10:48

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from 3461cc3 to d82fc5d Compare April 11, 2023 10:48

findinpath reviewed Apr 11, 2023

View reviewed changes

pajaks reviewed Apr 13, 2023

View reviewed changes

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from d82fc5d to df51254 Compare April 15, 2023 10:37

findinpath reviewed Apr 17, 2023

View reviewed changes

pajaks mentioned this pull request Apr 17, 2023

Populate stats when missing in transaction log #16743

Merged

pajaks reviewed Apr 17, 2023

View reviewed changes

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from df51254 to 884c04c Compare April 19, 2023 17:19

homar requested review from pajaks and findinpath April 20, 2023 08:45

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from ba36e2b to dabda92 Compare April 20, 2023 09:08

github-actions bot added the docs label Apr 20, 2023

pajaks reviewed Apr 20, 2023

View reviewed changes

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from dabda92 to b480958 Compare April 20, 2023 18:14

findinpath reviewed Apr 21, 2023

View reviewed changes

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from b480958 to 8a877e2 Compare April 22, 2023 17:56

pajaks reviewed Apr 24, 2023

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java Outdated Show resolved Hide resolved

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from 5915acc to 6c6516e Compare April 24, 2023 09:59

pajaks approved these changes Apr 24, 2023

View reviewed changes

findinpath approved these changes Apr 25, 2023

View reviewed changes

findinpath reviewed Apr 25, 2023

View reviewed changes

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from 6c6516e to 3a5eb99 Compare April 25, 2023 22:54

findinpath approved these changes Apr 26, 2023

View reviewed changes

ebyhr force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from 3a5eb99 to ee14ba8 Compare May 9, 2023 00:55

ebyhr reviewed May 9, 2023

View reviewed changes

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch 2 times, most recently from 2df4121 to 06e6d91 Compare May 9, 2023 21:56

findinpath requested a review from ebyhr May 10, 2023 04:26

ebyhr approved these changes Jun 8, 2023

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java Outdated Show resolved Hide resolved

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeAnalyzeProperties.java Outdated Show resolved Hide resolved

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from 06e6d91 to 5532e07 Compare June 12, 2023 10:29

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from 5532e07 to e7f32e7 Compare June 12, 2023 13:57

ebyhr reviewed Jun 13, 2023

View reviewed changes

Allow forcing Delta Lake connector to recalculate all statistics

0bcc679

homar force-pushed the homar_allow_forcing_delta_lake_to_recalculate_all_stats branch from e7f32e7 to 0bcc679 Compare June 13, 2023 08:52

homar requested a review from ebyhr June 13, 2023 20:59

ebyhr merged commit f79196f into trinodb:master Jun 14, 2023

github-actions bot added this to the 420 milestone Jun 14, 2023

colebow mentioned this pull request Jun 21, 2023

Add Trino 420 release notes #17997

Merged

		@@ -160,8 +161,10 @@ private Stream<DeltaLakeSplit> getSplits(

		return validDataFiles.stream()
		.flatMap(addAction -> {

		@@ -757,6 +757,11 @@ To collect statistics for a table, execute the following statement::

		ANALYZE table_schema.table_name;

		Because Delta Lake connector can also update stats on writes they can drift away from

	Because Delta Lake connector can also update stats on writes they can drift away from
	Because Delta Lake connector can't update stats on all writes they can drift away from

		@@ -757,6 +757,10 @@ To collect statistics for a table, execute the following statement::

		ANALYZE table_schema.table_name;

		To recalculate all stats for the table use additional parameter ``full_refresh``.

	To recalculate all stats for the table use additional parameter ``full_refresh``.
	Use additional parameter ``full_refresh`` to recalculate from scratch the statistics for the table.

Allow forcing delta lake to recalculate all statistics #16634

Allow forcing delta lake to recalculate all statistics #16634

Conversation

homar commented Mar 20, 2023 • edited by ebyhr Loading

Description

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pajaks Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath Apr 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath Apr 25, 2023 • edited Loading

Choose a reason for hiding this comment

findinpath commented Apr 25, 2023

findinpath left a comment

Choose a reason for hiding this comment

homar commented Apr 26, 2023

ebyhr commented May 9, 2023

ebyhr commented Jun 12, 2023

github-actions bot commented Jun 12, 2023

hashhar commented Jun 12, 2023 • edited by ebyhr Loading

homar commented Mar 20, 2023 •

edited by ebyhr

Loading

findinpath Apr 17, 2023 •

edited

Loading

pajaks Apr 17, 2023 •

edited

Loading

findinpath Apr 21, 2023 •

edited

Loading

findinpath Apr 25, 2023 •

edited

Loading

hashhar commented Jun 12, 2023 •

edited by ebyhr

Loading