Add Trino 446 release notes

[colebow]

Description

Assemble the release notes for the upcoming Trino 446 release.

Verification for each pull request

Format: PR/issue number, ✅ / ❌ rn ✅ / ❌ docs
✅ rn - release note added and verified, or assessed to be not necessary, set to ❌ rn before completion
✅ docs - need for docs assessed and merged, or assessed to be not necessary, set to ❌ docs before completion

17 April 2024

• Upgrade clickhouse image in tests #21469 ✅ rn ✅ docs
• Replace Aggregation RemoveInput manual WindowAggregation class #21349 ✅ rn ✅ docs
• Remove irrelevant RowExpression optimizations #21582 ✅ rn ✅ docs
• Delta Lake Azure Smoke Tests:Rewrite test to avoid flakiness #21560 ✅ rn ✅ docs

18 April 2024

• Add builders for MySQL, PostgreSQL, BigQuery query runners #21598 ✅ rn ✅ docs
• Query runner builders refactors #21597 ✅ rn ✅ docs
• Add builders for Oracle, SQL Server query runners #21605 ✅ rn ✅ docs
• Remove unused tests:hive label #21612 ✅ rn ✅ docs
• Set speculative flag at create time #21596 ✅ rn ✅ docs
• Use enhanced switch #21578 ✅ rn ✅ docs
• Fix build pt matrix script #21613 ✅ rn ✅ docs
• Fix list table in JDBC docs #21617 ✅ rn ✅ docs
• Convert metadata classes to records #21572 ✅ rn ✅ docs
• Update AWS SDK v1 to 1.12.703 #21614 ✅ rn ✅ docs
• Fix spelling in Salesforce docs #21518 ✅ rn ✅ docs
• Use binary search in SortedRangeSet overlaps #21611 ✅ rn ✅ docs
• Add builder for Mongo query runner and simplify Kafka query runner builder #21610 ✅ rn ✅ docs
• Simplify Kafka query runner builder #21615 ✅ rn ✅ docs
• Add builders for Redshift query runner #21606 ✅ rn ✅ docs
• Throw HIVE_UNSUPPORTED_FORMAT in case of missing SerdeInfo in Glue #21591 ✅ rn ✅ docs
• Convert Split to record #21583 ✅ rn ✅ docs

19 April 2024

• Use binary search in SortedRangeSet intersect #21623 ✅ rn ✅ docs
• Add builders for Elasticsearch, OpenSearch query runners #21628 ✅ rn ✅ docs
• Optimize join filters after ReorderJoins #21638 ❌ rn ✅ docs
• Randomize table name in BaseHiveConnectorTest #21629 ✅ rn ✅ docs

21 April 2024

• Include addresses in retainedSize of IcebergSplit #21644 ✅ rn ✅ docs
• Hive: Add documentation for timestamp_precision session property #21567 ✅ rn ✅ docs
• Make OrcFileWriter.orcWriter private #21643 ✅ rn ✅ docs

22 April 2024

• Revert "Convert Split to record" #21631 ✅ rn ✅ docs
• Convert MemorySplit to record #21636 ✅ rn ✅ docs
• Convert some classes to record in Kafka #21646 ✅ rn ✅ docs
• Move testConcurrentInsertsReconciliationForBlindInserts in Delta Lake #21618 ✅ rn ✅ docs
• Convert some classes to record in Cassandra #21645 ✅ rn ✅ docs
• Update maven-wrapper to 3.3.0 & dependencies #21641 ✅ rn ✅ docs
• Convert TestBlackHoleSmoke to test framework #21633 ✅ rn ✅ docs
• Avoid non-actionable warnings logged in Delta writer stats #21656 ✅ rn ✅ docs
• Update JLine to 3.26.0 #21660 ✅ rn ✅ docs
• Convert some classes to record in Prometheus #21652 ✅ rn ✅ docs
• Update CacheKeyProvider.getCacheKey parameter name #21647 ✅ rn ✅ docs
• Fix last openRecordGroup not processed in FlatArrayBuilder #21507 ✅ rn ✅ docs
• Use getSizeInBytes rather than getRetainedSizeInBytes #21653 ✅ rn ✅ docs
• Convert config properties to list-table in Prometheus documentation #21658 ✅ rn ✅ docs
• Normalize name when creating a new catalog #21409 ✅ rn ✅ docs

23 April 2024

• Relax condition for allowing writer tasks splitting #21661 ✅ rn ✅ docs
• Narrow synchronization scope #21664 ✅ rn ✅ docs
• Obtain page size only once #21668 ✅ rn ✅ docs
• Leave comment why testSelectInformationSchemaColumns is isolated in Cassandra #21670 ✅ rn ✅ docs
• Refactor Delta Lake query runner setup #21654 ✅ rn ✅ docs
• Extend automatic type coercion support in Iceberg table creation to char #21515 ✅ rn ✅ docs
• Add glacier skipping logic #21164 ✅ rn ✅ docs
• Fix failure when symbol name ends in colon #21676 ✅ rn ✅ docs
• fix file-system-access-control permission table #21674 ✅ rn ✅ docs

24 April 2024

• Make PushPartialAggregationThroughExchange deterministic #21677 ✅ rn ✅ docs
• Replace uses of newParameterizedType #21687 ✅ rn ✅ docs
• Switch GCS tests to native file system #21685 ✅ rn ✅ docs
• Fix failure when single quote exists in Pinot condition #21683 ✅ rn ✅ docs
• Support type mapping configs in Snowflake Connector #21528 ✅ rn ✅ docs
• Add Iceberg Snowflake catalog support #21365 ✅ rn ✅ docs
• Convert config properties to list-table in BigQuery documentation #21655 ✅ rn ✅ docs
• Handle exception when OTEL is not fully available to Trino JDBC #21701 ✅ rn ✅ docs
• Fix flaky TestCassandraConnectorTest #21696 ✅ rn ✅ docs
• Append random suffix in BaseHiveConnectorTest #21684 ✅ rn ✅ docs

25 April 2024

• Add builder for Cassandra query runner #21693 ✅ rn ✅ docs
• Fix compilation of SQL functions with labels #21691 ✅ rn ✅ docs
• Fix failure when migrating partitioned Hive table in Glue #21699 ✅ rn ✅ docs
• Deprecate 'schema' and 'table' arguments in Iceberg table_changes function #21698 ✅ rn ✅ docs
• Convert some classes to record in Pinot #21688 ✅ rn ✅ docs
• ConnectorTableLayout cleanup #21673 ✅ rn ✅ docs
• Add TPCH tables in PostgreSqlQueryRunner main method #21705 ✅ rn ✅ docs
• Increase margin in MariaDB testNullsFraction #21704 ✅ rn ✅ docs
• Exclude record getter-like fields from JSON serialization #21708 ✅ rn ✅ docs
• Add more per-task distribution statistics in query completion event #21694 ✅ rn ✅ docs
• Update sphinx to 7.3.7 with dependencies #21492 ✅ rn ✅ docs
• Add missing end lines #21712 ✅ rn ✅ docs
• Remove extra colons #21711 ✅ rn ✅ docs
• Fix handling special characters in Azure native FS  #21586 ✅ rn ✅ docs
• Parquet hadoop package refactor #21665 ✅ rn ✅ docs
• Enable bigquery.arrow-serialization.enabled by default #21580 ✅ rn ✅ docs

26 April 2024

• Decrease query.low-memory-killer.delay to 30s #21719 ✅ rn ✅ docs
• Update provisio to 1.0.24 #21723 ✅ rn ✅ docs
• Increase lock timeout from 1s to 5s in H2 #21722 ✅ rn ✅ docs
• Upgrade to Iceberg 1.5.1 #21714 ✅ rn ✅ docs
• Nullify DriverFactory#operatorFactories on noMoreDrivers #21706 ✅ rn ✅ docs
• Fail implicitly assigning null to non-null column in MERGE #21697 ✅ rn ✅ docs
• Update airbase to 155 and friends #21729 ✅ rn ✅ docs

28 April 2024

• Remove redundant empty line #21734 ✅ rn ✅ docs

29 April 2024

• Convert some classes to record in Hive #21740 ✅ rn ✅ docs
• TaskUpdateRequest cleanup #21742 ✅ rn ✅ docs
• Fix incorrect result with deletion vector on Delta partitioned table #21738 ✅ rn ✅ docs
• Suppress single view errors during metadata listing.  #21746 ✅ rn ✅ docs
• Name threads in all thread pools #21730 ✅ rn ✅ docs
• Serialize function dependencies for SpecialForm #21749 ✅ rn ✅ docs
• Fix tracing context propagation during connector invocation #21575 ✅ rn ✅ docs
• Document table_changes Iceberg table function #21406 ✅ rn ✅ docs
• Fix no suitable driver found in MySQL event listener #21625 ✅ rn ✅ docs
• Log a message if file system caching is disabled on coordinator #21666 ✅ rn ✅ docs

30 April 2024

• Add information about current open writers to Iceberg max open writers error message #21720 ✅ rn ✅ docs
• Minor refactoring in Google Sheets connector #21757 ✅ rn ✅ docs
• Always use G1GC for product tests #21311 ✅ rn ✅ docs
• Ignore failing Hive views during listing #21748 ✅ rn ✅ docs
• Remove unused import #21764 ✅ rn ✅ docs
• Update datasketches to 6.0.0 and other dependencies #21743 ✅ rn ✅ docs
• Convert RedisTableFieldDescription to record #21766 ✅ rn ✅ docs
• Minor refactoring in Prometheus connector #21767 ✅ rn ✅ docs
• Remove spurious logging in BigQuery tests #21768 ✅ rn ✅ docs
• Convert some classes to record in Kinesis #21770 ✅ rn ✅ docs
• Document replace tables for Delta Lake and update section for Iceberg #20819 ✅ rn ✅ docs
• Remove secondary constructors from SpecialForm  #21750 ✅ rn ✅ docs
• Enumerate EXECUTE privilege #21662 ✅ rn ✅ docs
• Added the verification of architectures parameters. #18098 ✅ rn ✅ docs
• Retry queries on Databricks when dealing with a communication failure #21747 ✅ rn ✅ docs
• Use only the biggest cardinality symbol for forced exchange #21726 ✅ rn ✅ docs
• Improve grammar for SQL routine example docs #21428 ✅ rn ✅ docs
• Fix flakiness in TestIcebergNessieCatalogConnectorSmokeTest #21765 ✅ rn ✅ docs
• Update runtime JDK to jdk-22.0.1+8 #21775 ✅ rn ✅ docs
• Update airbase to 156 #21763 ✅ rn ✅ docs
• Bump Cassandra images #21774 ✅ rn ✅ docs
• Improve speed when listing table comments in BigQuery #21581 ✅ rn ✅ docs
• Move config files to resources in Kinesis #21769 ✅ rn ✅ docs

1 May 2024

• Add builder to MetadataEntry in Delta Lake #21781 ✅ rn ✅ docs

[martint]

Hmm... we shouldn't include a breaking change that requires adding --add-opens options to the JVM. Why is that required? The implementation should not have to rely on users manually working around Java visibility rules. @ebyhr, we should revert this.

[wendigo]

This is due to an apache arrow requirement. There is no workaround for it as far as I know. The same applies to snowflake connector which is using Arrow as well.

[ebyhr]

The context is we want to remove another Avro serialization eventually since it's slower than Arrow and keeping both code requires maintenance costs.

[wendigo]

https://arrow.apache.org/docs/java/install.html#java-compatibility

[martint]

That’s a big problem. It means that those libraries are using internal JDK APIs that may disappear at any point.

Depending on such behavior may make it hard or impossible to update to new Java versions in the future. In the worst case, we may need to update to the next major Java version to fix security issues and be forced to break those connectors.

We did a lot of work when migrating to Java 11 to avoid having to use internal Java APIs to be where we are today. This is a step backwards.

[wendigo]

@martint yes it does and it's been problematic for some time

[martint]

Let's go ahead with the release for now and we can revisit whether we want to revert this once we have alignment in the next couple of days.

[dain]

Why do we have to give arrow native memory? It is impossible for it to use heap memory? Is this code pulling in native libraries also?

[dain]

Actually, we fork arrow to use heap memory. I have no idea why they think they need native memory

[wendigo]

@dain native - no. ArrowBuf allocates and uses off heap memory. The idea is to allow to pass vectors to other languages but that's not an actual case in these connectors

[mosabua]

what is that? A JMX metric? A system metadata? Both?

[mosabua]

Two minor items .. but good to go.