Release notes for 342

[martint]

Dain Sundstrom

• all checked
• 09-11 20:58:17 33d3cdc Fully buffer small RC files
• 09-11 20:58:17 4744925 Add getRetainedSize to OrcDataSource
• 09-11 20:58:17 551c599 Introduce FSDataInputStreamTail reads
• 09-11 20:58:17 563dcca Cleanup warnings in TestOrcPageSourceMemoryTracking
• 09-11 20:58:17 66d0a2e Cleanup ParquetPageSourceFactory
• 09-11 20:58:17 75f426e Support CSE-KMS S3 object file size detection via tail read
• 09-11 20:58:17 794b5e7 Add MemoryOrcDataSource
• 09-11 20:58:17 7ffe98d Change Parquet MetadataReader to use ParquetDataSource
• 09-11 20:58:17 95dedea Add readTail to OrcDataSource
• 09-11 20:58:17 b1cc663 Rename Hive fileSize to estimatedFileSize
• 09-11 20:58:17 b22d752 Change ParquetDataSource readFully to return Slice
• 09-11 20:58:17 cd6faef Remove unused method from ParquetDataSource
• 09-11 20:58:17 eb80079 Add readTail to ParquetDataSource
• 09-11 20:58:17 ffe79e2 Add memory tests for both cached and uncached ORC
• 09-17 10:18:57 2df6cca Remove empty section
• 09-17 10:18:57 42f7ac5 Document support for insecure LDAP connection
• 09-17 10:18:57 a477317 Update https over http forwarding property name

David Phillips

• all checked
• 09-11 12:21:35 10f8a36 Fix performance regression when hive SerDe doesn't prefer Writables
• 09-16 14:03:43 1ec879e Fix JAVA_HOME in container image
• 09-16 14:03:43 2d97150 Add new test to verify if JAVA_HOME works
• 09-16 14:17:42 bd3ad39 Add Docker requirement to README
• 09-21 22:29:26 c6bd981 Update documentation for time and timestamp types
• 09-22 18:39:29 a35e756 Update to Avro 1.9.2
• 09-22 18:44:38 643bfd1 Support Domain Expressions for GlueHiveMetastore
• 09-22 18:46:02 b070c0b Cleanup warnings in AbstractTestIcebergSmoke
• 09-23 11:58:30 fa0f7db Match Iceberg transforms for negative epoch values

Grzegorz Kokosiński

• all checked
• 09-22 02:20:56 5f9d7ae Close Postgres test resources in order
• 09-22 02:20:56 a223d02 Remove extra this

Karol Sobczak

• all checked
• 09-09 03:55:29 1feaa0f Support lazy dynamic filtering in hive connector
• 09-09 03:55:29 9188bbf Move searchScanFilterAndProjectOperatorStats
• 09-09 04:33:33 4e97cb0 Extract local variable
• 09-09 04:33:33 519086a Make 128-bit addition use 2*64 bit values
• 09-09 04:33:33 954611d Add unscaled values to decimal operators addition benchmark
• 09-09 04:33:33 ae85047 Change way of checking sign in 128-bit arithmetic
• 09-10 04:24:30 6803205 Add benchmark for projected column reads
• 09-10 04:24:30 92cfdd1 Support benchmark reads through HivePageSource
• 09-10 04:24:30 d784071 Extract TestData as a top-level class
• 09-10 04:27:15 ef5d7ae Make DynamicFilter future resilient to cancel
• 09-10 04:39:58 61c72c4 Extract common function in TestMemorySmoke
• 09-10 04:39:58 af4e856 Implement dynamic filtering for semi-joins
• 09-11 02:53:45 cb300dc Add DynamicFilter#isAwaitable method
• 09-11 03:02:30 99647e9 Wait for final QueryInfo before using in test
• 09-15 04:10:10 ad5a676 Remove duplicate tpc-ds queries
• 09-15 04:39:48 0fb16ab Add tests for semi-join dynamic filtering in hive
• 09-15 05:43:25 1e37382 Simplify TestCoordinatorDynamicFiltering tests
• 09-15 05:43:25 22e5d39 Support for lazy dynamic filters for replicated joins
• 09-15 12:08:58 ebee077 Change dynamic partition pruning tests to avoid failures
• 09-16 01:41:24 d8cbadf Fix misuse of HashSet::new
• 09-16 01:43:39 360bfca Use forEach on collection directly
• 09-16 01:44:41 83bcdff Use task status version for task status long polling notifications
• 09-16 02:09:03 fc80e63 Fetch dynamic filters continuously
• 09-16 02:57:38 db5be59 Revert "Fetch dynamic filters continuously"
• 09-16 02:58:54 39a165f Fetch dynamic filters continuously
• 09-16 08:33:47 9faddbe Remove OperatorStats related flaky assertions
• 09-16 11:08:56 73fef2d Remove flaky assertion
• 09-16 11:08:56 f06b7fe Fix flaky TestSqlTask#testDynamicFilters
• 09-17 03:39:05 25abaf0 Choose join side with small tables as build side
• 09-17 03:45:35 8dcd866 Add SQL annotation to test method
• 09-17 03:45:35 e0d1846 Add createColorSequenceBlock in BlockAssertions
• 09-17 03:45:35 f548a1e Implement collection of min/max values in DynamicFilterSourceOperator
• 09-18 05:28:52 b055601 Reuse Marker object
• 09-19 11:46:37 cccbee4 Fix using of session parameter in test
• 09-21 04:43:16 d1f57a9 Reduce size of serialized Range (and TupleDomain)
• 09-21 07:27:39 3a8bf64 Refactor more operators to use Page#getColumns where appropriate
• 09-21 07:27:39 a9e9f07 Add single int overload to Page#getColumns

Martin Traverso

• all checked
• 09-17 10:35:25 00c6d56 Add rule for updating ApplyNode correlation list
• 09-17 10:35:25 177c5a2 Remove pruning of correlation list from project-off rule
• 09-17 10:35:25 53c9d38 Remove pruning of correlation list from project-off rule
• 09-17 10:35:25 b635826 Add rule for updating CorrelatedJoinNode correlation list
• 09-22 13:31:15 353dba2 Add from_iso8601_timestamp_nanos function

Piotr Findeisen

• all checked
• 09-09 00:30:38 3034362 Fix class name to match code style
• 09-09 00:30:38 cd33b04 Remove some unnecessary usage of Number
• 09-09 00:30:38 da50533 Fix SQL type of property
• 09-09 02:06:03 8732f67 Fix compiler error for lambda parameter with non-letter
• 09-09 02:06:03 b848b84 Remove bogus braces from regex
• 09-09 02:06:03 ba1dc08 Remove redundant constructor
• 09-09 02:06:03 cc289c9 Fix indentation
• 09-09 02:56:23 8046ff5 Update hive catalog configuration for development
• 09-09 03:44:38 b1e46b9 Update reference link
• 09-09 07:34:52 7bb39ce Enable Cassandra insert test
• 09-09 23:52:47 7442472 Unimplement deprecated ConnectorSplitManager#getSplits variants
• 09-09 23:52:47 a52dbbf Use non-deprecated ConnectorSplitManager#getSplits overload in tests
• 09-10 00:14:03 3013dcc Remove duplicate assertion
• 09-10 00:14:03 a82169e Fix Atop predicate pushdown
• 09-10 01:26:33 8e73b0e Remove incorrect default conversion
• 09-10 05:25:42 2993a4a Remove not applicable entry from release notes
• 09-10 23:28:11 e7f8dd9 Update some usages of deprecated TIMESTAMP_WITH_TIME_ZONE
• 09-11 13:02:36 9e648a8 Remove redundant supression
• 09-11 13:05:05 fcea7c6 Add timestamp timezone configuration properties for HDP3 environment
• 09-15 04:00:02 dba39ac Fix environment startup retries
• 09-16 12:39:01 99bc355 Hide setting HDFS user/group behind a toggle
• 09-17 01:07:12 d39c3e3 Update docker images to version 33
• 09-17 02:00:19 00f7755 Fix raw class usage
• 09-17 02:00:19 7a6e3c0 Fix unused parameter in PostgreSQL test
• 09-17 02:00:19 a8b4e79 Fix typo
• 09-17 02:00:19 b250c27 Add unit test for Session
• 09-17 02:00:19 e04cdde Fix adding catalog property to session
• 09-18 00:26:11 f71c223 Fix TestHiveAzureConfig
• 09-18 01:28:48 b378f1f Use AssertJ for better exception message
• 09-18 01:28:48 d54e663 Rename TestngUtils to DataProviders
• 09-18 01:28:48 f7159ed Simplify collector definition
• 09-18 01:29:37 6061151 Require project version to be known
• 09-18 01:34:16 2fe7727 Deprecate isCharType, isVarcharType
• 09-18 02:28:54 486db08 Deprecate isVarbinaryType
• 09-19 01:31:19 27c4839 Validate char/varchar values read in JDBC connectors
• 09-19 12:50:07 aeb6571 Update deprecation notices
• 09-21 05:40:45 8b17712 Provide versionless link to server in product tests
• 09-21 06:33:35 5bdd1c1 Implement aggregation pushdown for SQL Server
• 09-21 13:42:42 5a73b4a Remove commented out code
• 09-22 00:23:00 093709c Use correct type in column declaration
• 09-22 00:23:00 b23d261 Deprecate JdbcTypeHandle constructor overload
• 09-22 00:23:00 ee3cdc6 Add timestamp test case before epoch with fraction
• 09-22 00:23:00 f6bf01b Make JdbcTypeHandle.decimalDigits optional
• 09-22 05:27:30 0701225 Document sqlserver aggregate function pushdown

Praveen Krishna

• all checked
• 09-14 03:27:47 7ee94c9 Use static imports for Preconditions

Yuya Ebihara

• all checked
• 09-08 19:18:30 2a2970c Remove unused WILDCARD_EXPRESSION
• 09-10 00:37:03 b22fc1e Resolve inconsistent error message of SHOW COLUMNS
• 09-12 20:58:39 05b1226 Allow INSERT null for SQL Server varbinary type
• 09-16 01:50:12 24d5931 Do not copy constraint for temporary table on non-GTID MySQL

Łukasz Osipiuk

• all checked
• 09-09 03:20:58 8b825dc Update caching limitations
• 09-09 11:23:37 55ad4cb Extract method
• 09-09 11:23:37 5671a6e Add support for precision for TIMESTAMP W/TZ in Postgresql type mapping
• 09-09 13:23:18 4ad9ab9 Inline unneeded methods
• 09-09 13:23:18 e15ff10 Rename TestOracleTypes to TestOracleTypeMapping
• 09-10 05:17:02 0b806e0 Simplify flow in timestamp with timezone mapping tests
• 09-10 05:17:02 267ada1 Do not use raw parametrized type
• 09-10 05:17:02 7a17dce Rename method paremeter to express its meaning
• 09-10 05:17:02 be2b97a Add support for precision for TIMESTAMP in PostgreSQL type mapping
• 09-10 05:17:02 cc78151 Use generic trueFalse data provider for timestamp tests
• 09-10 05:17:02 db779fc Inline addArrayTimestampTestIfSupported
• 09-10 05:17:02 ee7d359 Remove not needed annotations
• 09-10 05:17:02 effe2a0 Remove redundant cast
• 09-10 11:40:54 8b3f1fe Make Type column wider in Kafka columns documentation
• 09-10 11:40:54 a8345e2 Refactor KafkaInternalFieldDescription to KafkaInternalFieldManager
• 09-10 11:40:54 d446959 Add header column to Kafka Connector
• 09-11 05:55:55 2a32d3f Do not use oracleServer explicitly
• 09-11 05:55:55 65b57af Extract AbstractTestOracleTypeMapping
• 09-11 05:55:55 c9c4305 Rename method
• 09-11 11:05:29 7d4a7a4 Fix JDBC driver compatibility regarding TIME WITH TIME ZONE
• 09-13 23:38:06 0893602 Support temporal types in Kafka JSON encoder
• 09-13 23:38:06 1fec662 Add tests for Kafka JSON date time types
• 09-13 23:38:06 3ef8cfd Remove support for illogical types in rfc2822 record decoder
• 09-13 23:38:06 456e1fa Account for zone offset in record decoder TIME WITH TIME ZONE decoding
• 09-13 23:38:06 5634658 Add more functions to DateTimeTestingUtils
• 09-13 23:38:06 6f9f12d Expand record decoder time w/tz tests
• 09-13 23:38:06 feecdc9 Add docs for Kafka JSON temporal support
• 09-14 03:34:01 25cff68 Improve product tests environment startup
• 09-14 03:34:01 31bd350 Configure launcher bin location
• 09-14 03:34:01 54e9e28 Remove redundant pruneEnvironment call
• 09-14 03:34:01 5ee948f Refactor logCopyingListener initialization
• 09-14 03:34:01 7d3971b Add DockerContainer logical name
• 09-14 03:34:01 9291cd6 Pass startup retries to environment builder
• 09-14 03:34:01 94c8a5d Removeme: copy logs from containers
• 09-14 03:34:01 a0ce308 Fix suite describe command printing
• 09-14 03:34:01 a260eab Add container output handling modes
• 09-14 03:34:01 a482e37 Print container stats on container shutdown
• 09-14 03:34:01 a643e38 Make EnvironmentDefaults class final
• 09-14 03:34:01 ac86653 Drop environment configuration comments
• 09-14 03:34:01 b140166 Make launcher commands callable
• 09-14 03:34:01 f9e83fa List and copy log files from running container
• 09-14 05:25:42 31f570b Test current JDBC driver against old Presto releases
• 09-14 05:25:42 7896b52 Allow specifying Presto server version to be tested via env
• 09-14 05:25:42 85fc1ec Extract BaseTestJdbcResultSet
• 09-14 05:25:42 bb460c0 Guard JDBC tests against Presto server version
• 09-14 05:25:42 d17975c Rename presto-test-jdbc-compatibility to presto-test-jdbc-compatibility-old-driver
• 09-14 05:25:42 df348ba Make TestJdbcResultSet multi threaded
• 09-14 09:48:57 0ec519d Set host configuration of product tests containers
• 09-14 09:48:57 463f0e1 Fix displaying test run duration
• 09-14 09:48:57 5664993 Fix suite describe
• 09-14 09:48:57 c0edf8e Allow case insensitive enum values in launcher
• 09-14 09:48:57 cdbce1d Improve Suite toString method
• 09-14 09:48:57 f8f0089 Add timeout for suite and test execution
• 09-17 01:43:59 6149fa2 Remove unnecessary variable propagation
• 09-17 01:43:59 f9ef95e Allow to configure insert into Hive partition via configuration property
• 09-17 04:21:24 432bf86 Clean up PrestoAzureConfigurationInitializer
• 09-17 04:21:24 44e1bee Rename ABFS tests as ABFS access key tests
• 09-17 04:21:24 655a19d Share more setup code between hive test scripts
• 09-17 04:21:24 839476e Add superclass for tests using different ABFS authentication methods
• 09-17 04:21:24 a381973 Fix indentation and line continuations in hive test scripts
• 09-17 04:21:24 be89b2b Move TestHiveAzureConfig to azure package
• 09-17 04:21:24 cbabd27 Add tests for PrestoAzureConfigurationInitializer
• 09-17 04:28:23 2b3a1c3 Intercept and log listener exceptions
• 09-17 04:28:23 327d6ed Create new network on environment startup
• 09-17 04:28:23 6f417ae Fix starting multinode environment without presto
• 09-17 04:28:23 70e23db Improve hadoop-master-2 container configuration
• 09-17 04:28:23 8757e7f Make EnvironmentDown callable
• 09-17 04:28:23 9ccc78f Improve environment shutdown
• 09-17 04:28:23 cbd9131 Remove unused constant
• 09-17 04:28:23 d548911 Fix displaying stats
• 09-17 11:36:10 cc30d16 Add support for ABFS OAuth authentication
• 09-18 02:05:40 0893543 Refactor and improve coverage for Oracle integration tests

[findepi]

General
* Fix query failure when lambda expression references a table column containing a dot. ({issue}`5087`)

#5087

[sopel39]

Hive
* Add property (``hive.dynamic-filtering-probe-blocking-timeout``) for delaying table scans
  until dynamic partition pruning can be performed more efficiently. ({issue}`4991`)

#4991

[sopel39]

General
* Improve performance of queries that use decimal type. ({issue}`4886`)

#4886

[findepi]

Atop Connector Changes
* Fix incorrect query results when query contains predicates on `start_time` or `end_time` column. ({issue}`5125`)

#5125

[sopel39]

SPI
* Make dynamic filter futures resilient to cancellation. ({issue}`5099`)

#5099

[sopel39]

General
* Improve query performance by adding support for dynamic filtering and dynamic
  partition pruning to semi-join relational operator. ({issue}`5017`)

#5017

[losipiuk]

Kafka
* Expose message headers as a ``_headers`` column of ``map(VARCHAR, array(VARBINARY))`` type. ({issue}`4462`)

#4462

[sopel39]

SPI
* Add ``DynamicFilter#isAwaitable`` method that returns whether dynamic filter is not complete and can be
  awaited for via future. ({issue}`5043`)

#5043

[losipiuk]

PostgreSQL
* Extend type mapping to support variadic ``TIMESTAMP`` and ``TIMESTAMP WITH ZONE`` types. ({issue}`5124`, {issue}`5105`)

#5124
#5105

[ebyhr]

SQL Server
* Fix failure when inserting `NULL` to `VARBINARY` column. ({issue}`4846`)

#4846

[losipiuk]

Kafka
* Add write support for ``TIME``, ``TIME WITH TIME ZONE``, ``TIMESTAMP`` and ``TIMESTAMP WITH TIME ZONE`` 
  for Kafka connector when JSON encoder is in use. ({issue}`4743`)

#4743

[sopel39]

General/SPI
* Enable connectors to wait for dynamic filters derived from replicated join before generating splits. ({issue}`4685`)

#4685

[ebyhr]

MySQL
* Improve performance of `INSERT` statement when MySQL instance isn't running with GTID mode. ({issue}`4995`)

#4995

[sopel39]

General
* Improve dynamic partition pruning and query performance by reducing latency of dynamic filters collection. ({issue}`4988`)

#4988

[findepi]

Hive
* Disable matching the existing user and group of the table or partition when creating new files on HDFS.
  The functionality was added in 341 and is now disabled by default. You can enable it with `hive.fs.new-file-inherit-ownership`
  configuration property. ({issue}`5187`)

#5187

[losipiuk]

Hive
* Allow specifying what happens if data is inserted into existing Hive partition. 
  This can be done using ``hive.insert-existing-partitions-behavior`` config property. ({issue}`4999`)

#4999

[sopel39]

General
* Improve join performance when cost-based optimizer has missing or inaccurate stats. ({issue}`5141`)

#5141

[findepi]

## SQL Server Connector Changes

* Improve performance of aggregation queries by computing aggregations within SQL Server database.
  Currently, the following aggregate functions are eligible for pushdown:
  ``count``,  ``min``, ``max``, ``sum`` and ``avg``. ({issue}`4139`)

#4139 #5196

[mosabua]

the SQL server connector changes from @findepi above should change and just link to the docs and have a short sentence like

* Add :ref:`aggregate function pushdown <sqlserver-pushdown>` as performance improvement ({issue}`4139`)

see #5245

[losipiuk]

Azure
* Add support for ABFS OAuth authentication ({issue}`5052`)

#5052

[losipiuk]

Kafka
* In JSON decoder drop decoding support for temporal types for nonsenical combinations of input-format-type/data-type.
  Following combination are no longer supported:
  - ``rfc2822``:  ``DATE``, ``TIME``, ``TIME WITH TIME ZONE``
  - ``milliseconds-since-epoch``: ``TIME WITH TIME ZONE``, ``TIMESTAMP WITH TIME ZONE``    
  - ``seconds-since-epoch``: ``TIME WITH TIME ZONE``, ``TIMESTAMP WITH TIME ZONE``    
  ({issue}`4743`)

#4743

[dain]

## Hive
* Add support for S3 encrypted files. ({issue}`2536`)
* Improve performance of reading small file in RCFile format. ({issue}`2536`)

#2536

[sopel39]

General
* Reduce latency for queries where broadcast join is used and broadcasted table is large. ({issue}`5237`)

#5237

[findepi]

Hive
* Support reading timestamp with microsecond or nanosecond precision. This can be enabled with `hive.timestamp-precision`
  connector configuration property. ({issue}`4953`)

#4953
part of #3977

[electrum]

# Hive Connector Changes

* Improve performance when reading `JSON` and `CSV` file formats. ({issue}`5142`)

#5142

[electrum]

# Hive Connector Changes

* Improve planning time for queries with non-equality filters on
  partition columns when using the Glue metastore. ({issue}`5060`)

#5060

[electrum]

# Iceberg Connector Changes

* Fix partition transforms for temporal columns for dates before 1970. ({issue}`5273`)

#5273

[sopel39]

General
* Allow collection of dynamic filters for joins with large build side using the
  `enable-large-dynamic-filters` configuration property or the `enable_large_dynamic_filters`
  session property.
  The existing configuration properties `dynamic-filtering-max-per-driver-row-count`,
  `dynamic-filtering-max-per-driver-size`, `dynamic-filtering-range-row-limit-per-driver`
  and their corresponding session properties are now defunct.
  When large dynamic filters are enabled, limits on size of dynamic filters can be configured
  for each join distribution type using the configuration properties
  `dynamic-filtering.large-broadcast.max-distinct-values-per-driver`,
  `dynamic-filtering.large-broadcast.max-size-per-driver` and
  `dynamic-filtering.large-broadcast.range-row-limit-per-driver` and their equivalent for partitioned
  join distribution type.
  Similarly, limits for dynamic filters when `enable-large-dynamic-filters` is not enabled
  can be configured using configuration properties like
  `dynamic-filtering.large-partitioned.max-distinct-values-per-driver`. ({issue}`5262`)

#5262

[mosabua]

This is way too long @sopel39 .. please move this into the docs and then link to it