Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release notes for 301 #22

Closed
6 tasks done
electrum opened this issue Jan 22, 2019 · 18 comments
Closed
6 tasks done

Release notes for 301 #22

electrum opened this issue Jan 22, 2019 · 18 comments

Comments

@electrum
Copy link
Member

electrum commented Jan 22, 2019

Dain Sundstrom

  • all checked
  • 01-22 16:59:11 14db5b5 Add trace token to query events
  • 01-22 18:54:06 17112e0 Update node list on workers
  • 01-22 18:54:06 33c0fc0 Stop DiscoveryNodeManager in tests
  • 01-22 18:54:06 86d780a Only fire node change event on change
  • 01-22 18:54:06 b950d0f Add test for DiscoveryNodeManager listener

David Phillips

  • all checked
  • 01-23 00:10:15 33e41f3 Remove old Maven extensions
  • 01-23 00:10:15 7ef63e6 Use ValidationAssertions in TestPowerOfTwoValidator
  • 01-23 00:10:15 a5a7722 Update Sphinx plugin to 2.1
  • 01-23 00:10:15 accd094 Update to Airbase 90
  • 01-23 00:10:15 e261486 Fix undeclared dependency used for HTTP constants
  • 01-23 00:10:15 f0e4aa6 Fix deprecated usage of catchingAsync
  • 01-23 00:10:48 eb4f141 Use Xenial environment on Travis
  • 01-25 00:02:23 8456488 Move CLA bot config to clabot-config repo
  • 01-25 15:24:53 c408a78 Add current user security mode for views
  • 01-25 22:25:54 238f4b6 Update to Hive 2.3.4
  • 01-25 22:25:54 486a8f1 Fix value comparisons in TestHiveFileFormats
  • 01-25 22:25:54 507307e Remove unused Parquet classes
  • 01-25 22:39:27 95122b5 Improve error message for unsupported reduce_agg state type
  • 01-26 17:21:03 435fe92 Remove usages of TypeSignature from client
  • 01-26 17:21:03 904d511 Replace PrestoWarning with new client Warning
  • 01-26 17:21:03 950a56c Remove SPI as dependency for JDBC driver
  • 01-26 17:21:03 cbbf251 Convert JDBC driver to ClientTypeSignature
  • 01-26 17:21:03 d5dfb31 Convert JDBC driver from TimeZoneKey to ZoneId

Grzegorz Kokosiński

  • all checked
  • 01-25 00:04:33 b9a1a7e Limit access to ParametricAggregation fields
  • 01-28 03:22:46 1f284a1 Prefer SchemaTablePrefix#getSchema to create optional schema name
  • 01-28 03:22:46 5b63c49 Use Optional for nullable fields in SchemaTablePrefix
  • 01-28 03:22:46 73cc391 Prefer SchemaTablePrefix#toSchemaTableName
  • 01-28 05:03:23 efe8ca0 Add suppressed exceptions to exception that is thrown
  • 01-28 05:04:01 311abbb Extend JdbcSplit with additional predicate field
  • 01-28 05:10:15 9825bc8 Configure defaults for coordinator in DistributedQueryRunner in one place
  • 01-29 01:52:32 5368334 Fix formatting
  • 01-29 01:52:33 d13fbf9 Check if Presto is running before erasing RPM
  • 01-29 01:55:58 53ed213 Log test query durarion only if it run longer than 1s
  • 01-29 01:55:58 96cee95 Log Presto node role when creating distributed testing cluster
  • 01-29 01:55:58 bee9579 Log testing Presto server URL when it starts
  • 01-29 01:55:58 e77d698 Do not log test query uri
  • 01-29 07:20:30 120c28f Allow JDBC connector to define procedure
  • 01-30 01:55:54 fa65182 Add support for getTableStatistics to base-jdbc connector
  • 01-30 01:58:54 68393b0 Make use of hive temp staging directory configurable
  • 01-30 01:58:54 9227286 Make hive temporary staging directory location configurable

Karol Sobczak

  • all checked
  • 01-25 01:47:23 9305d99 Add write_hive_metastore_recording procedure
  • 01-25 05:21:20 47ba96e Add missing withGeneratedName to RecordingHiveMetastore exporter
  • 01-28 01:43:00 eb7a512 Add metadata recorder support to Glue
  • 01-29 02:52:01 eb7f677 Make dummy HiveEventClient use DEBUG log instead of INFO
  • 01-29 04:25:50 0481f18 Product tests for SHOW ROLES
  • 01-29 04:25:50 0546433 Add docs for SHOW ROLES
  • 01-29 04:25:50 07c5496 Introduce .information_schema.roles table
  • 01-29 04:25:50 0ba3120 Add grantor to HivePrivilegeInfo
  • 01-29 04:25:50 0beeb77 Add grantor_type and grantee_type columns to table_privileges
  • 01-29 04:25:50 0c2fa44 Make ThriftMetastoreUtil.list*Roles methods to return Stream
  • 01-29 04:25:50 0c3406c Add SHOW ROLE GRANTS syntax
  • 01-29 04:25:50 0f2e2e3 Move parsePrivilege to MetastoreUtil
  • 01-29 04:25:50 18603de Reorder methods in HivePrivilegeInfo
  • 01-29 04:25:50 1d11c06 Make list*TablePrivileges to return Stream
  • 01-29 04:25:50 20cf120 Fix listing privileges when admin role is set
  • 01-29 04:25:50 21ceed6 Catalog access control for roles
  • 01-29 04:25:50 23e3f31 Add Grant/Revoke/List roles authorization to the SPI
  • 01-29 04:25:50 2438df8 Assign admin role to subset of users in FileHiveMetastore
  • 01-29 04:25:50 266fa20 Implement listRoleGrants() in Hive
  • 01-29 04:25:50 26b421a Remove redundant checkDatabasePermission methods
  • 01-29 04:25:50 31e9d26 Reorder methods in SqlStandardAccessControl
  • 01-29 04:25:50 34ffbaa Accept ROLE in GRANT/REVOKE Privileges statements
  • 01-29 04:25:50 3dca463 Implement Grant/Revoke/ListApplicableRoles in Hive
  • 01-29 04:25:50 42ba503 Introduce CREATE ROLE and DROP ROLE statements
  • 01-29 04:25:50 4b90148 Add listRoleGrants to the SPI
  • 01-29 04:25:50 53b13c9 Refactor canCreateView security checks
  • 01-29 04:25:50 5d9615c Add SHOW CURRENT ROLES
  • 01-29 04:25:50 5db2f71 Remove unused metastore statistics beans
  • 01-29 04:25:50 5f9c74d Introduce GRANT/REVOKE roles statements
  • 01-29 04:25:50 62ed7d2 Introduce access control for GRANT/REVOKE ROLE
  • 01-29 04:25:50 671d653 Introduce ENABLED_ROLES view
  • 01-29 04:25:50 6910a27 Enumerate roles until enabled role is found
  • 01-29 04:25:50 709de9d Simplify checkTablePermission
  • 01-29 04:25:50 729f646 Fix rewrite SHOW GRANTS as a SELECT query
  • 01-29 04:25:50 7427e3a Extract role name constants
  • 01-29 04:25:50 745e4b8 Expose Create/Drop/List roles methods in SPI
  • 01-29 04:25:50 747c276 Introduce APPLICABLE_ROLES view
  • 01-29 04:25:50 776b453 Move PrincipalType to presto-spi
  • 01-29 04:25:50 8234650 Store catalog selected roles in Identity
  • 01-29 04:25:50 8464cbc Access control for SHOW ROLE GRANTS and SHOW CURRENT ROLES
  • 01-29 04:25:50 87063ee More product tests for SET ROLE
  • 01-29 04:25:50 91b38e4 Implement Create/Drop/List roles in Hive connector
  • 01-29 04:25:50 942cde9 Rename getGrantOptionForPrivilege to hasGrantOptionForPrivilege
  • 01-29 04:25:50 958edea Rewrite SHOW ROLES as a select query
  • 01-29 04:25:50 a20e9af Introduce isTableOwner method for readability
  • 01-29 04:25:50 aecd3b2 Set admin role before setting system properties in storage format test
  • 01-29 04:25:50 bbc0233 Make HiveQueryRunner default user have admin role
  • 01-29 04:25:50 be8189c Implement SET ROLE in Hive Connector
  • 01-29 04:25:50 c603474 Consider enabled roles for permissions
  • 01-29 04:25:50 c8d8733 Implement SHOW ROLE GRANTS rewrite
  • 01-29 04:25:50 cae5a86 Add access control checks for SHOW ROLES
  • 01-29 04:25:50 cb2da7b Refactor GRANT/REVOKE in Hive
  • 01-29 04:25:50 ce26168 Add docs for SHOW ROLE GRANTS
  • 01-29 04:25:50 d455185 Remove hive privilege null check
  • 01-29 04:25:50 dd9b64c Introduce SET ROLE statement
  • 01-29 04:25:50 de9890c Refactor HivePrivilegeInfo
  • 01-29 04:25:50 e5f2784 Allow all for admin role
  • 01-29 04:25:50 e8f81d9 Prepare metastore interface to accept ROLE for GRANT/REVOKE
  • 01-29 04:25:50 eadfb43 Remove 'IN catalog' syntax from role management commands
  • 01-29 04:25:50 f147d32 Add SHOW ROLES to the parser
  • 01-29 04:25:50 f5fe03d Document role management
  • 01-29 04:25:50 f5fe24d Introduce ConnectorIdentity
  • 01-29 04:25:50 f958f81 Refactor RecordingHiveMetastore to make it consistent between releases
  • 01-29 04:25:50 fb230fb Implement SET ROLE
  • 01-29 04:25:50 fb93b6e Fix listing privileges of tables owned by an user
  • 01-29 11:41:03 00aa33b Fix creating non-bucketed empty partition
  • 01-29 11:41:03 82574c8 ANALYZE statement: Parser
  • 01-29 11:41:03 82bc87f Add error message on column count mismatch when making partition name
  • 01-29 11:41:03 b747ecb ANALYZE statement: Implement ANALYZE in Hive connector
  • 01-29 11:41:03 be61588 Validate no partition value is null when making partition name
  • 01-29 11:41:03 df3f4df ANALYZE statement: Analyzer, planner and execution
  • 01-29 11:41:03 ecd0c0f Move getPartitionLocation to MetastoreUtil
  • 01-29 11:41:03 f800115 Fix partition column range statistics for empty partitions
  • 01-29 11:50:23 0c7d4ed Reduce parquet logging in Hive tests
  • 01-29 11:51:10 23f8793 Use processedInput* metrics in PipelineContext#getInput*

Martin Traverso

  • all checked
  • 01-22 15:20:16 bb0fd12 Update travis slack notification token
  • 01-25 14:17:15 d04152e Remove deprecated.legacy-log-function feature toggle
  • 01-25 14:17:15 f6044e6 Add standard log function
  • 01-25 14:27:45 ac02f3d Simplify coalesce expressions
  • 01-25 14:27:45 cf17d08 Flatten coalesce expressions
  • 01-25 19:31:27 a63829d Remove legacy array_agg semantics
  • 01-27 12:58:31 7c1250a Enable fusion of operators for queries with SYSTEM sampling

Piotr Findeisen

  • all checked
  • 01-24 12:52:31 0495b79 Add TYPE_NAME to JdbcTypeHandle
  • 01-24 12:52:44 a13d5d7 Remove unsupported types on write path in JDBC connectors
  • 01-24 12:52:44 b6faf92 Remove unsupported types on read path in JDBC connectors
  • 01-24 12:54:40 1455bcf Add section explaining legacy and new timestamp
  • 01-26 09:27:49 23e709f Fix port allocation race condition in EmbeddedZookeeper
  • 01-26 09:27:49 6d2cbce Use KafkaServer directly
  • 01-26 09:27:49 a881632 Code cleanup
  • 01-26 09:27:49 b0f1795 Update Kafka client to 0.10.2.2
  • 01-26 09:27:49 b410577 Fix port allocation race condition in EmbeddedKafka
  • 01-29 02:08:04 3fef5b6 Report top memory consumers when local memory limit is exceeded
  • 01-29 02:49:52 6dada93 Update to docker images version 11
  • 01-29 02:49:52 a1c4802 Support LDAP with anonymous bind disabled
@dain
Copy link
Member

dain commented Jan 24, 2019

SPI
---
Add session trace token to ``EventListener`` callbacks.

@dain
Copy link
Member

dain commented Jan 24, 2019

SPI
---
Fix regression in ``NodeManager`` where node list was not being refreshed on workers.

@martint
Copy link
Member

martint commented Jan 25, 2019

* Add  :func:`log` function with arbitrary base.
* Remove the ``deprecated.legacy-log-function`` configuration option. The legacy behavior for the :func:`log` function is no longer available.

#36

@martint
Copy link
Member

martint commented Jan 25, 2019

* Improve performance of `coalesce` expressions.

#35

@martint
Copy link
Member

martint commented Jan 26, 2019

* Remove the ``deprecated.legacy-array-agg`` configuration option. The legacy behavior for :func:`array_agg` is no longer available.

@electrum
Copy link
Member Author

* Improve error message for unsupported :func:`reduce_agg` state type.

@findepi
Copy link
Member

findepi commented Jan 26, 2019

Kafka Connector
-----
* Update Kafka client to 0.10.2.2. Kafka servers of older versions are no longer supported.

#53

@findepi
Copy link
Member

findepi commented Jan 29, 2019

Security
-----
* Support LDAP with anonymous bind disabled

#97

@martint
Copy link
Member

martint commented Jan 30, 2019

* Improve performance of queries involving ``SYSTEM`` table sampling and computations over the columns of the sampled table.

#29

@electrum
Copy link
Member Author

electrum commented Jan 30, 2019

General
---
* Improve error message for unsupported :func:`reduce_agg` state type (:issue:`55`).
* Add ``INVOKER`` security mode for :doc:`/sql/create-view` (:issue:`30`).

@kokosing
Copy link
Member

General
--------
 * Prevent RPM uninstallation when Presto server is not stopped correctly

#67

@kokosing
Copy link
Member

Hive Connector
-----------
 * Introduce ``hive.temporary-staging-directory-enabled`` configuration property and ``temporary_staging_directory_enabled`` session property to control if temporary staging directory should be used for write operations.
 * Introduce ``hive.temporary-staging-directory-path`` configuration property and ``temporary_staging_directory_path`` session property to control the location of temporary staging directory that is used for write operations. Use ${USER} placeholder to use different location for each user.

#70

@kokosing
Copy link
Member

Base-JDBC Connector
----
 * Allow Base-JDBC based connectors to define procedures.

#73

@kokosing
Copy link
Member

Base-JDBC Connector
----
 * Allow Base-JDBC based connectors to provide table statistics.

#72

@kokosing
Copy link
Member

Base-JDBC Connector
----
 * Add custom additional predicate to ``JdbcSplit`` that could be used when accessing data from underlying RDBMS.

#71

@findepi
Copy link
Member

findepi commented Jan 31, 2019

@kokosing #71, #72, #73 -- these are not user-facing changes. I think we usually do not list such changes in RNs. Should we?

@kokosing
Copy link
Member

@kokosing #71, #72, #73 -- these are not user-facing changes.

It depends. If user implemented his own Base-JDBC based connector (which happens) then it could be user facing One can see Base-JDBC in the same way as SPI. I also have a mixed feelings about it, but came to conclusion that it is better to say too much here.

@sopel39
Copy link
Member

sopel39 commented Jan 31, 2019

General
-------

* Add support for ROLE management including CREATE ROLE, DROP ROLE, GRANT ROLE, REVOKE ROLE, SET ROLE, SHOW CURRENT ROLES, SHOW ROLES and SHOW ROLE GRANTS commands.
  Using SET ROLE requires an up to date client.
* Add ANALYZE SQL statement.
* Fix computing of processed input data size in pipeline and task stats.

Hive Connector
--------------

* Add procedure for dumping Metastore recording.
* Add Metastore recorder support to Glue.

@dain dain closed this as completed in #115 Jan 31, 2019
chancez pushed a commit to chancez/presto that referenced this issue Nov 5, 2019
test.sh: mvn test presto-tests
yuuteng added a commit to yuuteng/trino that referenced this issue Jun 30, 2023
# This is the 1st commit message:

Add Snowflake JDBC Connector

# This is the commit message #2:

Update trino snapshot version to 372

# This is the commit message trinodb#3:

Update format of the doc of snowflake

# This is the commit message trinodb#4:

Update trino jdbc library import

# This is the commit message trinodb#5:

Fix date formatter from yyyy to uuuu

# This is the commit message trinodb#6:

Fix date test case

# This is the commit message trinodb#7:

Remove defunct property allow-drop-table

# This is the commit message trinodb#8:

Update trino version to 374

# This is the commit message trinodb#9:

Update snowflake config to adapt 374

# This is the commit message trinodb#10:

Update the range of the test of Date type

# This is the commit message trinodb#11:

Update to version 375

# This is the commit message trinodb#12:

Fix snowflake after updating to 375

# This is the commit message trinodb#13:

Update to 381

# This is the commit message trinodb#14:

Fix mvn pom import

# This is the commit message trinodb#15:

Format snowflake.rst

# This is the commit message trinodb#16:

Reorderd Data tests in type mapping

# This is the commit message trinodb#17:

Update function code

# This is the commit message trinodb#18:

Add product test

# This is the commit message trinodb#19:

Rename product test tablename

# This is the commit message trinodb#20:

Add Env, Suite and properties of Snowflake for production test

# This is the commit message trinodb#21:

Add trinoCreateAndInsert()

# This is the commit message trinodb#22:

Refactor snowflake from single node to multi node

# This is the commit message trinodb#23:

Pass product tests

# This is the commit message trinodb#24:

Removed snowflake.properties in trino server dev

# This is the commit message trinodb#25:

Resolved issues 19 05 2022 and fixed tests

# This is the commit message trinodb#26:

Remove Types.VARBINARY

# This is the commit message trinodb#27:

Add private static SliceWriteFunction charWriteFunction

# This is the commit message trinodb#28:

Update test case

# This is the commit message trinodb#29:

Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake/SnowflakeClient.java

Co-authored-by: Yuya Ebihara <[email protected]>
# This is the commit message trinodb#30:

Update docs/src/main/sphinx/connector/snowflake.rst

Co-authored-by: Yuya Ebihara <[email protected]>
# This is the commit message trinodb#31:

Update plugin/trino-snowflake/pom.xml

Co-authored-by: Yuya Ebihara <[email protected]>
# This is the commit message trinodb#32:

Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake/SnowflakeClient.java

Co-authored-by: Yuya Ebihara <[email protected]>
# This is the commit message trinodb#33:

Resolved review open issues

# This is the commit message trinodb#34:

Disabled JDBC_TREAT_DECIMAL_AS_INT and fixed test case

# This is the commit message trinodb#35:

Updated properties file

# This is the commit message trinodb#36:

Updated properties files

# This is the commit message trinodb#37:

Renamed properties in Testing

# This is the commit message trinodb#38:

Revert "Renamed properties in Testing"

This reverts commit 82f9eb3f3811e8d90a482f5359e98e7c729afa17.

# This is the commit message trinodb#39:

Renamed properties and fixed tests

# This is the commit message trinodb#40:

Update the way to pass ENV values for production test

# This is the commit message trinodb#41:

Update trino version to 388

# This is the commit message trinodb#42:

Update Trino version 391

# This is the commit message trinodb#43:

Update trino version to 394

# This is the commit message trinodb#44:

Update to 395

# This is the commit message trinodb#45:

Update to 411

# This is the commit message trinodb#46:

Update and fix errors

# This is the commit message trinodb#47:

Build successfully with 411

# This is the commit message trinodb#48:

Adding Bloomberg Snowflake connector.

Fully tested with Trino 406. Untested with 410.

Fix version number problem with build.

Adding varchar type to mapping Snowflake.

Adding --add-opens to enable support for Apache Arrow via shared memory buffers.

Fixing some tests.

Fix TestSnowflakeConnectorTest.

TODO: testDataMappingSmokeTest: time and timestamp
testSetColumnTypes: time and timestamp

Fix type mapper

Fix testconnector

Remove unused argument from DeltaLakeMetastore.getTableLocation

Extract removeS3Directory into DeltaLakeTestUtils

Additionally, replace toUnmodifiableList with toImmutableList.

Extract method in HiveMetastoreBackedDeltaLakeMetastore

Don't install flush_metadata_cache procedure in Iceberg

The procedure was unusable because Iceberg connector always
disables caching metastore.

Flush transaction log cache in Delta flush_metadata_cache procedure

Co-Authored-By: Marius Grama <[email protected]>

Remove extra digest copy during Digest merge

Reduce number of TDigestHistogram allocations

On coordinator operators stats from all tasks
will be merged. It does make sense to perform
merging as bulk operation.

Tune JDBC fetch-size automatically based on column count

PostgreSQL, Redshift and Oracle connectors had hard-coded fetch-size
value of 1000. The value was found not to be optimal when server is far
(high latency) or when number of columns selected is low. This commit
improves in the latter case by picking fetch size automatically based on
number of columns projected. After the change, the fetch size will be
automatically picked in the range 1000 to 100,000.

Remove redundant LongDoubleState interface

Fix import of wrong Preconditions class

Test Iceberg cost-based plans with small files on TPC-DS

Test against unpartitioned small Parquet files

Upgrade Pinot libraries to 0.12.1

Simplify MaxDataSizeForStats and SumDataSizeForStats

Block#getEstimatedDataSizeForStats is well defined for null positions.
We can use this to replace NullableLongState with LongState.

Fix formatting and simplify condition in HiveMetadata

Skip listing Glue tables with invalid column types

Exclude all synthetic columns in applyProjection validation

DefaultJdbcMetadata#applyProjection already excludes the delete row id
from validation. Same is now done for the merge row id column as well.

Fail fast on unexpected case

Fail, instead of returning, on an impossible case that was supposed to
be handled earlier in a method.

Remove redundant accessor calls

Leverage information already had within the method.

Remove Iceberg, Delta $data system table

It was not intentional to expose a table's data as `a_table$data`
"system" table. This commit removes support for these tables.

Encapsulate table name class constructor

Encapsulate constructors of IcebergTableName and DeltaLakeTableName. The
classes are used primarily as utility classes. Constructor encapsulation
is preparation to convert them into proper utility classes.

Remove table name/type `from` parsing method

After recent changes it became used only in tests. This also converts
the IcebergTableName, DeltaLakeTableName into utility classes.

Add more mirrors

Future-proof against Ubuntu codename update

This approach works as long the following assumptions hold:
- the location of `/etc/os-release` file does NOT change
- the name of the UBUNTU_CODENAME environment variable does NOT change
- `eclipse-temurin` still uses Ubuntu as it's base

Bring Docker build.sh help message in-line with reality

Add retries in TestImpersonation

Decrease number of old JDBC drivers tested

Test parquet column mapping using name based and index based mapping

Change t_char in AbstractTestHiveFileFormats to unpartitioned column

Set thread name for DDL queries

Before the change, the DDL tasks where being executed with a thread name
of `dispatch-query-%d`.

Fix thread names in TestEventDrivenTaskSource

Remove unused constant in environment definition

Capture column names in LocalQueryRunner

Remove un-necessary projected call with assertThat in redshift connector

Introduce new methods projected & exceptColumns take string varargs

For better readability, replace projected(int... columns) with
projected(String... columnNamesToInclude) and introduce
exceptColumns(String... columnNamesToExclude) leveraging
MaterializedResult.getColumnNames

Access fields directly in DeltaLakeTableHandle.withProjectedColumns

This makes it consistent with other methods like that, in particular
with `DeltaLakeTableHandle.forOptimize`.

Provide schema name in exception when Delta table lacks metadata

Test Delta connector behavior for a corrupted table

Handle corrupted Delta Lake tables with explicit table handle

Previously, a corrupted table handle was represented with a
`DeltaLakeTableHandle` missing a `MetadataEntry`.  When drop support for
corrupted tables was implemented, a connector could have only one table
handle class.  This commit improves the distinction by introducing a
dedicated table handle class to represent corrupted tables. As a
necessity, this class is explicitly handled in multiple
`DeltaLakeMetadata` methods. This sets viable example to follow for
implementing drop support for corrupted Iceberg tables as a follow-up.

Note: `testGetInsertLayoutTableNotFound` test was removed, instead of
being updated, since `ConnectorMetadata.getInsertLayout` cannot be
reached for a corrupted table, as getting column list will fail earlier.

Remove unnecessary override method in phoenix metadata

Make queryModifier final

Refactor testRenameTableToLongTableName to remove Iceberg override

Refactor `testRenameTableToLongTableName` test so that Iceberg tests do
not have to override the test method.

Improve Iceberg test testCreateTableLikeForFormat code formatting

Improve retries in AbstractTestHiveViews

Previously we were retrying on "Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask". As CI shown, the return
code may vary, sometimes it is e.g. 1.

In the meantime we introduced broader retry patterns for Hive query
failures, so let's use these.

Fix TestIcebergInsert.testIcebergConcurrentInsert timeout on CI

The test seems dominated by `finishInsert` time. since
bf04a72 `finishInsert` is slower as we
commit twice (first committing data then statistics).

Reduce lock contention in SqlStage

Threads in the application were blocked on locks for a total of 4 h 19 min before this patch and 2 h 59 min after in
concurrent benchmark with 40 nodes and 64 queries in parallel.

Reduce lock contention in Query

Use keySet in execution-query-purger

Make setup of bucketed tables in HiveQueryRunner optional

Bucketed tables are unnecessary in many tests

Remove superfluous accumulator add function

 - Enhance the test case as well

Add test for partitioned by non-lowercase column in Delta

Fix failure when partition column contains uppercase in Iceberg

Remove duplicate getParquetType method from ParquetPageSourceFactory

Remove redundant boolean state from LongDecimalWithOverflowAndLongState

Reduce synchronization on PipelinedStageExecution

Removes synchronization from beginScheduling() and
transitionToSchedulingSplits() both of which only perform state machine
updates (if necessary) and do not require accessing synchronized state.

Add NullablePosition to SumDataSizeForStats

Avoids an extra null check as block.getEstimatedDataSizeForStats
will also check for null

Remove unused HiveHudiPartitionInfo.getTable method

Co-Authored-By: Will Zhang <[email protected]>

Support DELETE statement in Ignite connector

Add an example JDBC connector plugin

Fix typo

Split createDriverRunner method for partitioned and unpartitioned cases

Use checkArgument formatting in StatementUtils

Avoids an eager and unnecessary String.format call by letting
checkArgument perform the required formatting only when the check
fails.

Avoid String.format in ExpressionFormatter

Also replaces unnecessary usages of Guava's Joiner in favor of
Collectors.joining where appropriate.

Replace String.format with String concat

Replaces simple String.format usages in non-exceptional code paths
with simple string concatenations where applicable.

Fix bad parameter count in code generator when column uses two slots

Document hive.max-outstanding-splits-size property

Add missing groups to testMigrateHiveBucketedOnMultipleColumns

Remove unused updatedColumns from IcebergTableHandle

Remove OkHttp as a runtime dependency for the engine

Remove unused dependencies from discovery-server

These are not used in embedded mode.

Update to ASM 9.4

Update object storage definition in glossary

Improve size accounting of SingleLongDecimalWithOverflowState

Make classes in LongDecimalWithOverflowAndLongStateFactory private final

Improve size accounting of SingleLongDecimalWithOverflowAndLongState

Simplify DecimalAverageAggregation#inputShortDecimal

Remove unsed NullableBooleanState

Fix Kerberos ticket refresh

The Hadoop UGI class handles ticket refresh only if the Subject is not
provided externally. For external Subject UGI expects the refresh will
be handled by the creator of the Subject which in our case we did not
do.

Because of this before this change any Trino query which ran longer than
the ticket_lifetime failed with errors like

    GSS initiate failed [Caused by GSSException: No valid credentials
    provided (Mechanism level: Failed to find any Kerberos tgt)].

In Hadoop code the UGI instance also gets re-used in some places (e.g.
DFSClient) which means we cannot just create a new UGI with refreshed
credentials and return that since other parts of code will keep using
the old UGI with expired credentials. So the fix is to create a new UGI,
extract the credentials from it and update the existing UGI's
credentials with them so that all users of the existing UGI also observe
the new valid credentials.

Extend the list of status codes retried in client

This commit extends list of codes on which client will retry to:
 * HTTP_BAD_GATEWAY (502)
 * HTTP_UNAVAILABLE (503)
 * HTTP_GATEWAY_TIMEOUT(504)

Allow listening for single container events

Make environment listener always required

Used enhanced switch

Remove redundant local variable

Remove redundant throws

Use StringBuilder instead of string concatenation

Fix typo in hive parquet doc

Add test for trailing space in location in hive metadata

Remove unnecessary and brittle tests

Column names coming out of the query are not necessarily
related to the column names in the table function. These
tests are testing behavior that is not necessarily expected
or guaranteed, so they are brittle and can break at any time.

A couple of reasons why it's problematic:
* Trino doesn't (yet) follow standard SQL identifier semantics. The
  column names might change between the output of the table function
  and the query output
* At the query output all columns have names. Within the query they
  might not. A table function can produce an anonymous column, but
  the test will see "_col0".

Upgrade Confluent version to 7.3.1

Updates transitive dependencies for Avro and ZooKeeper.
Wire 4.x is required for Confluent 7.3.1 and is updated
in the modules that need it, but leaves Wire at 3.x for
the remaining modules.

Fix potential Kerberos failure with SymlinkTextInputFormat

Add benchmark for array filter object

Optimize filter function performance with copyPositions

Before the change:
Benchmark                             (name)  Mode  Cnt   Score   Error  Units
BenchmarkArrayFilter.benchmark        filter  avgt   20  22.543 ± 0.979  ns/op
BenchmarkArrayFilter.benchmarkObject  filter  avgt   20  42.045 ± 2.088  ns/op

After the change:
Benchmark                             (name)  Mode  Cnt   Score   Error  Units
BenchmarkArrayFilter.benchmark        filter  avgt   20  13.327 ± 0.359  ns/op
BenchmarkArrayFilter.benchmarkObject  filter  avgt   20  34.443 ± 1.943  ns/op

Add quantile_at_value function

Co-authored-by: Peizhen Guo <[email protected]>

Use parent partitioning for aggregations

If parent partitioning provides enough parallelism,
and is a subset of the current node preferred
partitioning (grouping keys for the aggregation) we
can use the parent partitioning to skip data shuffle
required by the parent.

Extract MappedPageSource and MappedRecordSet to toolkit

Introduce BaseJdbcConnectorTableHandle

Extract methods in BaseJdbcClient

These methods can be reused for Procedures PTF
- Extract building columns from ResultSetMetaData as a separate method.
- Extract creating connection based on session

Add table function to execute stored procedure in SQLServer

Use URI path for Glue location in tests

Glue started throwing "InvalidInputException: One or more inputs failed validation"
when getting a table if the table location doesn't have "file:" prefix
in case of local file system.

Test trino-main with JDK 20

Clarify comment in BigQuery ReadSessionCreator

Consistently handle table types across BigQuery connector

This also fixes a bug where createEmptyProjection failed for non-TABLE
and non-VIEW even though those could be supported.

Combine some redundant tests in BigQuery

Remove duplicate test case

Disable CSV quoting when quote character is zero

Disable CSV escaping when escape character is zero

Fix race condition in the hive table stats cache

putIfAbsent method is not implemented in the EvictableCache
because of race condition with invalidation so to avoid the race
condition we use AtomicReference that at some cases can be thrown
away, but it makes cached value fresh even if invalidation happens
during value load

Provide convenience overload to get MV storage table in test

Setup global state before test methods

`storageSchemaName` is defined on class level, so the storage schema
should be created in `@BeforeClass`, not within a test.

Allow Iceberg MV with partitioning transforms on timestamptz

Allow creation of Iceberg Materialized Views partitioned with a
temporal partitioning function on a `timestamp with time zone` column.

In MVs, the `timestamp with time zone` columns are generally stored as
text to preserve time zone information. However, this prevents use of
temporal partitioning functions on these columns. The commit keeps
`timestamp with time zone` columns with partitioning applied on them as
`timestamp with time zone` in the storage table.

An obvious downside to this approach is that the time zone information
is erased and it is not known whether this aligns with user intention or
not. A better solution would be to introduce a point-in-time type
(trinodb#2273) to discern between the
cases where time zone information is important (like Java's
`ZonedDateTime`) from cases where only point-in-time matters (like
Java's `Instant`).

Remove backticks from backtick-unrelated test cases

They were probably copied over from the preceding backtick test case.

Reuse TrackingFileSystemFactory between connectors

Move TrackingFileSystemFactory out of Iceberg tests to allow reuse e.g.
with Delta Lake tests.

Refactor Delta file operations tests to encapsulate checked code

Pair tested operation and expected filesystem access counts in a single
assertion calls. Similar to how it's done in
`TestIcebergMetadataFileOperations`.

Convert TestIcebergMetadataFileOperations helper to record

Add Trino 411 release notes

[maven-release-plugin] prepare release 411

[maven-release-plugin] prepare for next development iteration

Enhance test for managed and external delta table location validation

The purpose of deleting the transaction log directory is solely to confirm
that when the DROP TABLE command is used, the table location is also removed
when table is MANAGED TABLE.

Improve naming of methods and fields to match Trino concepts

Fix incorrect result when hidden directory exist in migrate procedure

Add support for ADD COLUMN in Ignite connector

Use a more specific name for all connectors smoke test suite

Document the all connectors smoke test suite

Verify pass-through specifications in tests

Add more detailed check for TableFunctionProcessorNode.passThroughSpecifications
in TableFunctionProcessorMatcher.

Prune unreferenced pass-through columns of table function

Verify required columns in tests

Add check for TableFunctionProcessorNode.requiredSymbols
in TableFunctionProcessorMatcher.

Verify hashSymbol in tests

Add check for TableFunctionProcessorNode.hashSymbol
in TableFunctionProcessorMatcher.

Prune unreferenced columns of table function source

Test table function column pruning in query plan

Remove table function with empty source

Adds an optimizer rule to remove TableFunctionProcessorNode with
source being an empty relation, based on the "prune when empty"
property.

Test pruning of redundant table function in query plan

Test table functions with column and node pruning optimizations

Fix typo in TestDeltaLakePerTransactionMetastoreCache

Extract assertMetastoreInvocations method

Use CountingAccessHiveMetastore in TestDeltaLakePerTransactionMetastoreCache

Make cleanup methods alwaysRun

This attribute says that this after-method will get executed even if the
methods executed previously failed or were skipped. Note that it also
applies to skipped test methods, so if the tests were skipped for some
reason, the cleanup won't run. This attribute will ensure that the
clean-up will run even in this case.

Failure to run clean up may cause secondary effects, especially our
resource leak detector; failure on this will in turn mask other errors,
the ones which caused the tests to be skipped in the first place.

Add a check to enforce alwaysRun = true on test after-methods

See the previous commit for details. This check will enforce that the
`alwaysRun = true` is present.

Remove redundant call toString()

Remove use of deprecated isEqualToComparingFieldByFieldRecursively

Use usingRecursiveComparison instead deprecated isEqualToComparingFieldByFieldRecursively

Remove unused helper methods in delta-lake connector

Add an explict config to define standardSplitSizeInBytes in FTE

Implement adaptive task sizing for arbitrary distribution in FTE

Improve task sizing for hash distribution in FTE

Round up targetPartitionSizeInBytes to a multiple of minTargetPartitionSizeInBytes

For adaptive task sizing in ArbitraryDistributionSplitAssigner

Adjust retry policy for dropping delta tables backed by AWS Glue on Databricks

Fix output rendering issue in docs

Alphabetize glossary entries

Add use_cost_based_partitioning

Use use_cost_based_partitioning instead of use_exact_partitioning to
control the cost based optimization to prefer parent partitioning.
The motivation is to be able to disable the optimization if the NDV
statistics are overestimated and the optimization would hurt parallelism.

Provide injection mechanism for the file system factory

Reorder instance and static fields in FlushMetadataCacheProcedure

Flush extended statistics in Delta's flush_metadata_cache()

Clean up Delta code a bit

Test Delta Lake query file system accesses

Ensure that TestingHydraIdentityProvider is releasing resources

Update maven to 3.9.1

Expose rule stats in QueryStats

Expose optimizer rule statistics per query in QueryInfo JSON. The number of rules exposed could be
adjusted using the `query.reported-rule-stats-limit` configuration parameter.

Cleanup BigintGroupByHash instanceof checks

Include QueryId in assertDistrubutedQuery failure message

Remove TestClickHouseConnectorTest

There are 4 smoke tests and 2 connectors test.
Remove TestClickHouseConnectorTest as a redundant test.

Remove base class for ClickHouse connector test

Run smoke test for BigQuery arrow serialization

We want to verify SELECT behavior for Arrow serialization in BigQuery.
Smoke test and the existing type mapping test should be enough.

Make construction parameters final

Support arithmetic predicate pushdown for Phoenix

Make LeafTableFunctionOperator handle EmptySplit correctly

Remove CatalogHandle from TableFunctionProcessorNode

Exclude snakeyaml from elasticsearch dependencies

It's a transitive dependency of elasticsearch-x-content,
which we use in ElasticsearchLoader to load tpch data
to Elasticsearch with json encoding. Yaml support is not
needed at all.

Pass partition values as optional to DeltaLakePageSource

The partition values list is filled only when row ID column is
projected, so it's a conditional information. When row ID is not
present, pass it as the empty optional, rather than list that happens to
be empty.

Add cleaner FixedPageSource constructors

Previously, the only constructor would take `Iterable`, which is nice,
but it would also materialize it twice (once in the constructor to
calculate memory usage).

The commit adds a constructor taking a `List` (so double iteration is not
a problem) and one taking `Iterator` and delivering on the promise to
iterate once.

The old constructor is kept deprecated, but apparently all usages use
the new, list-based constructor.

Project a data column in MinIO access test

Read a data column to ensure the data file gets read.
This increases number of accesses to a file, because both footer and
data are being read.

Accelerate Delta when reading partition columns only

Regenerate expected test plans with one-click

Traverse into JsonObject members in AST

Before this change, JsonObject members were not visited
in AstVisitor. As a result, aggregations or parameters
inside the  members were not supported.

Traverse into JsonArray elements in AST

Before this change, JsonArray elements were not visited
in AstVisitor. As a result, aggregations or parameters
inside the elements were not supported.

Document avro.schema.literal property use for interpreting table data

Update Oracle JDBC driver version to 21.9.0.0

Document predicate pushdown support for string-type columns in SQL Server

Enable oracle.remarks-reporting.enabled in connector test

Remove unnecessary wrapping of IOException in TransactionLogTail

Translate `The specified key does not exist` to FileNotFoundException

Relax test assertion

It is possible for more than one task to fail due to injected failure.

Remove obsolete assertion

Lowercase bucketing and sort column names

In the metastore, the bucketing and sorting column names can differ
in case from its corresponding table column names.
This change makes certain that, even though a table can be
delivered by the metastore with such inconsistencies, Trino will lowercase
the same bucketing and sort column names to ensure they correspond to the
data column names.

Add test for RenameColumnTask

Migrate assertStatement in TestSqlParser.testRenameColumn

Allow configuring a custom DNS resolver for the JDBC driver

Reorganize Hudi connector documentation

Migrate some assertExpression in TestSqlParser

Look for a non-Trino protocol without using X-User-*

Update ASM to 9.5

Reorganize Iceberg connector documentation

Reorganize Delta Lake connector documentation

Fix handling of Hive ACID tables with hidden directories

Test CREATE TABLE AS SELECT in Ignite type mapping

Additionally, check time zones in setUp method.

Override equals and hashCode in Delta Identifier

Change Object to Row in testCheckConstraintCompatibility

Support arithmetic binary in Delta check constraints

Introduce EmptyTableFunctionHandle as default handle

Before this change, if a table function did not pass
a ConnectorTableFunctionHandle in the TableFunctionAnalysis,
the default handle was used, which was an anonymous
implementation of ConnectorTableFunctionHandle.

It did not work with table functions executed by operator,
due to lack of serialization.

This change introduces EmptyTableFunctionHandle, and sets
it as default.

Support returning anonymous columns by table functions

Per SQL standard, all columns must be named. In Trino,
we support unnamed columns.
This change adjusts table functions so that they can return
anonymous columns.
It is achieved by changing the Descriptor structure so that
field name is optional. This optionality can only be used
for the returned type of table functions. Descriptor arguments
pased by the user as well as default values for descriptor
argumens have mandatory field names.

Add table function `exclude_columns`

Bump spotbugs-annotations version

Airbase already has `4.7.3`

Remove ValidateLimitWithPresortedInput

It's not powerful enough to validate properties of plans
that get modified by predicate pushdown after AddExchanges
runs, resulting in false positives such as trinodb#16768

Use OrcReader#MAX_BATCH_SIZE = 8 * 1024

Previous value 8196 was bigger than PageProcessor#MAX_BATCH_SIZE
causing PageProcessor to create small, 4 position pages every other page.

Bring DistinguishedNameParser from okhttp3

Inline OkHostnameVerifier and Util.verifyAsIpAddress

They were removed in OkHttp 4.x and we still rely on the legacy SSL hostname verification

Update okhttp to 4.10.0

Fix testTableWithNonNullableColumns to update NOT NULL column in Delta

Add  for support creating table with comment for more Jdbc based connectors (trinodb#16135)

Allow PostgreSQL create table with comment

Also allow to set a comment for PostgreSQL tables

Support sum(distinct) for jdbc connectors

Add Trino 412 release notes

[maven-release-plugin] prepare release 412

[maven-release-plugin] prepare for next development iteration

Add doc for ignite join pushdown

Add missing config properties to Hive docs

Co-Authored-By: Marius Grama <[email protected]>

Fix layout in Iceberg documentation

Add docs for property to skip glue archive

Fix typo

Support nested timestamp with time zone in Delta Lake

Add test for duplicated partition statistics on Thrift metastore

Support table comment for oracle connector

Allow Oracle create table with comment

Also allow to set table comment for Oracle tables

Support MERGE for Phoenix connector

Remove unused class FallbackToFullNodePartitionMemoryEstimator.

Remove unnecessary dependency management in trino-pinot

The `protobuf-java` one was overriding a corresponding declaration in
the parent POM, and was effectively downgrading it. The other two were
not used at all.

Bump Protobuf version

Add Snowflake JDBC Connector

Update trino snapshot version to 372

Update format of the doc of snowflake

Update trino jdbc library import

Fix date formatter from yyyy to uuuu

Fix date test case

Remove defunct property allow-drop-table

Update trino version to 374

Update snowflake config to adapt 374

Update the range of the test of Date type

Update to version 375

Fix snowflake after updating to 375

Update to 381

Fix mvn pom import

Format snowflake.rst

Reorderd Data tests in type mapping

Update function code

Add product test

Rename product test tablename

Add Env, Suite and properties of Snowflake for production test

Add trinoCreateAndInsert()

Refactor snowflake from single node to multi node

Pass product tests

Removed snowflake.properties in trino server dev

Resolved issues 19 05 2022 and fixed tests

Remove Types.VARBINARY

Add private static SliceWriteFunction charWriteFunction

Update test case

Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake/SnowflakeClient.java

Co-authored-by: Yuya Ebihara <[email protected]>

Update docs/src/main/sphinx/connector/snowflake.rst

Co-authored-by: Yuya Ebihara <[email protected]>

Update plugin/trino-snowflake/pom.xml

Co-authored-by: Yuya Ebihara <[email protected]>

Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake/SnowflakeClient.java

Co-authored-by: Yuya Ebihara <[email protected]>

Resolved review open issues

Disabled JDBC_TREAT_DECIMAL_AS_INT and fixed test case

Updated properties file

Updated properties files

Renamed properties in Testing

Revert "Renamed properties in Testing"

This reverts commit 82f9eb3f3811e8d90a482f5359e98e7c729afa17.

Renamed properties and fixed tests

Update the way to pass ENV values for production test

Update trino version to 388

Update Trino version 391

Update trino version to 394

Update to 395

Update to 411

Update and fix errors

Build successfully with 411

Adding Bloomberg Snowflake connector.

Fully tested with Trino 406. Untested with 410.

Adding varchar type to mapping Snowflake.

Adding --add-opens to enable support for Apache Arrow via shared memory buffers.

Fixing some tests.

Fix TestSnowflakeConnectorTest.

TODO: testDataMappingSmokeTest: time and timestamp
testSetColumnTypes: time and timestamp

Fix type mapper

Fix testconnector

Update version to 413-SNAPSHOT

Added support for HTTP_PROXY testing.

Connector doesnt support setColumnType. This causes lots of problems with Snowflake server.

Disabled the testsetColumnTypes testing.

Fixed and skipped error tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

6 participants