Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow fetching relation names with their types #19832

Closed
wants to merge 4 commits into from
Closed

Allow fetching relation names with their types #19832

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9e310f0
Rename listTables to listRelations
findepi Nov 20, 2023
5ca3428
Rename listTables to listRelations in Hive
findepi Nov 20, 2023
0198cfb
Add getRelationTypes for listing relations with their types
findepi Nov 20, 2023
b979ffe
Implement getRelationTypes for Hive
findepi Nov 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 19 additions & 28 deletions client/trino-jdbc/src/test/java/io/trino/jdbc/TestTrinoDatabaseMetaData.java
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think "relation" is a clearer term. Views, materialized views, etc. are all just "tables" - at least in anything associated with the SQL spec. If we're going to attempt to distinguish regular tables from views and friends, then we should be talking about derived tables, base tables, views, temporary tables, etc. which are all just "tables". A "regular" table (i.e., not a view or materialized view) is just a "base table".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point and agree from sql perspective. "tables" is an overloaded term and sometimes it refers to regular tables (not views), sometimes to any relations (tables per above). The confusion is amplified by "list tables" and "list views" methods living in same interface (ConnectorMetadata, or HiveMetastore). They look complementary, and i've seen people confused about them, but they are not. The Trino SPI handles tables and views differently (eg getTableHandle vs getView), despite them being "just tables" from SQL perspective, making the confusion even easier.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Examples
#19842 (comment)
#11063

Original file line number Diff line number Diff line change
Expand Up @@ -1116,8 +1116,7 @@ public void testGetTablesMetadataCalls()
databaseMetaData -> databaseMetaData.getTables(null, null, null, null),
list("TABLE_CAT", "TABLE_SCHEM", "TABLE_NAME", "TABLE_TYPE")),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews")
.add("ConnectorMetadata.listTables")
.add("ConnectorMetadata.getRelationTypes")
.build());

// Equality predicate on catalog name
Expand All @@ -1127,8 +1126,7 @@ public void testGetTablesMetadataCalls()
databaseMetaData -> databaseMetaData.getTables(COUNTING_CATALOG, null, null, null),
list("TABLE_CAT", "TABLE_SCHEM", "TABLE_NAME", "TABLE_TYPE")),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews")
.add("ConnectorMetadata.listTables")
.add("ConnectorMetadata.getRelationTypes")
.build());

// Equality predicate on schema name
Expand All @@ -1142,8 +1140,7 @@ public void testGetTablesMetadataCalls()
.map(schemaTableName -> list(COUNTING_CATALOG, schemaTableName.getSchemaName(), schemaTableName.getTableName(), "TABLE"))
.collect(toImmutableList()),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews(schema=test_schema1)")
.add("ConnectorMetadata.listTables(schema=test_schema1)")
.add("ConnectorMetadata.getRelationTypes(schema=test_schema1)")
.build());

// LIKE predicate on schema name
Expand All @@ -1157,8 +1154,7 @@ public void testGetTablesMetadataCalls()
.map(schemaTableName -> list(COUNTING_CATALOG, schemaTableName.getSchemaName(), schemaTableName.getTableName(), "TABLE"))
.collect(toImmutableList()),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews")
.add("ConnectorMetadata.listTables")
.add("ConnectorMetadata.getRelationTypes")
.build());

// Equality predicate on table name
Expand All @@ -1171,8 +1167,7 @@ public void testGetTablesMetadataCalls()
list(COUNTING_CATALOG, "test_schema1", "test_table1", "TABLE"),
list(COUNTING_CATALOG, "test_schema2", "test_table1", "TABLE")),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews")
.add("ConnectorMetadata.listTables")
.add("ConnectorMetadata.getRelationTypes")
.build());

// LIKE predicate on table name
Expand All @@ -1185,8 +1180,7 @@ public void testGetTablesMetadataCalls()
list(COUNTING_CATALOG, "test_schema1", "test_table1", "TABLE"),
list(COUNTING_CATALOG, "test_schema2", "test_table1", "TABLE")),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews")
.add("ConnectorMetadata.listTables")
.add("ConnectorMetadata.getRelationTypes")
.build());

// Equality predicate on schema name and table name
Expand All @@ -1197,8 +1191,8 @@ public void testGetTablesMetadataCalls()
list("TABLE_CAT", "TABLE_SCHEM", "TABLE_NAME", "TABLE_TYPE")),
list(list(COUNTING_CATALOG, "test_schema1", "test_table1", "TABLE")),
ImmutableMultiset.<String>builder()
.addCopies("ConnectorMetadata.getSystemTable(schema=test_schema1, table=test_table1)", 5)
.addCopies("ConnectorMetadata.getView(schema=test_schema1, table=test_table1)", 2)
.addCopies("ConnectorMetadata.getSystemTable(schema=test_schema1, table=test_table1)", 4)
.add("ConnectorMetadata.getView(schema=test_schema1, table=test_table1)")
.add("ConnectorMetadata.getMaterializedView(schema=test_schema1, table=test_table1)")
.add("ConnectorMetadata.redirectTable(schema=test_schema1, table=test_table1)")
.add("ConnectorMetadata.getTableHandle(schema=test_schema1, table=test_table1)")
Expand All @@ -1212,8 +1206,7 @@ public void testGetTablesMetadataCalls()
list("TABLE_CAT", "TABLE_SCHEM", "TABLE_NAME", "TABLE_TYPE")),
list(list(COUNTING_CATALOG, "test_schema1", "test_table1", "TABLE")),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews")
.add("ConnectorMetadata.listTables")
.add("ConnectorMetadata.getRelationTypes")
.build());

// catalog does not exist
Expand Down Expand Up @@ -1337,7 +1330,7 @@ public void testGetColumnsMetadataCalls()
list(list(COUNTING_CATALOG, "test_schema1", "test_table1", "column_17", "varchar")),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listSchemaNames")
.add("ConnectorMetadata.listTables(schema=test_schema1)")
.add("ConnectorMetadata.listRelations(schema=test_schema1)")
.addCopies("ConnectorMetadata.getSystemTable(schema=test_schema1, table=test_table1)", 4)
.add("ConnectorMetadata.getMaterializedView(schema=test_schema1, table=test_table1)")
.add("ConnectorMetadata.getView(schema=test_schema1, table=test_table1)")
Expand All @@ -1357,7 +1350,7 @@ public void testGetColumnsMetadataCalls()
.collect(toImmutableList()),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listSchemaNames")
.add("ConnectorMetadata.listTables(schema=test_schema1)")
.add("ConnectorMetadata.listRelations(schema=test_schema1)")
.addCopies("ConnectorMetadata.getSystemTable(schema=test_schema1, table=test_table1)", 4)
.add("ConnectorMetadata.getMaterializedView(schema=test_schema1, table=test_table1)")
.add("ConnectorMetadata.getView(schema=test_schema1, table=test_table1)")
Expand Down Expand Up @@ -1395,10 +1388,10 @@ public void testGetColumnsMetadataCalls()
.collect(toImmutableList()),
ImmutableMultiset.<String>builder()
.addCopies("ConnectorMetadata.listSchemaNames", 5)
.add("ConnectorMetadata.listTables(schema=test_schema1)")
.add("ConnectorMetadata.listTables(schema=test_schema2)")
.add("ConnectorMetadata.listTables(schema=test_schema3_empty)")
.add("ConnectorMetadata.listTables(schema=test_schema4_empty)")
.add("ConnectorMetadata.listRelations(schema=test_schema1)")
.add("ConnectorMetadata.listRelations(schema=test_schema2)")
.add("ConnectorMetadata.listRelations(schema=test_schema3_empty)")
.add("ConnectorMetadata.listRelations(schema=test_schema4_empty)")
.addCopies("ConnectorMetadata.getSystemTable(schema=test_schema1, table=test_table1)", 20)
.addCopies("ConnectorMetadata.getMaterializedView(schema=test_schema1, table=test_table1)", 5)
.addCopies("ConnectorMetadata.getView(schema=test_schema1, table=test_table1)", 5)
Expand Down Expand Up @@ -1530,8 +1523,7 @@ private void testAssumeLiteralMetadataCalls(String escapeLiteralParameter)
.map(schemaTableName -> list(COUNTING_CATALOG, schemaTableName.getSchemaName(), schemaTableName.getTableName(), "TABLE"))
.collect(toImmutableList()),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews(schema=test_schema1)")
.add("ConnectorMetadata.listTables(schema=test_schema1)")
.add("ConnectorMetadata.getRelationTypes(schema=test_schema1)")
.build());

// getTables's schema and table name patterns treated as literals
Expand All @@ -1542,8 +1534,8 @@ private void testAssumeLiteralMetadataCalls(String escapeLiteralParameter)
list("TABLE_CAT", "TABLE_SCHEM", "TABLE_NAME", "TABLE_TYPE")),
list(list(COUNTING_CATALOG, "test_schema1", "test_table1", "TABLE")),
ImmutableMultiset.<String>builder()
.addCopies("ConnectorMetadata.getSystemTable(schema=test_schema1, table=test_table1)", 5)
.addCopies("ConnectorMetadata.getView(schema=test_schema1, table=test_table1)", 2)
.addCopies("ConnectorMetadata.getSystemTable(schema=test_schema1, table=test_table1)", 4)
.add("ConnectorMetadata.getView(schema=test_schema1, table=test_table1)")
.add("ConnectorMetadata.getMaterializedView(schema=test_schema1, table=test_table1)")
.add("ConnectorMetadata.redirectTable(schema=test_schema1, table=test_table1)")
.add("ConnectorMetadata.getTableHandle(schema=test_schema1, table=test_table1)")
Expand All @@ -1557,8 +1549,7 @@ private void testAssumeLiteralMetadataCalls(String escapeLiteralParameter)
list("TABLE_CAT", "TABLE_SCHEM", "TABLE_NAME", "TABLE_TYPE")),
list(),
ImmutableMultiset.<String>builder()
.add("ConnectorMetadata.listViews(schema=test_schema_)")
.add("ConnectorMetadata.listTables(schema=test_schema_)")
.add("ConnectorMetadata.getRelationTypes(schema=test_schema_)")
.build());

// getColumns's schema and table name patterns treated as literals
Expand Down
2 changes: 1 addition & 1 deletion ...no-main/src/main/java/io/trino/connector/informationschema/InformationSchemaMetadata.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,7 @@ private Set<QualifiedTablePrefix> calculatePrefixesWithTableName(
}

Set<QualifiedTablePrefix> tablePrefixes = prefixes.stream()
.flatMap(prefix -> metadata.listTables(session, prefix).stream())
.flatMap(prefix -> metadata.listRelations(session, prefix).stream())
.filter(objectName -> predicate.get().test(asFixedValues(objectName)))
.map(QualifiedObjectName::asQualifiedTablePrefix)
.distinct()
Expand Down
28 changes: 19 additions & 9 deletions ...-main/src/main/java/io/trino/connector/informationschema/InformationSchemaPageSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import io.trino.spi.connector.ColumnHandle;
import io.trino.spi.connector.ColumnMetadata;
import io.trino.spi.connector.ConnectorPageSource;
import io.trino.spi.connector.RelationType;
import io.trino.spi.connector.SchemaTableName;
import io.trino.spi.security.AccessDeniedException;
import io.trino.spi.security.GrantInfo;
Expand All @@ -37,6 +38,7 @@
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Optional;
import java.util.OptionalLong;
import java.util.Queue;
Expand All @@ -52,12 +54,12 @@
import static io.trino.SystemSessionProperties.isOmitDateTimeTypePrecision;
import static io.trino.connector.informationschema.InformationSchemaMetadata.defaultPrefixes;
import static io.trino.connector.informationschema.InformationSchemaMetadata.isTablesEnumeratingTable;
import static io.trino.metadata.MetadataListing.getRelationTypes;
import static io.trino.metadata.MetadataListing.getViews;
import static io.trino.metadata.MetadataListing.listRelations;
import static io.trino.metadata.MetadataListing.listSchemas;
import static io.trino.metadata.MetadataListing.listTableColumns;
import static io.trino.metadata.MetadataListing.listTablePrivileges;
import static io.trino.metadata.MetadataListing.listTables;
import static io.trino.metadata.MetadataListing.listViews;
import static io.trino.spi.security.PrincipalType.USER;
import static io.trino.spi.type.TypeUtils.writeNativeValue;
import static io.trino.type.TypeUtils.getDisplayLabel;
Expand Down Expand Up @@ -245,7 +247,7 @@ private void buildPages()

private void addColumnsRecords(QualifiedTablePrefix prefix)
{
for (Map.Entry<SchemaTableName, List<ColumnMetadata>> entry : listTableColumns(session, metadata, accessControl, prefix).entrySet()) {
for (Entry<SchemaTableName, List<ColumnMetadata>> entry : listTableColumns(session, metadata, accessControl, prefix).entrySet()) {
SchemaTableName tableName = entry.getKey();
long ordinalPosition = 1;

Expand Down Expand Up @@ -275,16 +277,24 @@ private void addColumnsRecords(QualifiedTablePrefix prefix)

private void addTablesRecords(QualifiedTablePrefix prefix)
{
Set<SchemaTableName> tables = listTables(session, metadata, accessControl, prefix);
boolean needsTableType = requiredColumns.contains("table_type");
Set<SchemaTableName> views = Set.of();
Set<SchemaTableName> relations;
Set<SchemaTableName> views;
if (needsTableType) {
// TODO introduce a dedicated method for getting relations with their type from the connector, instead of calling (potentially much more expensive) getViews
views = listViews(session, metadata, accessControl, prefix);
Map<SchemaTableName, RelationType> relationTypes = getRelationTypes(session, metadata, accessControl, prefix);
relations = relationTypes.keySet();
views = relationTypes.entrySet().stream()
.filter(entry -> entry.getValue() == RelationType.VIEW)
.map(Entry::getKey)
.collect(toImmutableSet());
}
else {
relations = listRelations(session, metadata, accessControl, prefix);
views = Set.of();
}
// TODO (https://github.com/trinodb/trino/issues/8207) define a type for materialized views

for (SchemaTableName name : tables) {
for (SchemaTableName name : relations) {
String type = null;
if (needsTableType) {
// if table and view names overlap, the view wins
Expand All @@ -304,7 +314,7 @@ private void addTablesRecords(QualifiedTablePrefix prefix)

private void addViewsRecords(QualifiedTablePrefix prefix)
{
for (Map.Entry<SchemaTableName, ViewInfo> entry : getViews(session, metadata, accessControl, prefix).entrySet()) {
for (Entry<SchemaTableName, ViewInfo> entry : getViews(session, metadata, accessControl, prefix).entrySet()) {
addRecord(
prefix.getCatalogName(),
entry.getKey().getSchemaName(),
Expand Down
6 changes: 3 additions & 3 deletions core/trino-main/src/main/java/io/trino/connector/system/jdbc/ColumnJdbcTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,9 @@
import static io.trino.connector.system.jdbc.FilterUtil.tablePrefix;
import static io.trino.connector.system.jdbc.FilterUtil.tryGetSingleVarcharValue;
import static io.trino.metadata.MetadataListing.listCatalogNames;
import static io.trino.metadata.MetadataListing.listRelations;
import static io.trino.metadata.MetadataListing.listSchemas;
import static io.trino.metadata.MetadataListing.listTableColumns;
import static io.trino.metadata.MetadataListing.listTables;
import static io.trino.metadata.MetadataUtil.TableMetadataBuilder.tableMetadataBuilder;
import static io.trino.spi.type.BigintType.BIGINT;
import static io.trino.spi.type.BooleanType.BOOLEAN;
Expand Down Expand Up @@ -211,7 +211,7 @@ TABLE_SCHEMA_COLUMN, toNullableValue(schemaName))))
QualifiedTablePrefix tablePrefix = tableFilter.isPresent()
? new QualifiedTablePrefix(schema.getCatalogName(), schema.getSchemaName(), tableFilter.get())
: new QualifiedTablePrefix(schema.getCatalogName(), schema.getSchemaName());
return listTables(session, metadata, accessControl, tablePrefix).stream()
return listRelations(session, metadata, accessControl, tablePrefix).stream()
.filter(schemaTableName -> predicate.test(ImmutableMap.of(
TABLE_CATALOG_COLUMN, toNullableValue(schema.getCatalogName()),
TABLE_SCHEMA_COLUMN, toNullableValue(schemaTableName.getSchemaName()),
Expand Down Expand Up @@ -278,7 +278,7 @@ public RecordCursor cursor(ConnectorTransactionHandle transactionHandle, Connect
QualifiedTablePrefix tablePrefix = tableFilter.isPresent()
? new QualifiedTablePrefix(catalog, schema, tableFilter.get())
: new QualifiedTablePrefix(catalog, schema);
Set<SchemaTableName> tables = listTables(session, metadata, accessControl, tablePrefix);
Set<SchemaTableName> tables = listRelations(session, metadata, accessControl, tablePrefix);
for (SchemaTableName schemaTableName : tables) {
String tableName = schemaTableName.getTableName();
if (!tableDomain.includesNullableValue(utf8Slice(tableName))) {
Expand Down
12 changes: 5 additions & 7 deletions core/trino-main/src/main/java/io/trino/connector/system/jdbc/TableJdbcTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,19 +26,18 @@
import io.trino.spi.connector.InMemoryRecordSet;
import io.trino.spi.connector.InMemoryRecordSet.Builder;
import io.trino.spi.connector.RecordCursor;
import io.trino.spi.connector.RelationType;
import io.trino.spi.connector.SchemaTableName;
import io.trino.spi.predicate.Domain;
import io.trino.spi.predicate.TupleDomain;

import java.util.Optional;
import java.util.Set;

import static io.trino.connector.system.jdbc.FilterUtil.isImpossibleObjectName;
import static io.trino.connector.system.jdbc.FilterUtil.tablePrefix;
import static io.trino.connector.system.jdbc.FilterUtil.tryGetSingleVarcharValue;
import static io.trino.metadata.MetadataListing.getRelationTypes;
import static io.trino.metadata.MetadataListing.listCatalogNames;
import static io.trino.metadata.MetadataListing.listTables;
import static io.trino.metadata.MetadataListing.listViews;
import static io.trino.metadata.MetadataUtil.TableMetadataBuilder.tableMetadataBuilder;
import static io.trino.spi.type.VarcharType.VARCHAR;
import static java.util.Objects.requireNonNull;
Expand Down Expand Up @@ -104,13 +103,12 @@ public RecordCursor cursor(ConnectorTransactionHandle transactionHandle, Connect
for (String catalog : listCatalogNames(session, metadata, accessControl, catalogDomain)) {
QualifiedTablePrefix prefix = tablePrefix(catalog, schemaFilter, tableFilter);

Set<SchemaTableName> views = listViews(session, metadata, accessControl, prefix);
for (SchemaTableName name : listTables(session, metadata, accessControl, prefix)) {
boolean isView = views.contains(name);
getRelationTypes(session, metadata, accessControl, prefix).forEach((name, type) -> {
boolean isView = type == RelationType.VIEW;
if ((includeTables && !isView) || (includeViews && isView)) {
table.addRow(tableRow(catalog, name, isView ? "VIEW" : "TABLE"));
}
}
});
}
return table.build().cursor();
}
Expand Down
2 changes: 1 addition & 1 deletion core/trino-main/src/main/java/io/trino/execution/DropSchemaTask.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,6 @@ private static boolean isSchemaEmpty(Session session, CatalogSchemaName schema,
QualifiedTablePrefix tablePrefix = new QualifiedTablePrefix(schema.getCatalogName(), schema.getSchemaName());

// This is a best effort check that doesn't provide any guarantees against concurrent DDL operations
return metadata.listTables(session, tablePrefix).isEmpty();
return metadata.listRelations(session, tablePrefix).isEmpty();
}
}
Loading