Feature: multi-catalog #4947

dantengsky · 2022-04-19T16:02:56Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

new feature: multiple catalogs

objects of catalog, e.g table, can be addressed as catalog.database.table
only the current ut/it are covered, there are definitely other codes that should be modified to fully support this
all the objects that are kept in databend meta server belong to the default catalog
new catalog can be registered by using the CatalogManager
currently, a simple impl of hive catalog is provided, which can be registered by using de configs.
- registered as catalog name hive
- can resolve metadata by using hive meta store
- a dummy implementation of hive database/table
  - only resolves simple and limit number of column types
  - returns empty dataset for queries
by default, the cargo feature hive of crate databend-query is NOT enabled
- to enable it
  cargo build --bin databend-query --features hive
- but the crate common-hive-meta-store will be built, even the hive feature is not specified
  - the only extra build time dependency is the thrift compiler which is built into dev docker images
  - if there is a way to conditionally compile a number of workspace, pls let me know.
A dedicate github action test_stateful_hive_standalone is added
- only linux + amd64 are tested
- setups hive cluster (and simple test data)

Changelog

New Feature
Not for changelog (changelog entry is not required)

Related Issues

Fixes #issue

vercel · 2022-04-19T16:03:01Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		May 10, 2022 at 10:17AM (UTC)

mergify · 2022-04-19T16:03:30Z

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

dantengsky · 2022-04-21T12:32:09Z

the files that thrift generated are HUGE. later, they will be generated by "build.rs"

dantengsky · 2022-05-10T11:14:45Z

query/src/storages/fuse/fuse_table.rs

-            engine_name: "FUSE".to_string(),
-            comment: "FUSE Storage Engine".to_string(),
-            support_order_key: true,
+    //    pub fn catalog_name(&self) -> Result<String> {


remove those

dantengsky · 2022-05-10T11:15:27Z

query/src/storages/fuse/fuse_table.rs

-        ctx: Arc<QueryContext>,
-        operations: Vec<DataBlock>,
-        overwrite: bool,
+        _ctx: Arc<QueryContext>,


rename these

dantengsky · 2022-05-10T11:16:20Z

query/src/storages/fuse/operations/commit.rs

@@ -242,7 +242,8 @@ impl FuseTable {
        table_info: &TableInfo,
        new_snapshot_location: String,
    ) -> Result<UpsertTableOptionReply> {
-        let catalog = ctx.get_catalog();
+        // TODO catalog name


use arg catalog_name

dantengsky · 2022-05-10T11:17:08Z

query/src/storages/fuse/table_functions/fuse_snapshots/fuse_snapshot_table.rs

@@ -200,7 +207,7 @@ impl AsyncSource for FuseHistorySource {
            let tenant_id = self.ctx.get_tenant();
            let tbl = self
                .ctx
-                .get_catalog()
+                .get_catalog(&self.catalog_name)? // TODO pass in this guy


clean up TODO

dantengsky · 2022-05-10T11:17:57Z

query/src/storages/memory/memory_table.rs

-        ctx: Arc<QueryContext>,
-        operations: Vec<DataBlock>,
-        overwrite: bool,
+        _ctx: Arc<QueryContext>,


rename these

dantengsky · 2022-05-10T11:18:32Z

query/src/storages/system/columns_table.rs

@@ -96,7 +95,8 @@ impl ColumnsTable {
        ctx: Arc<QueryContext>,
    ) -> Result<Vec<(String, String, DataField)>> {
        let tenant = ctx.get_tenant();
-        let catalog = ctx.get_catalog();
+        // TODO replace default with real cat
+        let catalog = ctx.get_catalog("default")?;


use constant DEFAULT_CATALOG

dantengsky · 2022-05-10T11:18:58Z

query/src/storages/system/engines_table.rs

-        let table_engine_descriptors = ctx.get_catalog().get_table_engines();
+    async fn get_full_data(&self, ctx: Arc<QueryContext>) -> Result<DataBlock> {
+        // TODO passin catalog name
+        let table_engine_descriptors = ctx.get_catalog("default")?.get_table_engines();


use the CONSTANT

dantengsky · 2022-05-10T11:19:11Z

query/src/storages/system/tables_table.rs

@@ -41,7 +40,8 @@ impl AsyncSystemTable for TablesTable {

    async fn get_full_data(&self, ctx: Arc<QueryContext>) -> Result<DataBlock> {
        let tenant = ctx.get_tenant();
-        let catalog = ctx.get_catalog();
+        // TODO pass catalog in or embed catalog in table info?
+        let catalog = ctx.get_catalog("default")?;


use the constant

dantengsky · 2022-05-10T11:20:04Z

query/tests/it/sql/statements/query/query_qualified_rewriter.rs

-            query: "SELECT COUNT(system.databases.name) AS name FROM system.databases WHERE system.databases.name = 'xxx'",
-            expect: "NormalQuery { filter: (name = xxx), aggregate: [COUNT(name)], projection: [COUNT(name) as name] }",
-        },
+//  TODO confirm this


what's going on herer?

dantengsky · 2022-05-10T11:20:31Z

query/tests/it/storages/fuse/pruning.rs

@@ -92,7 +93,7 @@ async fn test_block_pruner() -> Result<()> {
    interpreter.execute(None).await?;

    // get table
-    let catalog = ctx.get_catalog();
+    let catalog = ctx.get_catalog("default")?;


use constant

dantengsky · 2022-05-10T11:20:41Z

query/tests/it/storages/fuse/pruning.rs

@@ -229,7 +231,7 @@ async fn test_block_pruner_monotonic() -> Result<()> {
        order_keys: vec![],
    };

-    let catalog = ctx.get_catalog();
+    let catalog = ctx.get_catalog("default")?;


use constant

dantengsky · 2022-05-10T11:21:02Z

query/tests/it/storages/fuse/table_test_fixture.rs

@@ -100,6 +102,10 @@ impl TestFixture {
        gen_db_name(&self.prefix)
    }

+    pub fn default_catalog_name(&self) -> String {
+        "default".to_owned()


use constant

dantengsky · 2022-05-10T11:22:15Z

tests/suites/0_stateless/05_ddl/05_0006_ddl_grant_privilege.result

-GRANT SELECT ON 'db01'.'tb1' TO 'test-grant'@'localhost'
-GRANT SELECT ON 'db01'.'tb1' TO 'test-grant'@'localhost'
-GRANT SELECT ON 'default'.* TO 'test-grant-role'
+GRANT ALL ON 'default'.'default'.* TO 'test-grant'@'localhost'


append docs about privilege in the PR summary

BohuTANG · 2022-05-10T11:28:43Z

@dantengsky

LGTM.
I have seen your comments, I suggest refining them in another PR.
I will merge this monster PR first.

Xuanwo · 2022-05-10T11:23:34Z

common/hive-meta-store/build.rs

+        }
+    }
+
+    println!("cargo:rerun-if-changed=build.rs");


Do we need to move this line to the front?

Not sure about this : (
will moving this line to the front prevent us from some unexpected results? I am not quite familiar with the "mechanism" there.

Me too...

But most build.rs will place the println at the very first line. It's better to keep fit with them.

got it & thanks. I will fix it in PR #5284

BohuTANG · 2022-05-10T11:30:50Z

The review also welcomed on this PR cc @andylokandy @Xuanwo

refine: of PR #4947 multi-catalog

databend-bot added the need-review label Apr 19, 2022

mergify bot added pr-feature this PR introduces a new feature to the codebase pr-not-for-changelog labels Apr 19, 2022

dantengsky force-pushed the feat-multi-catalog branch 2 times, most recently from 9830127 to 0d119f1 Compare April 21, 2022 17:17

dantengsky added 3 commits April 22, 2022 10:29

feat: multi-catalog & simple hive meta cat

0f20bda

Merge remote-tracking branch 'origin/main' into multi-catalog-working-on

85b903d

resolve merge conflicts & fix UTs

2a044ad

dantengsky force-pushed the feat-multi-catalog branch from 0d119f1 to 2a044ad Compare April 22, 2022 05:43

dantengsky added 3 commits April 22, 2022 13:48

Merge remote-tracking branch 'origin/main' into feat-multi-catalog

2fa709d

fix: resolve merge conflict

e9894e5

Merge remote-tracking branch 'origin/main' into feat-multi-catalog

09b199c

dantengsky force-pushed the feat-multi-catalog branch from 5775c24 to 09b199c Compare April 22, 2022 13:06

dantengsky added 8 commits April 22, 2022 22:29

fix: stateless tests

1768655

Merge remote-tracking branch 'origin/main' into feat-multi-catalog

ca522ca

merge & resolve conflict with upstream/main

74e06a3

Merge remote-tracking branch 'origin/main' into feat-multi-catalog

3a50f62

fix: unit tests

dd8afd3

Merge remote-tracking branch 'origin/main' into feat-multi-catalog

f2010c7

hive catalog config

9ee5ed4

feat: use thrift to generate hms client stub code

a1cf11e

dantengsky force-pushed the feat-multi-catalog branch from 0311467 to a1cf11e Compare April 28, 2022 05:37

dantengsky added 5 commits April 29, 2022 20:34

Remove redundant files

1f62bed

Merge remote-tracking branch 'origin/main' into feat-multi-catalog

b4a4ecf

refine hive config and converters

b12e7ca

fix : unit tests

4959710

Merge remote-tracking branch 'origin/main' into feat-multi-catalog

1c50432

dantengsky force-pushed the feat-multi-catalog branch from 6f703cc to 5b9aaac Compare May 10, 2022 04:30

dantengsky added 2 commits May 10, 2022 14:49

refine hive statndalone stateful test

5f7d95a

Merge remote-tracking branch 'origin/main' into feat-multi-catalog

337f4ba

dantengsky force-pushed the feat-multi-catalog branch from f37c5bc to 337f4ba Compare May 10, 2022 07:19

dantengsky added 3 commits May 10, 2022 15:43

wait on listening port inside container

5bac618

wait on listening port inside container

0129d0b

refine wait-on

c72186d

dantengsky force-pushed the feat-multi-catalog branch from ed4098b to c72186d Compare May 10, 2022 08:10

dantengsky added 3 commits May 10, 2022 16:18

enable hive stateful tests

ea61b7d

restore normal developing workflow

91737c4

remove OPT_KEY_CATALOG

bb563a4

dantengsky changed the title ~~[WIP] Feature: multi-catalog~~ Feature: multi-catalog May 10, 2022

dantengsky marked this pull request as ready for review May 10, 2022 10:51

dantengsky requested a review from Xuanwo May 10, 2022 11:00

dantengsky commented May 10, 2022

View reviewed changes

BohuTANG merged commit 47af3e1 into databendlabs:main May 10, 2022

Xuanwo reviewed May 10, 2022

View reviewed changes

BohuTANG requested a review from andylokandy May 10, 2022 11:31

BohuTANG added a commit that referenced this pull request May 10, 2022

Merge pull request #5284 from dantengsky/refine-multi-cats

fab3ee5

refine: of PR #4947 multi-catalog

dantengsky mentioned this pull request May 22, 2022

Hive as an External Data Source #4826

Closed

4 tasks

FANNG1 mentioned this pull request Jun 22, 2022

improve the support for hive query #6123

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: multi-catalog #4947

Feature: multi-catalog #4947

dantengsky commented Apr 19, 2022 •

edited

Loading

vercel bot commented Apr 19, 2022 •

edited

Loading

mergify bot commented Apr 19, 2022

dantengsky commented Apr 21, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

dantengsky May 10, 2022

BohuTANG commented May 10, 2022

Xuanwo May 10, 2022

dantengsky May 10, 2022

Xuanwo May 10, 2022

dantengsky May 10, 2022

BohuTANG commented May 10, 2022

Feature: multi-catalog #4947

Feature: multi-catalog #4947

Conversation

dantengsky commented Apr 19, 2022 • edited Loading

Summary

Changelog

Related Issues

vercel bot commented Apr 19, 2022 • edited Loading

mergify bot commented Apr 19, 2022

dantengsky commented Apr 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BohuTANG commented May 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BohuTANG commented May 10, 2022

dantengsky commented Apr 19, 2022 •

edited

Loading

vercel bot commented Apr 19, 2022 •

edited

Loading