Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: multi-catalog #4947

Merged
merged 56 commits into from
May 10, 2022
Merged

Feature: multi-catalog #4947

Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
0f20bda
feat: multi-catalog & simple hive meta cat
dantengsky Apr 18, 2022
85b903d
Merge remote-tracking branch 'origin/main' into multi-catalog-working-on
dantengsky Apr 22, 2022
2a044ad
resolve merge conflicts & fix UTs
dantengsky Apr 22, 2022
2fa709d
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 22, 2022
e9894e5
fix: resolve merge conflict
dantengsky Apr 22, 2022
09b199c
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 22, 2022
1768655
fix: stateless tests
dantengsky Apr 22, 2022
ca522ca
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 23, 2022
74e06a3
merge & resolve conflict with upstream/main
dantengsky Apr 23, 2022
3a50f62
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 25, 2022
dd8afd3
fix: unit tests
dantengsky Apr 25, 2022
f2010c7
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 27, 2022
9ee5ed4
hive catalog config
dantengsky Apr 27, 2022
a1cf11e
feat: use thrift to generate hms client stub code
dantengsky Apr 27, 2022
1f62bed
Remove redundant files
dantengsky Apr 29, 2022
b4a4ecf
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 29, 2022
b12e7ca
refine hive config and converters
dantengsky Apr 29, 2022
4959710
fix : unit tests
dantengsky Apr 29, 2022
1c50432
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 29, 2022
727ff11
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 29, 2022
6572181
fix: add hms.thrift
dantengsky Apr 29, 2022
78de617
fix: add licenses header to hsm.thrift
dantengsky Apr 29, 2022
042a4a0
fix: merge conflicts
dantengsky Apr 29, 2022
50229ae
fix: remove unused dependencies from crate hive-meta-store
dantengsky Apr 29, 2022
b4fe55e
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky Apr 29, 2022
9fd4136
fix: brew install for "build_maxos" action
dantengsky Apr 30, 2022
2446397
fix: yamllint github action yml files
dantengsky Apr 30, 2022
d8df88b
refact: rm duplicated code (resolve_database/table)
dantengsky Apr 30, 2022
0098e95
fix: unit tests
dantengsky May 1, 2022
bc19e06
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 1, 2022
64a6947
fix: pass catalog to Sink
dantengsky May 1, 2022
fb5a3c5
attach catalog name to table option
dantengsky May 1, 2022
184bb81
remove unused mod
dantengsky May 2, 2022
f4b1a66
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 2, 2022
a2e2e42
tune stateless test case
dantengsky May 2, 2022
70a8c7c
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 5, 2022
7c25191
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 6, 2022
70b4ce2
Introduce cargo feature "hive"
dantengsky May 6, 2022
ec77751
fix unit test
dantengsky May 7, 2022
12c75a3
basic ut for hive-meta-store
dantengsky May 7, 2022
f21eacf
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 9, 2022
61acdbb
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 9, 2022
c9d4dde
use ctx catalog in fuse_snapshot/segment procedure
dantengsky May 9, 2022
1c28bf7
simple CI setup for hive
dantengsky May 9, 2022
78856ad
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 9, 2022
23b61fc
refine stateful hive action
dantengsky May 9, 2022
e4780a4
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 9, 2022
4b010a3
resovle compile err brought by non-conflict merge
dantengsky May 9, 2022
5f7d95a
refine hive statndalone stateful test
dantengsky May 9, 2022
337f4ba
Merge remote-tracking branch 'origin/main' into feat-multi-catalog
dantengsky May 10, 2022
5bac618
wait on listening port inside container
dantengsky May 10, 2022
0129d0b
wait on listening port inside container
dantengsky May 10, 2022
c72186d
refine wait-on
dantengsky May 10, 2022
ea61b7d
enable hive stateful tests
dantengsky May 10, 2022
91737c4
restore normal developing workflow
dantengsky May 10, 2022
bb563a4
remove OPT_KEY_CATALOG
dantengsky May 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/actions/build_debug/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ runs:
if: "contains(inputs.target, 'apple')"
run: rustup target add ${{ inputs.target }}

- shell: bash
if: "contains(inputs.target, 'apple')"
run: brew install thrift

- name: Setup Build Tool
if: "contains(inputs.target, 'linux')"
uses: ./.github/actions/setup_build_tool
Expand Down
4 changes: 4 additions & 0 deletions .github/actions/build_release/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ runs:
if: "contains(inputs.target, 'apple')"
run: rustup target add ${{ inputs.target }}

- shell: bash
if: "contains(inputs.target, 'apple')"
run: brew install thrift

- name: Setup Build Tool
if: "contains(inputs.target, 'linux')"
uses: ./.github/actions/setup_build_tool
Expand Down
9 changes: 9 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ members = [
"common/codegen",
"common/tracing",
"common/metrics",
"common/hive-meta-store",

# Query
"query",
Expand Down
8 changes: 6 additions & 2 deletions common/ast/src/ast/query.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ pub enum Indirection {
pub enum TableReference {
// Table name
Table {
// Could be `db.table` or `table`
catalog: Option<Identifier>,
database: Option<Identifier>,
table: Identifier,
alias: Option<TableAlias>,
Expand Down Expand Up @@ -204,11 +204,15 @@ impl Display for TableReference {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
match self {
TableReference::Table {
catalog,
database,
table,
alias,
} => {
write_period_separated_list(f, database.iter().chain(Some(table)))?;
write_period_separated_list(
f,
catalog.iter().chain(database.iter()).chain(Some(table)),
)?;
if let Some(alias) = alias {
write!(f, " AS {alias}")?;
}
Expand Down
15 changes: 10 additions & 5 deletions common/ast/src/parser/query.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,15 +126,20 @@ pub fn table_reference(i: Input) -> IResult<TableReference> {
pub fn aliased_table(i: Input) -> IResult<TableReference> {
map(
rule! {
#ident ~ ( "." ~ #ident )? ~ #table_alias?
#ident ~ ( "." ~ #ident )? ~ ( "." ~ #ident )? ~ #table_alias?
},
|(fst, snd, alias)| {
let (database, table) = match (fst, snd) {
(database, Some((_, table))) => (Some(database), table),
(table, None) => (None, table),
|(fst, snd, third, alias)| {
let (catalog, database, table) = match (fst, snd, third) {
(catalog, Some((_, database)), Some((_, table))) => {
(Some(catalog), Some(database), table)
}
(database, Some((_, table)), None) => (None, Some(database), table),
(database, None, Some((_, table))) => (None, Some(database), table),
(table, None, None) => (None, None, table),
};

TableReference::Table {
catalog,
database,
table,
alias,
Expand Down
3 changes: 3 additions & 0 deletions common/ast/tests/it/testdata/query-error.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Query {
],
from: Some(
Table {
catalog: None,
database: None,
table: Identifier {
name: "customer",
Expand Down Expand Up @@ -71,6 +72,7 @@ Query {
op: Inner,
condition: Natural,
left: Table {
catalog: None,
database: None,
table: Identifier {
name: "customer",
Expand All @@ -79,6 +81,7 @@ Query {
alias: None,
},
right: Table {
catalog: None,
database: None,
table: Identifier {
name: "orders",
Expand Down
13 changes: 13 additions & 0 deletions common/ast/tests/it/testdata/query.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Query {
},
),
left: Table {
catalog: None,
database: None,
table: Identifier {
name: "customer",
Expand All @@ -48,6 +49,7 @@ Query {
alias: None,
},
right: Table {
catalog: None,
database: None,
table: Identifier {
name: "orders",
Expand Down Expand Up @@ -117,6 +119,7 @@ Query {
},
),
left: Table {
catalog: None,
database: None,
table: Identifier {
name: "customer",
Expand All @@ -125,6 +128,7 @@ Query {
alias: None,
},
right: Table {
catalog: None,
database: None,
table: Identifier {
name: "orders",
Expand Down Expand Up @@ -180,6 +184,7 @@ Query {
op: FullOuter,
condition: Natural,
left: Table {
catalog: None,
database: None,
table: Identifier {
name: "customer",
Expand All @@ -188,6 +193,7 @@ Query {
alias: None,
},
right: Table {
catalog: None,
database: None,
table: Identifier {
name: "orders",
Expand Down Expand Up @@ -242,6 +248,7 @@ Query {
op: Inner,
condition: Natural,
left: Table {
catalog: None,
database: None,
table: Identifier {
name: "customer",
Expand All @@ -250,6 +257,7 @@ Query {
alias: None,
},
right: Table {
catalog: None,
database: None,
table: Identifier {
name: "orders",
Expand All @@ -260,6 +268,7 @@ Query {
},
),
right: Table {
catalog: None,
database: None,
table: Identifier {
name: "detail",
Expand Down Expand Up @@ -363,6 +372,7 @@ Query {
op: CrossJoin,
condition: None,
left: Table {
catalog: None,
database: None,
table: Identifier {
name: "customer",
Expand All @@ -371,6 +381,7 @@ Query {
alias: None,
},
right: Table {
catalog: None,
database: None,
table: Identifier {
name: "orders",
Expand Down Expand Up @@ -472,6 +483,7 @@ Query {
},
),
left: Table {
catalog: None,
database: None,
table: Identifier {
name: "customer",
Expand All @@ -480,6 +492,7 @@ Query {
alias: None,
},
right: Table {
catalog: None,
database: None,
table: Identifier {
name: "orders",
Expand Down
Loading