Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adapter,interchange,parser: Support avro sink comments #22019

Merged
merged 5 commits into from
Oct 10, 2023
Merged

adapter,interchange,parser: Support avro sink comments #22019

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
87eb196
Implementing avro doc comments
Sep 27, 2023
71488b2
Rebased from main to pull in changes for new ColumnName type
Oct 6, 2023
1d48efc
Add tests for avro sink comments
def- Oct 6, 2023
388e025
Fixing debezium case
Oct 7, 2023
e393529
Putting the doc on option behind feature flag
Oct 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions doc/developer/design/20230903_avro_doc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,8 @@ to the value schema; a `DOC ON ...` option applies to both types of schemas.
Each `DOC ON` option applies to either a column or a type:

* Column comments (`[KEY|VALUE] DOC ON COLUMN ...`) name a column in the
format `[[[db.]schema.]object.]column`. If the object name is omitted, the
column name is taken to be a name of a column in the top level of the
materialized view. Object names are looked up according to usual SQL name
resolution rules for the search path and active database.
format `[[db.]schema.]object.column`. Object names are looked up according to usual
Copy link
Contributor Author

@moulimukherjee moulimukherjee Oct 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the implementatoin to keep it consistent with the naming resolution for the type below [[db.]schema.]object and with the COMMENT ON sql which expects the object to be specified. Also, sinks can't be queried and they don't have a relation desc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏽

SQL name resolution rules for the search path and active database.

* Type comments (`[KEY|VALUE] DOC ON TYPE ...`) specify a type in the
format `[[db.]schema.]object`. Object names are looked up according to usual
Expand Down
26 changes: 25 additions & 1 deletion misc/python/materialize/checks/identifiers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ def _can_run(self, e: Executor) -> bool:
"sink2": "17",
"alias": "18",
"role": "19",
"comment_table": "20",
"comment_column": "21",
},
{
"db": "-1.0",
Expand All @@ -95,6 +97,8 @@ def _can_run(self, e: Executor) -> bool:
"sink2": "17.0",
"alias": "18.0",
"role": "19.0",
"comment_table": "20.0",
"comment_column": "21.0",
},
{
"db": "\u0001\u0002\u0003\u0004\u0005\u0006\u0007\b\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015\u0016\u0017\u0018\u0019\u001a\u001b\u001c\u001d\u001e\u001f^?",
Expand All @@ -118,6 +122,8 @@ def _can_run(self, e: Executor) -> bool:
"sink2": "",
"alias": "₀₁₂",
"role": "⁰⁴⁵₀₁₂",
"comment_table": "ด้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็ ด้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็ ด้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็็้้้้้้้้็็็็็้้้้้็็็็",
"comment_column": "٠١٢٣٤٥٦٧٨٩",
},
{
"db": "찦차를 타고 온 펲시맨과 쑛다리 똠방각하",
Expand All @@ -141,6 +147,8 @@ def _can_run(self, e: Executor) -> bool:
"sink2": "｀ｨ(´∀｀∩",
"alias": "⅛⅜⅝⅞",
"role": "ЁЂЃЄЅІЇЈЉЊЋЌЍЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюя",
"comment_table": "`⁄€‹›fifl‡°·‚—±",
"comment_column": "Œ„´‰ˇÁ¨ˆØ∏”’",
},
{
"db": "❤️ 💔 💌 💕 💞 💓 💗 💖 💘 💝 💟 💜 💛 💚 💙",
Expand All @@ -164,6 +172,8 @@ def _can_run(self, e: Executor) -> bool:
"sink2": "‫test‫",
"alias": "1#INF",
"role": "0xffffffffffffffff",
"comment_table": "0xabad1dea",
"comment_column": "123456789012345678901234567890123456789",
},
{
"db": "ﺍﻺﻃﻼﻗ ﻊﻟ ﺈﻳﻭ.",
Expand All @@ -186,7 +196,9 @@ def _can_run(self, e: Executor) -> bool:
"sink1": "⁦test⁧",
"sink2": "‪‪᚛                 ᚜‪",
"alias": "0xabad1dea",
"role": "0xffffffffffffffff",
"role": "1.000.000,00",
"comment_table": "2.2250738585072011e-308",
"comment_column": "09",
},
]

Expand Down Expand Up @@ -230,6 +242,12 @@ def initialize(self) -> Testdrive:
cmds += f"""
> CREATE SECRET {dq(self.ident["secret"])} as {sq(self.ident["secret_value"])};
"""
if self.base_version >= MzVersion(0, 72, 0):
cmds += f"""
> COMMENT ON TABLE {dq(self.ident["schema"])}.{dq(self.ident["table"])} IS {sq(self.ident["comment_table"])};

> COMMENT ON COLUMN {dq(self.ident["schema"])}.{dq(self.ident["table"])}.{dq(self.ident["column"])} IS {sq(self.ident["comment_column"])};
"""

return Testdrive(schemas() + cluster() + dedent(cmds))

Expand Down Expand Up @@ -293,6 +311,12 @@ def validate(self) -> Testdrive:
> SELECT * FROM {dq(self.ident["source_view"])};
U2 A 1000
"""
if self.base_version >= MzVersion(0, 72, 0):
cmds += f"""
> SELECT object_sub_id, comment FROM mz_internal.mz_comments JOIN mz_tables ON mz_internal.mz_comments.id = mz_tables.id WHERE name = {sq(self.ident["table"])};
<null> {dq_print(self.ident["comment_table"])}
1 {dq_print(self.ident["comment_column"])}
"""
if self.base_version >= MzVersion(0, 44, 0):
cmds += f"""
> SHOW SECRETS;
Expand Down
Loading