Support table identifier contains dot with backticks #768

LantaoJin · 2024-10-11T06:17:42Z

Description

Table name `_CWLBasic`.default.`Canary/service.log` is incorrectly converted to default + Canary/service + log.

It should be converted to _CWLBasic + default + Canary/service.log as long as the table identifier is wrapped by backticks.

Issues Resolved

#767

Check List

Updated documentation (ppl-spark-integration/README.md)
Implemented unit tests
Implemented tests for combination with other commands
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin · 2024-10-11T07:35:37Z

...est/scala/org/opensearch/flint/spark/ppl/PPLLogicalPlanBasicQueriesTranslatorTestSuite.scala

  test("test describe FQN table clause") {
    val context = new CatalystPlanContext
    val logPlan =
-      planTransformer.visit(plan(pplParser, "describe schema.default.http_logs"), context)
+      planTransformer.visit(plan(pplParser, "describe catalog.schema.http_logs"), context)


This PR also fixes a bug of describe command.
The pattern catalog.schema.table will be converted to
TableIdentifier(table, Some(schema), Some(catalog))
Without patch, it was TableIdentifier(table, Some(catalog), Some(schema))

This "describe" bug introduced in https://github.com/opensearch-project/opensearch-spark/pull/768/files#diff-fdf005d13b7478348f948571a114e747641004ac34b11a3ab53425d26559cde5L163-L164

@LantaoJin nice catch - thanks

LantaoJin · 2024-10-11T11:38:32Z

ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystQueryPlanVisitor.java

+        node.getQualifiedNames().forEach(q ->
                // Resolving the qualifiedName which is composed of a datasource.schema.table
-                context.withRelation(new UnresolvedRelation(seq(of(t.split("\\."))), CaseInsensitiveStringMap.empty(), false))
+                context.withRelation(new UnresolvedRelation(getTableIdentifier(q).nameParts(), CaseInsensitiveStringMap.empty(), false))


These are the key lines to fix the bug of "dot in backticks"

Replace simple string manipulation with building a Spark TableIdentifier. This fixing not only could identify a valid pattern catalog.schema.`table.name`, but also could identify an invalid pattern catalog.schema.table.name (more than 3 parts and without backticks)

YANG-DB

@LantaoJin thanks for the quick fix !!

opensearch-trigger-bot · 2024-10-11T17:44:37Z

The backport to 0.5 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/opensearch-spark/backport-0.5 0.5
# Navigate to the new working tree
pushd ../.worktrees/opensearch-spark/backport-0.5
# Create a new branch
git switch --create backport/backport-768-to-0.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fe5148cb4968917e532a1ddd745a80d4f58e576b
# Push it to GitHub
git push --set-upstream origin backport/backport-768-to-0.5
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/opensearch-spark/backport-0.5

Then, create a pull request where the base branch is 0.5 and the compare/head branch is backport/backport-768-to-0.5.

opensearch-trigger-bot · 2024-10-11T17:44:38Z

The backport to 0.5-nexus failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/opensearch-spark/backport-0.5-nexus 0.5-nexus
# Navigate to the new working tree
pushd ../.worktrees/opensearch-spark/backport-0.5-nexus
# Create a new branch
git switch --create backport/backport-768-to-0.5-nexus
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fe5148cb4968917e532a1ddd745a80d4f58e576b
# Push it to GitHub
git push --set-upstream origin backport/backport-768-to-0.5-nexus
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/opensearch-spark/backport-0.5-nexus

Then, create a pull request where the base branch is 0.5-nexus and the compare/head branch is backport/backport-768-to-0.5-nexus.

seankao-az · 2024-10-11T21:05:33Z

Backport to 0.5-nexus failed. Manually backporting

seankao-az · 2024-10-11T21:12:23Z

Found some other missing commits due to backport failure of other PR, e.g. #743

opensearch-trigger-bot · 2024-10-11T21:16:02Z

The backport to 0.5 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/opensearch-spark/backport-0.5 0.5
# Navigate to the new working tree
pushd ../.worktrees/opensearch-spark/backport-0.5
# Create a new branch
git switch --create backport/backport-768-to-0.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fe5148cb4968917e532a1ddd745a80d4f58e576b
# Push it to GitHub
git push --set-upstream origin backport/backport-768-to-0.5
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/opensearch-spark/backport-0.5

Then, create a pull request where the base branch is 0.5 and the compare/head branch is backport/backport-768-to-0.5.

Signed-off-by: YANGDB <[email protected]>

…icks (#774) * backport #768 to 0.5-nexus Signed-off-by: YANGDB <[email protected]> * support spark prior to 3.5 with its extended table identifier (existing table identifier only has 2 parts) Signed-off-by: YANGDB <[email protected]> --------- Signed-off-by: YANGDB <[email protected]>

Support table identifier contains dot with backticks

6d62567

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin added the 0.6 label Oct 11, 2024

LantaoJin added 2 commits October 11, 2024 15:21

also fix the bug of describe

ee80aaf

Signed-off-by: Lantao Jin <[email protected]>

add doc

3de578d

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin commented Oct 11, 2024

View reviewed changes

LantaoJin marked this pull request as ready for review October 11, 2024 07:38

LantaoJin requested review from dai-chen, rupal-bq, vamsimanohar, penghuo, seankao-az, anirudha, kaituo and YANG-DB as code owners October 11, 2024 07:38

LantaoJin commented Oct 11, 2024

View reviewed changes

YANG-DB approved these changes Oct 11, 2024

View reviewed changes

YANG-DB merged commit fe5148c into opensearch-project:main Oct 11, 2024
4 checks passed

seankao-az added backport 0.5 backport 0.5-nexus labels Oct 11, 2024

seankao-az added Lang:PPL Pipe Processing Language support and removed backport 0.5 labels Oct 11, 2024

seankao-az added the backport 0.5 label Oct 11, 2024

YANG-DB added a commit to YANG-DB/opensearch-spark that referenced this pull request Oct 11, 2024

backport opensearch-project#768 to 0.5-nexus

b6c4b0f

Signed-off-by: YANGDB <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support table identifier contains dot with backticks #768

Support table identifier contains dot with backticks #768

LantaoJin commented Oct 11, 2024 •

edited

Loading

LantaoJin Oct 11, 2024

LantaoJin Oct 11, 2024 •

edited

Loading

YANG-DB Oct 11, 2024

LantaoJin Oct 11, 2024 •

edited

Loading

LantaoJin Oct 11, 2024 •

edited

Loading

YANG-DB left a comment

opensearch-trigger-bot bot commented Oct 11, 2024

opensearch-trigger-bot bot commented Oct 11, 2024

seankao-az commented Oct 11, 2024

seankao-az commented Oct 11, 2024

opensearch-trigger-bot bot commented Oct 11, 2024

Support table identifier contains dot with backticks #768

Support table identifier contains dot with backticks #768

Conversation

LantaoJin commented Oct 11, 2024 • edited Loading

Description

Issues Resolved

Check List

LantaoJin Oct 11, 2024

Choose a reason for hiding this comment

LantaoJin Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

YANG-DB Oct 11, 2024

Choose a reason for hiding this comment

LantaoJin Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

LantaoJin Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

YANG-DB left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Oct 11, 2024

opensearch-trigger-bot bot commented Oct 11, 2024

seankao-az commented Oct 11, 2024

seankao-az commented Oct 11, 2024

opensearch-trigger-bot bot commented Oct 11, 2024

LantaoJin commented Oct 11, 2024 •

edited

Loading

LantaoJin Oct 11, 2024 •

edited

Loading

LantaoJin Oct 11, 2024 •

edited

Loading

LantaoJin Oct 11, 2024 •

edited

Loading