[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

vmmusings · 2022-11-08T22:41:01Z

Signed-off-by: vamsi-amazon [email protected]
(cherry picked from commit 3e30379)

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit 3e30379)

joshuali925

should 2.4 be merged into 2.x?

vmmusings · 2022-11-08T22:50:55Z

should 2.4 be merged into 2.x?

Yes, But I am not sure if there are any pending PRs which are not merged into 2.x

vmmusings · 2022-11-08T22:51:12Z

should 2.4 be merged into 2.x?

Yes, But I am not sure if there are any pending PRs which are not merged into 2.x

This is the only one I could find.

codecov-commenter · 2022-11-08T22:58:07Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.80%. Comparing base (1a52511) to head (8a05fe7).
Report is 467 commits behind head on 2.x.

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x    #1049      +/-   ##
============================================
- Coverage     98.21%   95.80%   -2.42%     
- Complexity     3268     3410     +142     
============================================
  Files           314      336      +22     
  Lines          8197     9326    +1129     
  Branches        533      700     +167     
============================================
+ Hits           8051     8935     +884     
- Misses          142      334     +192     
- Partials          4       57      +53

Flag	Coverage Δ
query-workbench	`62.76% <ø> (?)`
sql-engine	`98.31% <100.00%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

joshuali925 · 2022-11-08T22:59:46Z

ok i checked the other missing one is #1050, should be good

YANG-DB · 2022-11-08T23:18:25Z

docs/user/general/identifiers.rst

@@ -231,7 +231,7 @@ tableName = ``http_requests_total``.
 2. ``logs.12.13.1``


-catalogName = ``@opensearch`` [Resolves to default @opensearch connector since [my_prometheus] is the only catalog configured name.]
+datasourceName = ``@opensearch`` [Resolves to default @opensearch connector since [my_prometheus] is the only catalog configured name.]


missed one here ...

This is a backport PR can't do much. This is a good catch will fix it in coming PR.
The idea of this PR is to remove only public references[Documentation and show catlalogs command only] as it was done on the last day of release.

All the references in the implementation will be removed in a new PR.

YANG-DB · 2022-11-08T23:18:43Z

docs/user/general/identifiers.rst


 schemaName = ``default`` [No supported schema found, so default to `default`].

 tableName = ``prometheus.http_requests_total``.

 5. ``prometheus.default.http_requests_total.1.2.3``

-catalogName = ``@opensearch`` [Resolves to default @opensearch connector since [my_prometheus] is the only catalog configured name.]
+datasourceName = ``@opensearch`` [Resolves to default @opensearch connector since [my_prometheus] is the only catalog configured name.]


missed one here

YANG-DB · 2022-11-08T23:20:33Z

doctest/build.gradle

@@ -104,7 +104,7 @@ String mlCommonsPlugin = 'opensearch-ml'

 testClusters {
    docTestCluster {
-        keystore 'plugins.query.federation.catalog.config', new File("$projectDir/catalog", 'catalog.json')
+        keystore 'plugins.query.federation.datasources.config', new File("$projectDir/catalog", 'catalog.json')


why not change the json file name also into datasource.json ?

YANG-DB · 2022-11-08T23:21:04Z

integ-test/src/test/java/org/opensearch/sql/ppl/ShowCatalogsCommandIT.java

@@ -20,25 +20,25 @@ public class ShowCatalogsCommandIT extends PPLIntegTestCase {

  @Test
  public void testShowCatalogsCommands() throws IOException {


should be testShowDatasourcesCommands()...

YANG-DB · 2022-11-08T23:21:30Z

integ-test/src/test/java/org/opensearch/sql/ppl/ShowCatalogsCommandIT.java

    verifyDataRows(result,
        rows("my_prometheus", "PROMETHEUS"),
        rows("@opensearch", "OPENSEARCH"));
    verifyColumn(
        result,
-        columnName("CATALOG_NAME"),
+        columnName("DATASOURCE_NAME"),
        columnName("CONNECTOR_TYPE")
    );
  }

  @Test
  public void testShowCatalogsCommandsWithWhereClause() throws IOException {


change test name here also...

YANG-DB · 2022-11-08T23:22:46Z

integ-test/build.gradle

@@ -111,7 +111,7 @@ testClusters.all {

 testClusters.integTest {
    plugin ":opensearch-sql-plugin"
-    keystore 'plugins.query.federation.catalog.config', new File("$projectDir/src/test/resources/catalog/", 'catalog.json')
+    keystore 'plugins.query.federation.datasources.config', new File("$projectDir/src/test/resources/catalog/", 'catalog.json')


can the catalog.json also be changed into datasource.json ?

* Fix `FLOAT` -> `DOUBLE` cast. (#1025) Signed-off-by: Yury-Fridlyand <[email protected]> * Fix error messaging from prometheus. (#1029) (#1037) Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit 4a9cef3) Co-authored-by: vamsi-amazon <[email protected]> * Add `query` function as alternate syntax to `query_string` function (#1010) This maintains backwards compatibility with the v1 engine. * Update DATE and TIME functions to parse string input as datetime (#991) Add option to accept datetime like string in both TIME and DATE (eg. accept "1999-01-02 12:12:12" for both TIME and DATE. Strict check on date for testing for valid dates (eg. Don't accept Feb 30th as a valid date) and throws a SemanticCheckException. Co-authored-by: Yury-Fridlyand <[email protected]> Signed-off-by: MitchellGale-BitQuill <[email protected]> * back quote fix (#1041) (#1050) Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit d3bb902) Co-authored-by: vamsi-amazon <[email protected]> * Catalog to Datasource changes (#1027) (#1049) Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit 3e30379) * Bump jackson to 2.14.0 (#1058) Signed-off-by: Joshua Li <[email protected]> (cherry picked from commit 5a1adb2) * Add valueOf() to Expression (#1055) Signed-off-by: Joshua Li <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: MitchellGale-BitQuill <[email protected]> Signed-off-by: Joshua Li <[email protected]> Signed-off-by: Chen Dai <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: vamsi-amazon <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MitchellGale-BitQuill <[email protected]> Co-authored-by: Joshua Li <[email protected]>

Catalog to Datasource changes (#1027)

8a05fe7

Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit 3e30379)

vmmusings requested a review from a team as a code owner November 8, 2022 22:41

vmmusings changed the title ~~Catalog to Datasource changes (#1027)~~ [Backport to 2.x]Catalog to Datasource changes (#1027) Nov 8, 2022

joshuali925 approved these changes Nov 8, 2022

View reviewed changes

ps48 approved these changes Nov 8, 2022

View reviewed changes

vmmusings merged commit c2ccc91 into 2.x Nov 8, 2022

github-actions bot deleted the backport/backport-1027-to-2.x branch November 8, 2022 23:23

YANG-DB reviewed Nov 8, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

vmmusings commented Nov 8, 2022

joshuali925 left a comment

vmmusings commented Nov 8, 2022

vmmusings commented Nov 8, 2022

codecov-commenter commented Nov 8, 2022 •

edited by codecov bot

Loading

joshuali925 commented Nov 8, 2022

YANG-DB Nov 8, 2022

vmmusings Nov 8, 2022

YANG-DB Nov 8, 2022

YANG-DB Nov 8, 2022

YANG-DB Nov 8, 2022

YANG-DB Nov 8, 2022

YANG-DB Nov 8, 2022

		@@ -20,25 +20,25 @@ public class ShowCatalogsCommandIT extends PPLIntegTestCase {

		@Test
		public void testShowCatalogsCommands() throws IOException {

[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

[Backport to 2.x]Catalog to Datasource changes (#1027) #1049

Conversation

vmmusings commented Nov 8, 2022

Description

Issues Resolved

Check List

joshuali925 left a comment

Choose a reason for hiding this comment

vmmusings commented Nov 8, 2022

vmmusings commented Nov 8, 2022

codecov-commenter commented Nov 8, 2022 • edited by codecov bot Loading

Codecov Report

joshuali925 commented Nov 8, 2022

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

vmmusings Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

YANG-DB Nov 8, 2022

Choose a reason for hiding this comment

codecov-commenter commented Nov 8, 2022 •

edited by codecov bot

Loading