Add spark connector #1780

rupal-bq · 2023-06-26T16:52:09Z

Description

Setup spark connector module

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Vamsi Manohar <[email protected]>

Signed-off-by: Rupal Mahajan <[email protected]>

codecov · 2023-06-26T17:17:00Z

Codecov Report

Merging #1780 (be15dae) into main (fa840e0) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1780      +/-   ##
============================================
+ Coverage     97.32%   97.33%   +0.01%     
- Complexity     4458     4490      +32     
============================================
  Files           388      394       +6     
  Lines         11050    11118      +68     
  Branches        790      795       +5     
============================================
+ Hits          10754    10822      +68     
  Misses          289      289              
  Partials          7        7

Flag	Coverage Δ
sql-engine	`97.33% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...implementation/SparkSqlFunctionImplementation.java	`100.00% <100.00%> (ø)`
...ctions/resolver/SparkSqlTableFunctionResolver.java	`100.00% <100.00%> (ø)`
...nctions/scan/SparkSqlFunctionTableScanBuilder.java	`100.00% <100.00%> (ø)`
...ensearch/sql/spark/storage/SparkStorageEngine.java	`100.00% <100.00%> (ø)`
...nsearch/sql/spark/storage/SparkStorageFactory.java	`100.00% <100.00%> (ø)`
...a/org/opensearch/sql/spark/storage/SparkTable.java	`100.00% <100.00%> (ø)`

Signed-off-by: Rupal Mahajan <[email protected]>

spark/lombok.config

spark/src/main/java/org/opensearch/sql/spark/client/SparkClient.java

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java

Yury-Fridlyand · 2023-06-27T19:32:19Z

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java

+
+    SparkQueryRequest sparkQueryRequest = new SparkQueryRequest();
+    arguments.forEach(arg -> {
+      String argName = ((NamedArgumentExpression) arg).getArgName();


Is arg always NamedArgumentExpression? How do you guarantee that?
You can do cast in constructor though.

Yury-Fridlyand · 2023-06-27T19:35:05Z

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageEngine.java

+
+  @Override
+  public Table getTable(DataSourceSchemaName dataSourceSchemaName, String tableName) {
+    return null;


Why? Maybe throw?

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageEngine.java

Yury-Fridlyand · 2023-06-27T19:35:20Z

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageFactory.java

+   */
+  StorageEngine getStorageEngine(Map<String, String> requiredConfig) {
+    SparkClient sparkClient = null;
+    //TODO: Initialize spark client


All TODOs in this PR will be resolved in follow up #1790. Trying to split spark connector in 3 parts to avoid large PR.

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageFactory.java

Yury-Fridlyand · 2023-06-27T19:35:37Z

spark/src/test/java/org/opensearch/sql/spark/functions/SqlFunctionImplementationTest.java

+    assertEquals("Invalid Function Argument:tmp", exception.getMessage());
+  }
+
+}


Yury-Fridlyand · 2023-06-27T19:36:12Z

Can you add ITs, docs and doctests?

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java

Yury-Fridlyand · 2023-06-27T20:06:15Z

Is explain supported by spark queries? If not, please, add an IT. If yes, please, add an IT too (and maybe doctest/docs too).

Signed-off-by: Rupal Mahajan <[email protected]>

rupal-bq · 2023-06-28T17:47:54Z

Can you add ITs, docs and doctests?

Yes, I need at least one client implementation for adding proper ITs. So I was planning to add ITs, docs and doctests in follow up #1790

spark/src/main/java/org/opensearch/sql/spark/request/SparkQueryRequest.java

spark/src/main/java/org/opensearch/sql/spark/storage/SparkMetricTable.java

spark/.gitignore

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java

core/src/main/java/org/opensearch/sql/datasource/model/DataSourceType.java

spark/lombok.config

spark/build.gradle

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageFactory.java

spark/src/main/java/org/opensearch/sql/spark/functions/resolver/SqlTableFunctionResolver.java

Signed-off-by: Rupal Mahajan <[email protected]>

penghuo

Thanks for the change. Please add IT in the following PR.

dai-chen

Thanks for the changes!

* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> (cherry picked from commit a816a58)

* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]>

* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> (cherry picked from commit a816a58)

* Add spark connector (#1780) * Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> (cherry picked from commit a816a58) * Upgrade httpclient to 4.5.14 to fix build Signed-off-by: Rupal Mahajan <[email protected]> * Revert Upgrade httpclient to 4.5.14 Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Rupal Mahajan <[email protected]>

* Create Spark Connector Signed-off-by: Vamsi Manohar <[email protected]> * Add spark client and engine Signed-off-by: Rupal Mahajan <[email protected]> * Remove vars Signed-off-by: Rupal Mahajan <[email protected]> * Spark connector draft Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix license header Signed-off-by: Rupal Mahajan <[email protected]> * Add spark storage test Signed-off-by: Rupal Mahajan <[email protected]> * Update comments Signed-off-by: Rupal Mahajan <[email protected]> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <[email protected]> * Update tests Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comments Signed-off-by: Rupal Mahajan <[email protected]> * Refactor class name Signed-off-by: Rupal Mahajan <[email protected]> * Address PR comment Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Rupal Mahajan <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> (cherry picked from commit a816a58) Co-authored-by: Rupal Mahajan <[email protected]>

vmmusings and others added 7 commits June 7, 2023 18:00

Create Spark Connector

70588d6

Signed-off-by: Vamsi Manohar <[email protected]>

Add spark client and engine

2b8ceb7

Signed-off-by: Rupal Mahajan <[email protected]>

Remove vars

f595ff4

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'main' into create-spark-connector

95a86d5

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'main' into create-spark-connector

51700bd

Signed-off-by: Rupal Mahajan <[email protected]>

Spark connector draft

d0d5042

Signed-off-by: Rupal Mahajan <[email protected]>

nit

5ac01b2

Signed-off-by: Rupal Mahajan <[email protected]>

rupal-bq added 8 commits June 26, 2023 11:13

Fix checkstyle errors

4fcf4ba

Signed-off-by: Rupal Mahajan <[email protected]>

nit

52a044d

Signed-off-by: Rupal Mahajan <[email protected]>

Fix license header

69ca116

Signed-off-by: Rupal Mahajan <[email protected]>

Add spark storage test

901e762

Signed-off-by: Rupal Mahajan <[email protected]>

Update comments

1b81453

Signed-off-by: Rupal Mahajan <[email protected]>

Fix checkstyle in comments

106218e

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'main' into create-spark-connector

6da93f3

Signed-off-by: Rupal Mahajan <[email protected]>

Update tests

b953aa6

Signed-off-by: Rupal Mahajan <[email protected]>

rupal-bq marked this pull request as ready for review June 26, 2023 23:15

rupal-bq requested review from pjfitzgibbons, ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, vmmusings, Swiddis, penghuo, seankao-az and MaxKsyunz as code owners June 26, 2023 23:15

Yury-Fridlyand reviewed Jun 27, 2023

View reviewed changes

matthewryanwells reviewed Jun 27, 2023

View reviewed changes

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java Outdated Show resolved Hide resolved

Address PR comments

0df37a8

Signed-off-by: Rupal Mahajan <[email protected]>

dai-chen reviewed Jun 29, 2023

View reviewed changes