forked from opensearch-project/sql
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'opensearch-project:2.x' into 2.x
- Loading branch information
Showing
414 changed files
with
18,306 additions
and
6,657 deletions.
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| # This should match the owning team set up in https://github.com/orgs/opensearch-project/teams | ||
| * @pjfitzgibbons @ps48 @kavithacm @derek-ho @joshuali925 @dai-chen @YANG-DB @rupal-bq @mengweieric @vamsi-amazon @swiddis @penghuo @seankao-az @MaxKsyunz @Yury-Fridlyand @anirudha @forestmvey @acarbonetto @GumpacG | ||
| * @pjfitzgibbons @ps48 @kavithacm @derek-ho @joshuali925 @dai-chen @YANG-DB @rupal-bq @mengweieric @vamsi-amazon @swiddis @penghuo @seankao-az @MaxKsyunz @Yury-Fridlyand @anirudha @forestmvey @acarbonetto @GumpacG @ykmr1224 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| .gradle | ||
| build/ | ||
| !gradle/wrapper/gradle-wrapper.jar | ||
| !src/main/**/build/ | ||
| !src/test/**/build/ | ||
|
|
||
| ### IntelliJ IDEA ### | ||
| .idea/modules.xml | ||
| .idea/jarRepositories.xml | ||
| .idea/compiler.xml | ||
| .idea/libraries/ | ||
| *.iws | ||
| *.iml | ||
| *.ipr | ||
| out/ | ||
| !src/main/**/out/ | ||
| !src/test/**/out/ | ||
|
|
||
| ### Eclipse ### | ||
| .apt_generated | ||
| .classpath | ||
| .factorypath | ||
| .project | ||
| .settings | ||
| .springBeans | ||
| .sts4-cache | ||
| bin/ | ||
| !src/main/**/bin/ | ||
| !src/test/**/bin/ | ||
|
|
||
| ### NetBeans ### | ||
| /nbproject/private/ | ||
| /nbbuild/ | ||
| /dist/ | ||
| /nbdist/ | ||
| /.nb-gradle/ | ||
|
|
||
| ### VS Code ### | ||
| .vscode/ | ||
|
|
||
| ### Mac OS ### | ||
| .DS_Store |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # async-query-core library | ||
|
|
||
| This directory contains async-query-core library, which implements the core logic of async-query and provide extension points to allow plugin different implementation of data storage, etc. | ||
| `async-query` module provides implementations for OpenSearch index based implementation. | ||
|
|
||
| ## Type of queries | ||
| There are following types of queries, and the type is automatically identified by analysing the query. | ||
| - BatchQuery: Execute single query in Spark | ||
| - InteractiveQuery: Establish session and execute queries in single Spark session | ||
| - IndexDMLQuery: Handles DROP/ALTER/VACUUM operation for Flint indices | ||
| - RefreshQuery: One time query request to refresh(update) Flint index | ||
| - StreamingQuery: Continuously update flint index in single Spark session | ||
|
|
||
| ## Extension points | ||
| Following is the list of extension points where the consumer of the library needs to provide their own implementation. | ||
|
|
||
| - Data store interface | ||
| - [AsyncQueryJobMetadataStorageService](src/main/java/org/opensearch/sql/spark/asyncquery/AsyncQueryJobMetadataStorageService.java) | ||
| - [SessionStorageService](java/org/opensearch/sql/spark/execution/statestore/SessionStorageService.java) | ||
| - [StatementStorageService](src/main/java/org/opensearch/sql/spark/execution/statestore/StatementStorageService.java) | ||
| - [FlintIndexMetadataService](src/main/java/org/opensearch/sql/spark/flint/FlintIndexMetadataService.java) | ||
| - [FlintIndexStateModelService](src/main/java/org/opensearch/sql/spark/flint/FlintIndexStateModelService.java) | ||
| - [IndexDMLResultStorageService](src/main/java/org/opensearch/sql/spark/flint/IndexDMLResultStorageService.java) | ||
| - Other | ||
| - [LeaseManager](src/main/java/org/opensearch/sql/spark/leasemanager/LeaseManager.java) | ||
| - [JobExecutionResponseReader](src/main/java/org/opensearch/sql/spark/response/JobExecutionResponseReader.java) | ||
| - [QueryIdProvider](src/main/java/org/opensearch/sql/spark/dispatcher/QueryIdProvider.java) | ||
| - [SessionIdProvider](src/main/java/org/opensearch/sql/spark/execution/session/SessionIdProvider.java) | ||
| - [SessionConfigSupplier](src/main/java/org/opensearch/sql/spark/execution/session/SessionConfigSupplier.java) | ||
| - [SparkExecutionEngineConfigSupplier](src/main/java/org/opensearch/sql/spark/config/SparkExecutionEngineConfigSupplier.java) | ||
| - [SparkSubmitParameterModifier](src/main/java/org/opensearch/sql/spark/config/SparkSubmitParameterModifier.java) | ||
| - [EMRServerlessClientFactory](src/main/java/org/opensearch/sql/spark/client/EMRServerlessClientFactory.java) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| /* | ||
| * Copyright OpenSearch Contributors | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| plugins { | ||
| id 'java-library' | ||
| id "io.freefair.lombok" | ||
| id 'jacoco' | ||
| id 'antlr' | ||
| id 'com.diffplug.spotless' version '6.22.0' | ||
| id 'com.github.johnrengelman.shadow' | ||
| } | ||
|
|
||
| repositories { | ||
| mavenCentral() | ||
| } | ||
|
|
||
| tasks.register('downloadG4Files', Exec) { | ||
| description = 'Download remote .g4 files from GitHub' | ||
|
|
||
| executable 'curl' | ||
|
|
||
| args '-o', 'src/main/antlr/FlintSparkSqlExtensions.g4', 'https://raw.githubusercontent.com/opensearch-project/opensearch-spark/main/flint-spark-integration/src/main/antlr4/FlintSparkSqlExtensions.g4' | ||
| args '-o', 'src/main/antlr/SparkSqlBase.g4', 'https://raw.githubusercontent.com/opensearch-project/opensearch-spark/main/flint-spark-integration/src/main/antlr4/SparkSqlBase.g4' | ||
| args '-o', 'src/main/antlr/SqlBaseParser.g4', 'https://raw.githubusercontent.com/apache/spark/master/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4' | ||
| args '-o', 'src/main/antlr/SqlBaseLexer.g4', 'https://raw.githubusercontent.com/apache/spark/master/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4' | ||
| } | ||
|
|
||
| generateGrammarSource { | ||
| arguments += ['-visitor', '-package', 'org.opensearch.sql.spark.antlr.parser'] | ||
| source = sourceSets.main.antlr | ||
| outputDirectory = file("build/generated-src/antlr/main/org/opensearch/sql/asyncquery/antlr/parser") | ||
| } | ||
| configurations { | ||
| compile { | ||
| extendsFrom = extendsFrom.findAll { it != configurations.antlr } | ||
| } | ||
| } | ||
|
|
||
| // skip download in case of offline build | ||
| if (!gradle.startParameter.offline) { | ||
| // Make sure the downloadG4File task runs before the generateGrammarSource task | ||
| generateGrammarSource.dependsOn downloadG4Files | ||
| } | ||
|
|
||
| dependencies { | ||
| antlr "org.antlr:antlr4:4.7.1" | ||
|
|
||
| implementation project(':core') | ||
| implementation project(':spark') // TODO: dependency to spark should be eliminated | ||
| implementation project(':datasources') // TODO: dependency to datasources should be eliminated | ||
| implementation project(':legacy') // TODO: dependency to legacy should be eliminated | ||
| implementation 'org.json:json:20231013' | ||
| implementation 'com.google.code.gson:gson:2.8.9' | ||
|
|
||
| testImplementation(platform("org.junit:junit-bom:5.9.3")) | ||
|
|
||
| testCompileOnly('org.junit.jupiter:junit-jupiter') | ||
| testImplementation 'org.mockito:mockito-core:5.7.0' | ||
| testImplementation 'org.mockito:mockito-junit-jupiter:5.7.0' | ||
|
|
||
| testCompileOnly('junit:junit:4.13.1') { | ||
| exclude group: 'org.hamcrest', module: 'hamcrest-core' | ||
| } | ||
| testRuntimeOnly("org.junit.vintage:junit-vintage-engine") { | ||
| exclude group: 'org.hamcrest', module: 'hamcrest-core' | ||
| } | ||
| testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine") { | ||
| exclude group: 'org.hamcrest', module: 'hamcrest-core' | ||
| } | ||
| testRuntimeOnly("org.junit.platform:junit-platform-launcher") { | ||
| because 'allows tests to run from IDEs that bundle older version of launcher' | ||
| } | ||
| } | ||
|
|
||
| spotless { | ||
| java { | ||
| target fileTree('.') { | ||
| include '**/*.java' | ||
| exclude '**/build/**', '**/build-*/**' | ||
| } | ||
| importOrder() | ||
| removeUnusedImports() | ||
| trimTrailingWhitespace() | ||
| endWithNewline() | ||
| googleJavaFormat('1.17.0').reflowLongStrings().groupArtifact('com.google.googlejavaformat:google-java-format') | ||
| } | ||
| } | ||
|
|
||
| test { | ||
| useJUnitPlatform() | ||
| testLogging { | ||
| events "skipped", "failed" | ||
| exceptionFormat "full" | ||
| } | ||
| } | ||
|
|
||
| jacocoTestReport { | ||
| reports { | ||
| html.required = true | ||
| xml.required = true | ||
| } | ||
| afterEvaluate { | ||
| classDirectories.setFrom(files(classDirectories.files.collect { | ||
| fileTree(dir: it, exclude: ['**/antlr/parser/**']) | ||
| })) | ||
| } | ||
| } | ||
| test.finalizedBy(project.tasks.jacocoTestReport) | ||
| jacocoTestCoverageVerification { | ||
| violationRules { | ||
| rule { | ||
| element = 'CLASS' | ||
| // TODO: Add unit tests in async-query-core and remove exclusions | ||
| excludes = [ | ||
| 'org.opensearch.sql.spark.asyncquery.model.*', | ||
| 'org.opensearch.sql.spark.data.constants.*', | ||
| 'org.opensearch.sql.spark.dispatcher.model.*', | ||
| 'org.opensearch.sql.spark.dispatcher.*', | ||
| 'org.opensearch.sql.spark.execution.session.*', | ||
| 'org.opensearch.sql.spark.execution.statement.*', | ||
| 'org.opensearch.sql.spark.flint.*', | ||
| 'org.opensearch.sql.spark.flint.operation.*', | ||
| 'org.opensearch.sql.spark.rest.*', | ||
| 'org.opensearch.sql.spark.utils.SQLQueryUtils.*' | ||
| ] | ||
| limit { | ||
| counter = 'LINE' | ||
| minimum = 1.0 | ||
| } | ||
| limit { | ||
| counter = 'BRANCH' | ||
| minimum = 1.0 | ||
| } | ||
| } | ||
| } | ||
| afterEvaluate { | ||
| classDirectories.setFrom(files(classDirectories.files.collect { | ||
| fileTree(dir: it, exclude: ['**/antlr/parser/**']) | ||
| })) | ||
| } | ||
| } | ||
| check.dependsOn jacocoTestCoverageVerification | ||
|
|
||
| shadowJar { | ||
| archiveBaseName.set('async-query-core') | ||
| archiveVersion.set('1.0.0') // Set the desired version | ||
| archiveClassifier.set('all') | ||
|
|
||
| from sourceSets.main.output | ||
| configurations = [project.configurations.runtimeClasspath] | ||
| } |
Empty file.
Oops, something went wrong.