Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JDBC driver jenkins CI workflow #957

Closed

Conversation

Yury-Fridlyand
Copy link
Collaborator

Description

Please, see team review discussion in Bit-Quill#140

Jenkins file

Combined from this and this.
It calls publishing gradle workflow to generate artifacts.

Gradle workflow

New workflow (should be called from sql-jdbc dir)

./gradlew publishPublishMavenPublicationToLocalRepoRepository
Files generated by gradle
$ tree build | grep -vF .class
build
├── classes
│   └── java
│       └── main
│           └── org
│               └── opensearch
│                   └── jdbc
│                       ├── auth
│                       ├── config
│                       ├── internal
│                       │   ├── exceptions
│                       │   ├── results
│                       │   └── util
│                       ├── logging
│                       ├── protocol
│                       │   ├── exceptions
│                       │   └── http
│                       ├── transport
│                       │   └── http
│                       │       └── auth
│                       │           └── aws
│                       └── types
├── libs
│   └── opensearch-sql-jdbc-2.0.0.0.jar
├── publications
│   └── publishMaven
│       └── pom-default.xml
├── repository
│   └── org
│       └── opensearch
│           └── client
│               └── opensearch-sql-jdbc
│                   ├── 2.0.0.0
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom.md5
│                   │   └── opensearch-sql-jdbc-2.0.0.0.pom.sha1
│                   ├── maven-metadata.xml
│                   ├── maven-metadata.xml.md5
│                   └── maven-metadata.xml.sha1
├── resources
│   └── main
│       └── META-INF
│           └── services
│               └── java.sql.Driver
└── tmp
    ├── compileJava
    ├── jar
    │   └── MANIFEST.MF
    └── publishPublishMavenPublicationToLocalRepoRepository
        └── org
            └── opensearch
                └── client
                    └── opensearch-sql-jdbc
                        ├── maven-metadata-remote.xml
                        └── resolver-status.properties

42 directories, 159 files

See artifacts in artifacts.zip. The jar file has no dependencies inside.

jar content
$ tree
.
├── META-INF
│   ├── MANIFEST.MF
│   └── services
│       └── java.sql.Driver
└── org
    └── opensearch
        └── jdbc
            ├── ConnectionImpl.class
            ├── DatabaseMetaDataImpl$ColumnMetadataStatement$ColumnMetadataResultSet.class
            ├── DatabaseMetaDataImpl$ColumnMetadataStatement.class
            ├── DatabaseMetaDataImpl$ResultSetColumnDescriptor.class
            ├── DatabaseMetaDataImpl.class
            ├── Driver.class
            ├── OpenSearchConnection.class
            ├── OpenSearchDataSource.class
            ├── OpenSearchVersion.class
            ├── PreparedStatementImpl.class
            ├── ResultSetImpl.class
            ├── ResultSetMetaDataImpl.class
            ├── StatementImpl.class
            ├── auth
            │   └── AuthenticationType.class
            ├── config
            │   ├── AuthConnectionProperty.class
            │   ├── AwsCredentialsProviderProperty.class
            │   ├── BoolConnectionProperty.class
            │   ├── ConnectionConfig$1.class
            │   ├── ConnectionConfig$Builder.class
            │   ├── ConnectionConfig.class
            │   ├── ConnectionProperty.class
            │   ├── ConnectionPropertyException.class
            │   ├── FetchSizeProperty.class
            │   ├── HostConnectionProperty.class
            │   ├── HostnameVerificationConnectionProperty.class
            │   ├── IntConnectionProperty.class
            │   ├── KeyStoreLocationConnectionProperty.class
            │   ├── KeyStorePasswordConnectionProperty.class
            │   ├── KeyStoreTypeConnectionProperty.class
            │   ├── LogLevelConnectionProperty.class
            │   ├── LogOutputConnectionProperty.class
            │   ├── LoginTimeoutConnectionProperty.class
            │   ├── PasswordConnectionProperty.class
            │   ├── PathConnectionProperty.class
            │   ├── PortConnectionProperty.class
            │   ├── RegionConnectionProperty.class
            │   ├── RequestCompressionConnectionProperty.class
            │   ├── StringConnectionProperty.class
            │   ├── TrustSelfSignedConnectionProperty.class
            │   ├── TrustStoreLocationConnectionProperty.class
            │   ├── TrustStorePasswordConnectionProperty.class
            │   ├── TrustStoreTypeConnectionProperty.class
            │   ├── UseSSLConnectionProperty.class
            │   └── UserConnectionProperty.class
            ├── internal
            │   ├── JdbcWrapper.class
            │   ├── Version.class
            │   ├── exceptions
            │   │   └── ObjectClosedException.class
            │   ├── results
            │   │   ├── ColumnMetaData.class
            │   │   ├── Cursor.class
            │   │   ├── Row.class
            │   │   └── Schema.class
            │   └── util
            │       ├── AwsHostNameUtil.class
            │       ├── JavaUtil.class
            │       ├── SqlParser.class
            │       └── UrlParser.class
            ├── logging
            │   ├── FilePrintWriterLogger.class
            │   ├── Layout.class
            │   ├── LogLevel.class
            │   ├── Logger.class
            │   ├── LoggerFactory.class
            │   ├── LoggingSource$1.class
            │   ├── LoggingSource.class
            │   ├── NoOpLogger.class
            │   ├── PrintWriterLogger.class
            │   └── StandardLayout.class
            ├── protocol
            │   ├── ClusterMetadata.class
            │   ├── ColumnDescriptor.class
            │   ├── ConnectionResponse.class
            │   ├── JdbcDateTimeFormatter.class
            │   ├── JdbcQueryParam.class
            │   ├── JdbcQueryRequest.class
            │   ├── Parameter.class
            │   ├── Protocol.class
            │   ├── ProtocolFactory.class
            │   ├── QueryRequest.class
            │   ├── QueryResponse.class
            │   ├── RequestError.class
            │   ├── exceptions
            │   │   ├── InternalServerErrorException.class
            │   │   ├── MalformedResponseException.class
            │   │   └── ResponseException.class
            │   └── http
            │       ├── HttpException.class
            │       ├── HttpResponseHandler.class
            │       ├── JdbcCursorQueryRequest.class
            │       ├── JsonClusterMetadata.class
            │       ├── JsonConnectionResponse.class
            │       ├── JsonCursorHttpProtocol.class
            │       ├── JsonCursorHttpProtocolFactory.class
            │       ├── JsonCursorQueryRequest.class
            │       ├── JsonHttpProtocol.class
            │       ├── JsonHttpProtocolFactory.class
            │       ├── JsonHttpResponseHandler$JsonContentParser.class
            │       ├── JsonHttpResponseHandler.class
            │       ├── JsonOpenSearchVersion.class
            │       ├── JsonQueryRequest.class
            │       ├── JsonQueryResponse$JsonRequestError.class
            │       ├── JsonQueryResponse$SchemaEntry.class
            │       └── JsonQueryResponse.class
            ├── transport
            │   ├── Transport.class
            │   ├── TransportException.class
            │   ├── TransportFactory.class
            │   └── http
            │       ├── ApacheHttpClientConnectionFactory.class
            │       ├── ApacheHttpTransport.class
            │       ├── ApacheHttpTransportFactory.class
            │       ├── HttpParam.class
            │       ├── HttpTransport.class
            │       ├── JclLoggerAdapter.class
            │       ├── LoggingInputStream.class
            │       ├── LoggingManagedHttpClientConnection.class
            │       ├── LoggingOutputStream.class
            │       └── auth
            │           └── aws
            │               └── AWSRequestSigningApacheInterceptor.class
            └── types
                ├── BaseTypeConverter.class
                ├── BinaryType.class
                ├── BooleanType.class
                ├── ByteType.class
                ├── DateType.class
                ├── DoubleType.class
                ├── FloatType.class
                ├── IntegerType.class
                ├── LongType.class
                ├── NumberType.class
                ├── OpenSearchType.class
                ├── ShortType.class
                ├── StringType.class
                ├── TimeType.class
                ├── TimestampType.class
                ├── TypeConverter.class
                ├── TypeConverters$1.class
                ├── TypeConverters$BigIntTypeConverter.class
                ├── TypeConverters$BinaryTypeConverter.class
                ├── TypeConverters$BooleanTypeConverter.class
                ├── TypeConverters$DateTypeConverter.class
                ├── TypeConverters$DoubleTypeConverter.class
                ├── TypeConverters$FloatTypeConverter.class
                ├── TypeConverters$IntegerTypeConverter.class
                ├── TypeConverters$NullTypeConverter.class
                ├── TypeConverters$RealTypeConverter.class
                ├── TypeConverters$SmallIntTypeConverter.class
                ├── TypeConverters$TimeTypeConverter.class
                ├── TypeConverters$TimestampTypeConverter.class
                ├── TypeConverters$TinyIntTypeConverter.class
                ├── TypeConverters$VarcharTypeConverter.class
                ├── TypeConverters.class
                ├── TypeHelper.class
                └── UnrecognizedOpenSearchTypeException.class

20 directories, 146 files

Shadow copy publishing could be invoked by

./gradlew publishShadowPublicationToLocalRepoRepository
Artifacts
$ tree build | grep -vF .class
build
├── classes
│   └── java
│       └── main
│           └── org
│               └── opensearch
│                   └── jdbc
│                       ├── auth
│                       ├── config
│                       ├── internal
│                       │   ├── exceptions
│                       │   ├── results
│                       │   └── util
│                       ├── logging
│                       ├── protocol
│                       │   ├── exceptions
│                       │   └── http
│                       ├── transport
│                       │   └── http
│                       │       └── auth
│                       │           └── aws
│                       └── types
├── libs
│   ├── opensearch-sql-jdbc-2.0.0.0-javadoc.jar
│   ├── opensearch-sql-jdbc-2.0.0.0-sources.jar
│   └── opensearch-sql-jdbc-2.0.0.0.jar
├── publications
│   └── shadow
│       └── pom-default.xml
├── repository
│   └── org
│       └── opensearch
│           └── client
│               └── opensearch-sql-jdbc
│                   ├── 2.0.0.0
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-javadoc.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-javadoc.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-javadoc.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-sources.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-sources.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-sources.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom.md5
│                   │   └── opensearch-sql-jdbc-2.0.0.0.pom.sha1
│                   ├── maven-metadata.xml
│                   ├── maven-metadata.xml.md5
│                   └── maven-metadata.xml.sha1
├── resources
│   └── main
│       └── META-INF
│           └── services
│               └── java.sql.Driver
└── tmp
    ├── compileJava
    ├── javadocJar
    │   └── MANIFEST.MF
    ├── publishShadowPublicationToLocalRepoRepository
    │   └── org
    │       └── opensearch
    │           └── client
    │               └── opensearch-sql-jdbc
    │                   ├── maven-metadata-remote.xml
    │                   └── resolver-status.properties
    ├── shadowJar
    │   └── MANIFEST.MF
    └── sourcesJar
        └── MANIFEST.MF

44 directories, 169 files
Shadow copy content
$ tree . | grep -vF .class
.
├── META-INF
│   ├── MANIFEST.MF
│   ├── services
│   │   └── java.sql.Driver
│   └── versions
│       └── 9
├── mozilla
│   └── public-suffix-list.txt
└── org
    └── opensearch
        ├── jdbc
        │   ├── auth
        │   ├── config
        │   ├── internal
        │   │   ├── exceptions
        │   │   ├── results
        │   │   └── util
        │   ├── logging
        │   ├── protocol
        │   │   ├── exceptions
        │   │   └── http
        │   ├── transport
        │   │   └── http
        │   │       └── auth
        │   │           └── aws
        │   └── types
        └── sql
            └── jdbc
                └── shadow
                    ├── com
                    │   ├── amazonaws
                    │   │   ...
                    │   └── fasterxml
                    │       └── jackson
                    │           ...
                    ├── org
                    │   ├── apache
                    │   │   ├── commons
                    │   │   │   ├── codec
                    │   │   │   │   ...
                    │   │   │   └── logging
                    │   │   │       └── impl
                    │   │   └── http
                    │   │       ...
                    │   └── joda
                    │       └── time
                    │           ...
                    └── software
                        └── amazon
                            └── ion
                                ...

223 directories, 3902 files

Obsolete references

I removed obsolete references to com.amazonaws.opensearch (replaced by org.opensearch where applicable).

Issues Resolved

Adds JDBC driver publishing workflow

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Add JDBC driver jenkins CI workflow
@codecov-commenter

This comment was marked as spam.

Copy link
Member

@peterzhuamazon peterzhuamazon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Yury-Fridlyand please see these comments and we can discuss more later.
Thanks

jenkins/jdbc-stage-maven-release.jenkinsfile Outdated Show resolved Hide resolved
jenkins/jdbc-stage-maven-release.jenkinsfile Outdated Show resolved Hide resolved
jenkins/jdbc-stage-maven-release.jenkinsfile Show resolved Hide resolved
Comment on lines +33 to +46
stage('Publish to Maven Local') {
steps {
// checkout the commit
checkout([
$class: 'GitSCM', userRemoteConfigs: [[url: 'https://github.com/opensearch-project/sql.git']],
branches: [[name: "$ref"]]
])

dir('sql-jdbc') {
// publish maven artifacts
sh('./gradlew --no-daemon publishPublishMavenPublicationToLocalRepoRepository')
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will need to discuss a bit as we want two options: triggered by tag or manually trigger on Jenkins.
Just in case there is issues with github and we can still release.
See maps server jenkinsfile for more references:
https://github.com/peterzhuamazon/maps/blob/a40788a611e1d0f573d715e0b1ce59b7fb33e4d0/jenkins/maps-server-release.jenkinsfile

}
}
}
publishMaven(MavenPublication) { publication ->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the group of the jdbc?
org.opensearch.client.opensearch-sql-jdbc

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, org.opensearch.client. What should be?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Yury-Fridlyand and @peterzhuamazon should this be org.opensearch.driver as groupId? and artifact name should be opensearch-sql-jdbc.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 79deba0.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my next comment on this.

jenkins/jdbc-stage-maven-release.jenkinsfile Show resolved Hide resolved
@Yury-Fridlyand Yury-Fridlyand marked this pull request as draft October 26, 2022 01:03
Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
@@ -39,7 +39,7 @@ pipeline {
)
}
environment {
ARTIFACT_PATH = "$WORKSPACE/build/repository/org/opensearch/client/sql/$VERSION"
ARTIFACT_PATH = "$WORKSPACE/build/repository/org/opensearch/driver/opensearch-sql-jdbc/$VERSION"
Copy link
Member

@peterzhuamazon peterzhuamazon Oct 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to put the jdbc in this location during ODFE.
Probably we want to keep similar subfolder structure?

https://repo1.maven.org/maven2/com/amazon/opendistroforelasticsearch/client/opendistro-sql-jdbc/

Thanks.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the path where gradle places compiled jar:

$ tree build | grep -v class
build
....
├── repository
│   └── org
│       └── opensearch
│           └── driver
│               └── opensearch-sql-jdbc
│                   ├── 2.0.0.0
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar
....

In general, there is the same structure, but we renamed

  • com.amazon to org.opensearch
  • opendistroforelasticsearch and opendistro to opensearch
  • client to driver

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I revert this?

url = "https://github.com/opensearch-project/sql/sql-jdbc"
description = "OpenSearch SQL JDBC driver"
scm {
connection = "scm:[email protected]:opensearch-project/sql.git"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move to opensearch-project/sql-jdbc.git

scm {
connection = "scm:[email protected]:opensearch-project/sql.git"
developerConnection = "scm:[email protected]:opensearch-project/sql.git"
url = "[email protected]:opensearch-project/sql.git"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move to opensearch-project/sql-jdbc.git

developers {
developer {
name = 'OpenSearch'
url = 'https://github.com/opensearch-project/sql'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move to opensearch-project/sql-jdbc.git

url = "http://www.apache.org/licenses/LICENSE-2.0.txt"
name = "OpenSearch SQL JDBC Driver"
packaging = "jar"
url = "https://github.com/opensearch-project/sql/sql-jdbc"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move to opensearch-project/sql-jdbc

pom {
name = "OpenSearch SQL JDBC Driver"
packaging = "jar"
url = "https://github.com/opensearch-project/sql/sql-jdbc"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move to opensearch-project/sql-jdbc

url = "https://github.com/opensearch-project/sql/sql-jdbc"
description = "OpenSearch SQL JDBC driver"
scm {
connection = "scm:[email protected]:opensearch-project/sql.git"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move to opensearch-project/sql-jdbc.git

scm {
connection = "scm:[email protected]:opensearch-project/sql.git"
developerConnection = "scm:[email protected]:opensearch-project/sql.git"
url = "[email protected]:opensearch-project/sql.git"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move to opensearch-project/sql-jdbc.git

developers {
developer {
name = 'OpenSearch'
url = 'https://github.com/opensearch-project/sql'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move to opensearch-project/sql-jdbc

@YANG-DB YANG-DB marked this pull request as ready for review December 2, 2022 22:13
@YANG-DB
Copy link
Member

YANG-DB commented Dec 6, 2022

Hi Yury
Can you plz move this PR to the new dedicated sql-jdbc repository ?
https://github.com/opensearch-project/sql-jdbc
Thanks...

Yury-Fridlyand added a commit to Bit-Quill/sql-jdbc that referenced this pull request Dec 9, 2022
@Yury-Fridlyand
Copy link
Collaborator Author

Closing - changes and discussion to be tracked in opensearch-project/sql-jdbc#15.

@Yury-Fridlyand Yury-Fridlyand deleted the integ-jdbc-jenkins-ci branch December 9, 2022 03:53
YANG-DB pushed a commit to opensearch-project/sql-jdbc that referenced this pull request Dec 9, 2022
Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>
Yury-Fridlyand added a commit to Bit-Quill/sql-jdbc that referenced this pull request Jan 5, 2023
Signed-off-by: Yury-Fridlyand <[email protected]>
(cherry picked from commit 0e8de37)

Signed-off-by: Yury-Fridlyand <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants