Skip to content

Commit

Permalink
OpenSearch migration for ODBC driver (#7)
Browse files Browse the repository at this point in the history
* Rename Elasticsearch

* Bump plugin version to 1.15

* Rename es_api30

* Rename es_apifunc

* Rename es_communication

* Rename es_connection

* Fix connection.c

* Rename es_driver_connect and es_semaphore

* Rename es_helper

* Fix test and rename es_info

* Rename es_parse_result

* Rename es_statement

* Rename es_result_queue

* Rename es_utility

* Rename es_types

* Rename es_odbc

* Fix mylog

* Fix header and rename ESLogLevel

* Rename es_result

* Rename esNAME, esconn

* Rename estype_ functions

* Rename eserror

* Rename more variables and functions

* Rename more variables and functions

* Fix error and rename ESAPI_

* Fix error

* Fix error

* Fix error

* More renaming

* Fix elasticenlist error

* Replace Kibana

* Fix links to old repo

* Fix plugin name in github action

* Change plugin name to OpenSearch

* Replace all other ODFE and links

* Revert release notes
  • Loading branch information
dai-chen authored Apr 17, 2021
1 parent efde457 commit 13c022a
Show file tree
Hide file tree
Showing 136 changed files with 3,133 additions and 3,028 deletions.
12 changes: 6 additions & 6 deletions .github/workflows/sql-odbc-release-workflow.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: Build and Release SQL-ODBC
# This workflow is triggered on creating tags to master or an opendistro release branch
# This workflow is triggered on creating tags to main or an opensearch release branch
on:
push:
tags:
Expand All @@ -11,8 +11,8 @@ env:
ODBC_BIN_PATH: "./build/odbc/bin"
ODBC_BUILD_PATH: "./build/odbc/build"
AWS_SDK_INSTALL_PATH: "./build/aws-sdk/install"
PLUGIN_NAME: opendistro-sql-odbc
OD_VERSION: 1.13.0.0
PLUGIN_NAME: opensearch-sql-odbc
OD_VERSION: 1.15.0.0

jobs:
build-mac:
Expand Down Expand Up @@ -82,7 +82,7 @@ jobs:
run: |
cd installer
pkg=`ls -1t *.pkg | grep "Open Distro for Elasticsearch SQL ODBC Driver" | head -1`
pkg=`ls -1t *.pkg | grep "OpenSearch SQL ODBC Driver" | head -1`
mv "$pkg" "${{ env.PLUGIN_NAME }}-${{ env.OD_VERSION }}-macos-x64.pkg"
pkg=`ls -1t *.pkg | grep "${{ env.PLUGIN_NAME }}-${{ env.OD_VERSION }}-macos-x64.pkg" | head -1`
Expand Down Expand Up @@ -140,7 +140,7 @@ jobs:
run: |
cd ci-output/installer
msi=`ls -1t *.msi | grep "Open Distro for Elasticsearch SQL ODBC Driver" | head -1`
msi=`ls -1t *.msi | grep "OpenSearch SQL ODBC Driver" | head -1`
mv "$msi" "${{ env.PLUGIN_NAME }}-${{ env.OD_VERSION }}-windows-x86.msi"
msi=`ls -1t *.msi | grep "${{ env.PLUGIN_NAME }}-${{ env.OD_VERSION }}-windows-x86.msi" | head -1`
Expand Down Expand Up @@ -198,7 +198,7 @@ jobs:
run: |
cd ci-output/installer
msi=`ls -1t *.msi | grep "Open Distro for Elasticsearch SQL ODBC Driver" | head -1`
msi=`ls -1t *.msi | grep "OpenSearch SQL ODBC Driver" | head -1`
mv "$msi" "${{ env.PLUGIN_NAME }}-${{ env.OD_VERSION }}-windows-x64.msi"
msi=`ls -1t *.msi | grep "${{ env.PLUGIN_NAME }}-${{ env.OD_VERSION }}-windows-x64.msi" | head -1`
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/sql-odbc-rename-and-release-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
- rename*

env:
OD_VERSION: 1.13.0.0
OD_VERSION: 1.15.0.0

jobs:
upload-odbc:
Expand All @@ -27,8 +27,8 @@ jobs:
run: |
mkdir macos
cd macos
aws s3 cp "s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/mac/Open Distro for Elasticsearch SQL ODBC Driver 64-bit-1.11.0.0-Darwin.pkg" "Open Distro for Elasticsearch SQL ODBC Driver 64-bit-${{ env.OD_VERSION }}-Darwin.pkg"
mac_installer=`ls -1t *.pkg | grep "Open Distro for Elasticsearch SQL ODBC Driver" | head -1`
aws s3 cp "s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/mac/OpenSearch SQL ODBC Driver 64-bit-1.11.0.0-Darwin.pkg" "OpenSearch SQL ODBC Driver 64-bit-${{ env.OD_VERSION }}-Darwin.pkg"
mac_installer=`ls -1t *.pkg | grep "OpenSearch SQL ODBC Driver" | head -1`
echo $mac_installer
aws s3 cp "$mac_installer" s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/mac/
cd ..
Expand All @@ -37,8 +37,8 @@ jobs:
run: |
mkdir win32
cd win32
aws s3 cp "s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/windows/Open Distro for Elasticsearch SQL ODBC Driver 32-bit-1.11.0.0-Windows.msi" "Open Distro for Elasticsearch SQL ODBC Driver 32-bit-${{ env.OD_VERSION }}-Windows.msi"
windows_installer=`ls -1t *.msi | grep "Open Distro for Elasticsearch SQL ODBC Driver" | head -1`
aws s3 cp "s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/windows/OpenSearch SQL ODBC Driver 32-bit-1.11.0.0-Windows.msi" "OpenSearch SQL ODBC Driver 32-bit-${{ env.OD_VERSION }}-Windows.msi"
windows_installer=`ls -1t *.msi | grep "OpenSearch SQL ODBC Driver" | head -1`
echo $windows_installer
aws s3 cp "$windows_installer" s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/windows/
cd ..
Expand All @@ -47,8 +47,8 @@ jobs:
run: |
mkdir win64
cd win64
aws s3 cp "s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/windows/Open Distro for Elasticsearch SQL ODBC Driver 64-bit-1.11.0.0-Windows.msi" "Open Distro for Elasticsearch SQL ODBC Driver 64-bit-${{ env.OD_VERSION }}-Windows.msi"
windows_installer=`ls -1t *.msi | grep "Open Distro for Elasticsearch SQL ODBC Driver" | head -1`
aws s3 cp "s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/windows/OpenSearch SQL ODBC Driver 64-bit-1.11.0.0-Windows.msi" "OpenSearch SQL ODBC Driver 64-bit-${{ env.OD_VERSION }}-Windows.msi"
windows_installer=`ls -1t *.msi | grep "OpenSearch SQL ODBC Driver" | head -1`
echo $windows_installer
aws s3 cp "$windows_installer" s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-odbc/windows/
2 changes: 1 addition & 1 deletion sql-odbc/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,6 @@ If you discover a potential security issue in this project we ask that you notif

## Licensing

See the [LICENSE](https://github.com/opendistro-for-elasticsearch/sql/blob/master/sql-odbc/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
See the [LICENSE](https://github.com/opensearch-project/sql/blob/main/sql-odbc/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.

We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.
22 changes: 11 additions & 11 deletions sql-odbc/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Open Distro for Elasticsearch ODBC Driver
# OpenSearch ODBC Driver

ElasticsearchODBC is a read-only ODBC driver for Windows and Mac for connecting to Open Distro for Elasticsearch SQL support.
OpenSearchODBC is a read-only ODBC driver for Windows and Mac for connecting to OpenSearch SQL support.

## Specifications

Expand All @@ -25,33 +25,33 @@ You can use the installers generated as part of the most recent release.

### Windows

1. Run the `.msi` installer to install the Open Distro for Elasticsearch SQL ODBC Driver.
1. Run the `.msi` installer to install the OpenSearch SQL ODBC Driver.
2. [Test connection](./docs/user/windows_configure_dsn.md) using ODBC Data Source Administrator.

To use the driver with Tableau:
1. Copy the `.tdc` file from `<driver-install-dir>/resources` to `<windows-user-dir>/Documents/My Tableau Repository/Datasources`.

This will customize the connection from Tableau to Open Distro for Elasticsearch, ensuring that the correct forms of queries are used.
This will customize the connection from Tableau to OpenSearch, ensuring that the correct forms of queries are used.

### Mac

iODBC Driver Manager should be installed before installing the Open Distro for Elasticsearch SQL ODBC Driver on Mac.
iODBC Driver Manager should be installed before installing the OpenSearch SQL ODBC Driver on Mac.

1. Run the `.pkg` installer to install the Open Distro for Elasticsearch SQL ODBC Driver.
2. Configure a Driver and DSN entry for the Open Distro for Elasticsearch SQL ODBC Driver, following the instructions [here](./docs/user/mac_configure_dsn.md).
1. Run the `.pkg` installer to install the OpenSearch SQL ODBC Driver.
2. Configure a Driver and DSN entry for the OpenSearch SQL ODBC Driver, following the instructions [here](./docs/user/mac_configure_dsn.md).

To use the driver with Tableau:
1. Copy the `.tdc` file from `<driver-install-dir>/resources` to `<mac-user-dir>/Documents/My Tableau Repository/Datasources`.

This will customize the connection from Tableau to Open Distro for Elasticsearch, ensuring that the correct forms of queries are used.
This will customize the connection from Tableau to OpenSearch, ensuring that the correct forms of queries are used.

## Using the Driver

The driver comes in the form of a library file:
* Windows: `odfesqlodbc.dll`
* Mac: `libodfesqlodbc.dylib`

If using with ODBC compatible BI tools, refer to the tool documentation on configuring a new ODBC driver. In most cases, you will need to make the tool aware of the location of the driver library file and then use it to setup Open Distro for Elasticsearch database connections.
If using with ODBC compatible BI tools, refer to the tool documentation on configuring a new ODBC driver. In most cases, you will need to make the tool aware of the location of the driver library file and then use it to setup OpenSearch database connections.

### Connection Strings and Configuring the Driver

Expand All @@ -60,7 +60,7 @@ A list of options available for configuring driver behaviour is available [here]
To setup a connection, the driver uses an ODBC connection string. Connection strings are semicolon-delimited strings specifying the set of options to use for a connection. Typically, a connection string will either:

1. specify a Data Source Name containing a pre-configured set of options (`DSN=xxx;User=xxx;Password=xxx;`)
2. or configure options explicitly using the string (`Host=xxx;Port=xxx;LogLevel=ES_DEBUG;...`)
2. or configure options explicitly using the string (`Host=xxx;Port=xxx;LogLevel=OPENSEARCH_DEBUG;...`)

## Building from source

Expand All @@ -82,7 +82,7 @@ From the projects root directory, then build the project using Visual Studio (Wi

## Documentation

Please refer to the [documentation](https://opendistro.github.io/for-elasticsearch-docs/) for detailed information on installing and configuring Open Distro for Elasticsearch.
Please refer to the [documentation](https://opendistro.github.io/for-elasticsearch-docs/) for detailed information on installing and configuring OpenSearch.

## Code of Conduct

Expand Down
8 changes: 4 additions & 4 deletions sql-odbc/docs/dev/BUILD_INSTRUCTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ build
msbuild .\build\odbc\PACKAGE.vcxproj -p:Configuration=Release
```

`Open Distro for Elasticsearch SQL ODBC Driver-<version>-Windows-<Bitness>-bit.msi` will be generated in the build directory.
`OpenSearch SQL ODBC Driver-<version>-Windows-<Bitness>-bit.msi` will be generated in the build directory.

### Testing
See [run_tests.md](./run_tests.md)
Expand Down Expand Up @@ -96,7 +96,7 @@ cd cmake-build64/
cpack .
```

`Open Distro for Elasticsearch SQL ODBC Driver-<version>-Darwin.pkg` will be generated in the build directory.
`OpenSearch SQL ODBC Driver-<version>-Darwin.pkg` will be generated in the build directory.

### Testing
See [run_tests.md](./run_tests.md)
Expand All @@ -113,9 +113,9 @@ See [run_tests.md](./run_tests.md)

To disable SSL/TLS in the tests, the main CMakeLists.txt file must be edited. This can be found in the project 'src' directory. In the 'General compiler definitions' in the CMakeLists.txt file, USE_SSL is set. Remove this from the add_compile_definitions function to stop SSL/TLS from being used in the tests.

To enable SSL/TLS on Elasticsearch, you must edit the Elasticsearch.yml file, found in the config directory of Elasticsearch. An example Elasticsearch yml file can be found in the dev folder of this project. The certificates specified MUST be in the config directory of the Elasticsearch instance. For more information, please refer to the [Elasticsearch security settings documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html).
To enable SSL/TLS on OpenSearch, you must edit the OpenSearch.yml file, found in the config directory of OpenSearch. An example OpenSearch yml file can be found in the dev folder of this project. The certificates specified MUST be in the config directory of the OpenSearch instance. For more information, please refer to the [OpenSearch security settings documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html).

If you plan to use Kibana, as suggested for this project, you must also edit the Kibana settings. Notice, when specifying a certificate for Kibana, you do not need to place it in the Kibana config directory, but instead must provide the absolute path to it. An example Kibana.yml file can be found in the dev folder of this project. For more information, please refer to the [Kibana settings documentation](https://www.elastic.co/guide/en/kibana/current/settings.html).
If you plan to use OpenSearch Dashboards, as suggested for this project, you must also edit the OpenSearch Dashboards settings. Notice, when specifying a certificate for OpenSearch Dashboards, you do not need to place it in the OpenSearch Dashboards config directory, but instead must provide the absolute path to it. An example opensearch_dashboards.yml file can be found in the dev folder of this project. For more information, please refer to the [OpenSearch Dashboards settings documentation](https://www.elastic.co/guide/en/kibana/current/settings.html).

### Setting up a DSN

Expand Down
8 changes: 4 additions & 4 deletions sql-odbc/docs/dev/Pagination.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Opendistro ODBC Driver Cursor (Pagination) Support Design Documentation
# OpenSearch ODBC Driver Cursor (Pagination) Support Design Documentation

## Overview
Elasticsearch ODBC Driver supports forward-only cursor. This document illustrates how the cursor(pagination) is handled in the driver.
OpenSearch ODBC Driver supports forward-only cursor. This document illustrates how the cursor(pagination) is handled in the driver.

For information on how the pagination is supported on Elasticsearch server, check [Opendistro SQL Cursor (Pagination) Support](https://github.com/opendistro-for-elasticsearch/sql/blob/master/docs/dev/Pagination.md).
For information on how the pagination is supported on OpenSearch server, check [Opendistro SQL Cursor (Pagination) Support](https://github.com/opensearch-project/sql/blob/main/docs/dev/Pagination.md).

## Data Flow
<p align="center">
Expand All @@ -14,7 +14,7 @@ For information on how the pagination is supported on Elasticsearch server, chec
* Step 5 will send a request to close cursor whenever the connection is closed.
* ODBC Driver will provide an option to define fetch size as a connection parameter.
* If fetch size is zero, query will fallback to non-cursor behavior.
* If fetch size is not given then the number of rows per request will be as per server-defined [default fetch size](https://github.com/opendistro-for-elasticsearch/sql/blob/master/docs/dev/Pagination.md#42-salient-points).
* If fetch size is not given then the number of rows per request will be as per server-defined [default fetch size](https://github.com/opensearch-project/sql/blob/main/docs/dev/Pagination.md#42-salient-points).
* ODBC Driver will send the request to close cursor whenever the connection is closed.

## Detailed Design
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# kibana_sample_data_types
# opensearch_dashboards_sample_data_types

## Mapping

```json
PUT /kibana_sample_data_types
PUT /opensearch_dashboards_sample_data_types
{
"mappings": {
"properties": {
Expand Down Expand Up @@ -32,7 +32,7 @@ PUT /kibana_sample_data_types
## Data

```json
POST /kibana_sample_data_types/_doc
POST /opensearch_dashboards_sample_data_types/_doc
{
"type_boolean": true,
"type_byte" : -120,
Expand All @@ -52,7 +52,7 @@ POST /kibana_sample_data_types/_doc
```

```json
POST /kibana_sample_data_types/_doc
POST /opensearch_dashboards_sample_data_types/_doc
{
"type_boolean": false,
"type_byte" : 100,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version='1.0' encoding='utf-8' ?>
<connection-customization class='genericodbc' enabled='true' version='7.8'>
<vendor name='Elasticsearch' />
<vendor name='OpenSearch' />
<driver name='<your_driver_name>' />
<customizations>
<customization name='CAP_CREATE_TEMP_TABLES' value='no' />
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://localhost:9200"]
# The URLs of the OpenSearch instances to use for all your queries.
opensearch.hosts: ["https://localhost:9200"]

elasticsearch.ssl.verificationMode: none
opensearch.ssl.verificationMode: none

server.ssl.enabled: true
server.ssl.certificate: <PATH_TO_YOUR_CERT>/certificate.pem
Expand Down
14 changes: 7 additions & 7 deletions sql-odbc/docs/dev/run_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,28 @@

## Requirements

* Latest version of [Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/docs/install/)
* Latest version of [OpenSearch](https://opendistro.github.io/for-elasticsearch-docs/docs/install/)
* [Required datasets loaded](#set-up-test-datasets)
* [DSN configured](#set-up-dsn)

### Set up test datasets

Loading a dataset requires an [OpenDistro for Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/docs/install/) service running with [Kibana](https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/). If either of these are missing, please refer to the documentation on how to set them up.
Loading a dataset requires an [OpenSearch](https://opendistro.github.io/for-elasticsearch-docs/docs/install/) service running with [OpenSearch Dashboards](https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/). If either of these are missing, please refer to the documentation on how to set them up.

Note, if you wish to work with SSL/TLS, you need to configure ODFE and Kibana to support it. See the [build instructions](./BUILD_INSTRUCTIONS.md) for more info.
Note, if you wish to work with SSL/TLS, you need to configure ODFE and OpenSearch Dashboards to support it. See the [build instructions](./BUILD_INSTRUCTIONS.md) for more info.

First load the sample datasets provided by kibana.
First load the sample datasets provided by OpenSearch Dashboards.

1. Select home (top left corner)
2. Select 'Load a data set and a Kibana dashboard'
2. Select 'Load a data set and a OpenSearch Dashboards dashboard'
3. Select 'Add data' under 'Sample flight data'
4. Select 'Add data' under 'Sample eCommerce orders'
5. Select 'Add data' under 'Sample web logs'

Then load the following custom data sets using the kibana console.
Then load the following custom data sets using the OpenSearch Dashboards console.
Select the wrench on the left control panel. Enter the following commands into the console and hit the play button after each one.

* [kibana_sample_data_types](./datasets/kibana_sample_data_types.md)
* [opensearch_dashboards_sample_data_types](datasets/opensearch_dashboards_sample_data_types.md)

### Set up DSN

Expand Down
6 changes: 3 additions & 3 deletions sql-odbc/docs/dev/sequence_diagrams/SQLDriverConnect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ ESConn->libes (C interface):ESConn object
libes (C interface)->ESConn:DB Parameters
ESConn->libes (C interface):DB Parameters Validation Result
libes (C interface)->ESConn:Connect to DB
database ElasticSearch
ESConn->ElasticSearch:Get
ElasticSearch->ESConn:data
database OpenSearch
ESConn->OpenSearch:Get
OpenSearch->ESConn:data
ESConn->libes (C interface):DB Connection Result
libes (C interface)->psqlodbc Driver:ESConn object with DB Connection
psqlodbc Driver->ODBCTest:SQL Code Result
6 changes: 3 additions & 3 deletions sql-odbc/docs/dev/sequence_diagrams/SQLExecDirect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ psqlodbc Driver->libes (C interface):Query
libes (C interface)->libes (C interface):Active Connection
libes (C interface)->ESConn:Connection, Query
ESConn->ESConn:Connection, Query validation
database ElasticSearch
ESConn->ElasticSearch:Post
ElasticSearch->ESConn:Data
database OpenSearch
ESConn->OpenSearch:Post
OpenSearch->ESConn:Data
ESConn->libes (C interface):Data
libes (C interface)->psqlodbc Driver:Data
psqlodbc Driver->ODBCTest:Query Execution Status
Loading

0 comments on commit 13c022a

Please sign in to comment.