Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenSearch migration for ODBC driver #7

Merged
merged 37 commits into from
Apr 17, 2021
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
7cb34e3
Rename Elasticsearch
dai-chen Apr 13, 2021
5b2193f
Bump plugin version to 1.15
dai-chen Apr 13, 2021
5c0f431
Rename es_api30
dai-chen Apr 13, 2021
0288ff1
Rename es_apifunc
dai-chen Apr 13, 2021
3bbe24f
Rename es_communication
dai-chen Apr 13, 2021
4213a5d
Rename es_connection
dai-chen Apr 13, 2021
8311ba9
Fix connection.c
dai-chen Apr 13, 2021
e92631b
Rename es_driver_connect and es_semaphore
dai-chen Apr 13, 2021
3c319ac
Rename es_helper
dai-chen Apr 13, 2021
d6920f7
Fix test and rename es_info
dai-chen Apr 13, 2021
9f2f5d3
Rename es_parse_result
dai-chen Apr 13, 2021
4873929
Rename es_statement
dai-chen Apr 13, 2021
e86dd33
Rename es_result_queue
dai-chen Apr 13, 2021
78cff89
Rename es_utility
dai-chen Apr 14, 2021
cc3179c
Rename es_types
dai-chen Apr 14, 2021
5c3f538
Rename es_odbc
dai-chen Apr 14, 2021
7fd0fad
Fix mylog
dai-chen Apr 14, 2021
fa1d5bc
Fix header and rename ESLogLevel
dai-chen Apr 14, 2021
8990a33
Rename es_result
dai-chen Apr 14, 2021
d2586f5
Rename esNAME, esconn
dai-chen Apr 14, 2021
ecd8002
Rename estype_ functions
dai-chen Apr 14, 2021
a14a8aa
Rename eserror
dai-chen Apr 14, 2021
14b4b4d
Rename more variables and functions
dai-chen Apr 14, 2021
c6fff52
Rename more variables and functions
dai-chen Apr 14, 2021
aafd199
Fix error and rename ESAPI_
dai-chen Apr 14, 2021
3a0b07d
Fix error
dai-chen Apr 14, 2021
76eb56a
Fix error
dai-chen Apr 14, 2021
fb33284
Fix error
dai-chen Apr 14, 2021
f1beabc
More renaming
dai-chen Apr 14, 2021
eab79f8
Fix elasticenlist error
dai-chen Apr 14, 2021
4e819ba
Replace Kibana
dai-chen Apr 15, 2021
afe4c3f
Fix links to old repo
dai-chen Apr 16, 2021
ada67ba
Fix plugin name in github action
dai-chen Apr 16, 2021
0127662
Change plugin name to OpenSearch
dai-chen Apr 16, 2021
5d62b4a
Replace all other ODFE and links
dai-chen Apr 16, 2021
5d1f434
Revert release notes
dai-chen Apr 16, 2021
a665ecd
Merge branch 'develop' into opensearch/odbc-build
dai-chen Apr 16, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/sql-odbc-release-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ env:
ODBC_BUILD_PATH: "./build/odbc/build"
AWS_SDK_INSTALL_PATH: "./build/aws-sdk/install"
PLUGIN_NAME: opendistro-sql-odbc
dai-chen marked this conversation as resolved.
Show resolved Hide resolved
OD_VERSION: 1.13.0.0
OD_VERSION: 1.15.0.0

jobs:
build-mac:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/sql-odbc-rename-and-release-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
- rename*

env:
OD_VERSION: 1.13.0.0
OD_VERSION: 1.15.0.0

jobs:
upload-odbc:
Expand Down
4 changes: 2 additions & 2 deletions sql-odbc/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Open Distro for Elasticsearch ODBC Driver

ElasticsearchODBC is a read-only ODBC driver for Windows and Mac for connecting to Open Distro for Elasticsearch SQL support.
OpenSearchODBC is a read-only ODBC driver for Windows and Mac for connecting to Open Distro for Elasticsearch SQL support.

## Specifications

Expand Down Expand Up @@ -60,7 +60,7 @@ A list of options available for configuring driver behaviour is available [here]
To setup a connection, the driver uses an ODBC connection string. Connection strings are semicolon-delimited strings specifying the set of options to use for a connection. Typically, a connection string will either:

1. specify a Data Source Name containing a pre-configured set of options (`DSN=xxx;User=xxx;Password=xxx;`)
2. or configure options explicitly using the string (`Host=xxx;Port=xxx;LogLevel=ES_DEBUG;...`)
2. or configure options explicitly using the string (`Host=xxx;Port=xxx;LogLevel=OPENSEARCH_DEBUG;...`)

## Building from source

Expand Down
4 changes: 2 additions & 2 deletions sql-odbc/docs/dev/BUILD_INSTRUCTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,9 @@ See [run_tests.md](./run_tests.md)

To disable SSL/TLS in the tests, the main CMakeLists.txt file must be edited. This can be found in the project 'src' directory. In the 'General compiler definitions' in the CMakeLists.txt file, USE_SSL is set. Remove this from the add_compile_definitions function to stop SSL/TLS from being used in the tests.

To enable SSL/TLS on Elasticsearch, you must edit the Elasticsearch.yml file, found in the config directory of Elasticsearch. An example Elasticsearch yml file can be found in the dev folder of this project. The certificates specified MUST be in the config directory of the Elasticsearch instance. For more information, please refer to the [Elasticsearch security settings documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html).
To enable SSL/TLS on OpenSearch, you must edit the OpenSearch.yml file, found in the config directory of OpenSearch. An example OpenSearch yml file can be found in the dev folder of this project. The certificates specified MUST be in the config directory of the OpenSearch instance. For more information, please refer to the [OpenSearch security settings documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html).

If you plan to use Kibana, as suggested for this project, you must also edit the Kibana settings. Notice, when specifying a certificate for Kibana, you do not need to place it in the Kibana config directory, but instead must provide the absolute path to it. An example Kibana.yml file can be found in the dev folder of this project. For more information, please refer to the [Kibana settings documentation](https://www.elastic.co/guide/en/kibana/current/settings.html).
If you plan to use OpenSearch Dashboards, as suggested for this project, you must also edit the OpenSearch Dashboards settings. Notice, when specifying a certificate for OpenSearch Dashboards, you do not need to place it in the OpenSearch Dashboards config directory, but instead must provide the absolute path to it. An example opensearch_dashboards.yml file can be found in the dev folder of this project. For more information, please refer to the [OpenSearch Dashboards settings documentation](https://www.elastic.co/guide/en/kibana/current/settings.html).

### Setting up a DSN

Expand Down
4 changes: 2 additions & 2 deletions sql-odbc/docs/dev/Pagination.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Opendistro ODBC Driver Cursor (Pagination) Support Design Documentation
dai-chen marked this conversation as resolved.
Show resolved Hide resolved

## Overview
Elasticsearch ODBC Driver supports forward-only cursor. This document illustrates how the cursor(pagination) is handled in the driver.
OpenSearch ODBC Driver supports forward-only cursor. This document illustrates how the cursor(pagination) is handled in the driver.

For information on how the pagination is supported on Elasticsearch server, check [Opendistro SQL Cursor (Pagination) Support](https://github.com/opendistro-for-elasticsearch/sql/blob/master/docs/dev/Pagination.md).
For information on how the pagination is supported on OpenSearch server, check [Opendistro SQL Cursor (Pagination) Support](https://github.com/opendistro-for-elasticsearch/sql/blob/master/docs/dev/Pagination.md).
dai-chen marked this conversation as resolved.
Show resolved Hide resolved

## Data Flow
<p align="center">
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# kibana_sample_data_types
# opensearch_dashboards_sample_data_types

## Mapping

```json
PUT /kibana_sample_data_types
PUT /opensearch_dashboards_sample_data_types
{
"mappings": {
"properties": {
Expand Down Expand Up @@ -32,7 +32,7 @@ PUT /kibana_sample_data_types
## Data

```json
POST /kibana_sample_data_types/_doc
POST /opensearch_dashboards_sample_data_types/_doc
{
"type_boolean": true,
"type_byte" : -120,
Expand All @@ -52,7 +52,7 @@ POST /kibana_sample_data_types/_doc
```

```json
POST /kibana_sample_data_types/_doc
POST /opensearch_dashboards_sample_data_types/_doc
{
"type_boolean": false,
"type_byte" : 100,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version='1.0' encoding='utf-8' ?>
<connection-customization class='genericodbc' enabled='true' version='7.8'>
<vendor name='Elasticsearch' />
<vendor name='OpenSearch' />
<driver name='<your_driver_name>' />
<customizations>
<customization name='CAP_CREATE_TEMP_TABLES' value='no' />
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://localhost:9200"]
# The URLs of the OpenSearch instances to use for all your queries.
opensearch.hosts: ["https://localhost:9200"]

elasticsearch.ssl.verificationMode: none
opensearch.ssl.verificationMode: none

server.ssl.enabled: true
server.ssl.certificate: <PATH_TO_YOUR_CERT>/certificate.pem
Expand Down
12 changes: 6 additions & 6 deletions sql-odbc/docs/dev/run_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,22 @@

### Set up test datasets

Loading a dataset requires an [OpenDistro for Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/docs/install/) service running with [Kibana](https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/). If either of these are missing, please refer to the documentation on how to set them up.
Loading a dataset requires an [OpenDistro for Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/docs/install/) service running with [OpenSearch Dashboards](https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/). If either of these are missing, please refer to the documentation on how to set them up.

Note, if you wish to work with SSL/TLS, you need to configure ODFE and Kibana to support it. See the [build instructions](./BUILD_INSTRUCTIONS.md) for more info.
Note, if you wish to work with SSL/TLS, you need to configure ODFE and OpenSearch Dashboards to support it. See the [build instructions](./BUILD_INSTRUCTIONS.md) for more info.

First load the sample datasets provided by kibana.
First load the sample datasets provided by OpenSearch Dashboards.

1. Select home (top left corner)
2. Select 'Load a data set and a Kibana dashboard'
2. Select 'Load a data set and a OpenSearch Dashboards dashboard'
3. Select 'Add data' under 'Sample flight data'
4. Select 'Add data' under 'Sample eCommerce orders'
5. Select 'Add data' under 'Sample web logs'

Then load the following custom data sets using the kibana console.
Then load the following custom data sets using the OpenSearch Dashboards console.
Select the wrench on the left control panel. Enter the following commands into the console and hit the play button after each one.

* [kibana_sample_data_types](./datasets/kibana_sample_data_types.md)
* [opensearch_dashboards_sample_data_types](datasets/opensearch_dashboards_sample_data_types.md)

### Set up DSN

Expand Down
6 changes: 3 additions & 3 deletions sql-odbc/docs/dev/sequence_diagrams/SQLDriverConnect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ ESConn->libes (C interface):ESConn object
libes (C interface)->ESConn:DB Parameters
ESConn->libes (C interface):DB Parameters Validation Result
libes (C interface)->ESConn:Connect to DB
database ElasticSearch
ESConn->ElasticSearch:Get
ElasticSearch->ESConn:data
database OpenSearch
ESConn->OpenSearch:Get
OpenSearch->ESConn:data
ESConn->libes (C interface):DB Connection Result
libes (C interface)->psqlodbc Driver:ESConn object with DB Connection
psqlodbc Driver->ODBCTest:SQL Code Result
6 changes: 3 additions & 3 deletions sql-odbc/docs/dev/sequence_diagrams/SQLExecDirect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ psqlodbc Driver->libes (C interface):Query
libes (C interface)->libes (C interface):Active Connection
libes (C interface)->ESConn:Connection, Query
ESConn->ESConn:Connection, Query validation
database ElasticSearch
ESConn->ElasticSearch:Post
ElasticSearch->ESConn:Data
database OpenSearch
ESConn->OpenSearch:Post
OpenSearch->ESConn:Data
ESConn->libes (C interface):Data
libes (C interface)->psqlodbc Driver:Data
psqlodbc Driver->ODBCTest:Query Execution Status
2 changes: 1 addition & 1 deletion sql-odbc/docs/test/excel_connection.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

## Microsoft Excel Connectivity

There are multiple ways to load data from Elasticsearch in Microsoft Excel.
There are multiple ways to load data from OpenSearch in Microsoft Excel.
* [ODBC as Data Source](odbc_data_source_connection.md)
* [Microsoft Query](microsoft_query_connection.md)
* [Query Wizard](query_wizard_connection.md)
Expand Down
2 changes: 1 addition & 1 deletion sql-odbc/docs/test/exporting_csv_files.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

* Load data from Open Distro For Elasticsearch by any using any [connection mode](excel_connection.md).

For this example, [ODBC](odbc_data_source_connection.md) connection with SQL statement `SELECT * FROM kibana_sample_data_flights LIMIT 50` in advanced options is used.
For this example, [ODBC](odbc_data_source_connection.md) connection with SQL statement `SELECT * FROM opensearch_dashboards_sample_data_flights LIMIT 50` in advanced options is used.

<img src="img/export_loaded_data.png">

Expand Down
2 changes: 1 addition & 1 deletion sql-odbc/docs/test/query_wizard_connection.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

<img src="img/query_wizard_select_tables.png" width="500">

* Specify conditions to apply filter if needed. Ensure selected operations are supported by Elasticsearch. Click on **Next**.
* Specify conditions to apply filter if needed. Ensure selected operations are supported by OpenSearch. Click on **Next**.

<img src="img/query_wizard_filter_data.png" width="500">

Expand Down
2 changes: 1 addition & 1 deletion sql-odbc/docs/test/refresh_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ For this example, [ODBC](odbc_data_source_connection.md) connection is used with

<img src="img/refresh_load_data.png">

* Add a datarow in the table `sample_data_for_excel` using kibana.
* Add a datarow in the table `sample_data_for_excel` using OpenSearch Dashboards.

<img src="img/refresh_add_datarow.png" width=450>

Expand Down
2 changes: 1 addition & 1 deletion sql-odbc/docs/user/configuration_options.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@

| Option | Description | Type | Default |
|--------|-------------|------|---------------|
| `LogLevel` | Severity level for driver logs. | one of `ES_OFF`, `ES_FATAL`, `ES_ERROR`, `ES_INFO`, `ES_DEBUG`, `ES_TRACE`, `ES_ALL` | `ES_WARNING` |
| `LogLevel` | Severity level for driver logs. | one of `OPENSEARCH_OFF`, `OPENSEARCH_FATAL`, `OPENSEARCH_ERROR`, `OPENSEARCH_INFO`, `OPENSEARCH_DEBUG`, `OPENSEARCH_TRACE`, `OPENSEARCH_ALL` | `OPENSEARCH_WARNING` |
| `LogOutput` | Location for storing driver logs. | string | WIN: `C:\`, MAC: `/tmp` |

**NOTE:** Administrative privileges are required to change the value of logging options on Windows.
2 changes: 1 addition & 1 deletion sql-odbc/docs/user/microsoft_excel_support.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ In this case, please use advanced options while connecting to the data source.

<img src="img/excel_advanced_option.png" width=500>

Also, make sure query doesn't include the name of nested or object type column name. For example, `SELECT products FROM kibana_sample_data_ecommerce` where product is nested type column might have data loss.
Also, make sure query doesn't include the name of nested or object type column name. For example, `SELECT products FROM opensearch_dashboards_sample_data_ecommerce` where product is nested type column might have data loss.

This issue will be resolved when [Object field is missing in SELECT result](https://github.com/opendistro-for-elasticsearch/sql/issues/564) is fixed.

2 changes: 1 addition & 1 deletion sql-odbc/docs/user/power_bi_service_support.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ For **AWS_SIGV4**, select **Key** and set aws access credentials for user **PBIE

## Publish Report

* Follow [instructions](./power_bi_support.md) to create graph using Open Disto For Elasticsearch Data connector.
* Follow [instructions](./power_bi_support.md) to create graph using Open Distro For Elasticsearch Data connector.
* Click on **Publish** to publish the report on Power BI service.

<img src="img/pbi_publish_report.png">
Expand Down
8 changes: 4 additions & 4 deletions sql-odbc/src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,11 @@ set(INSTALL_SRC "${CMAKE_CURRENT_SOURCE_DIR}/installer")
set(DSN_INSTALLER_SRC "${CMAKE_CURRENT_SOURCE_DIR}/DSNInstaller")

# ODBC Driver version
set(DRIVER_PACKAGE_VERSION "1.13.0.0")
set(DRIVER_PACKAGE_VERSION_COMMA_SEPARATED "1,13,0,0")
add_compile_definitions( ES_ODBC_VERSION="${DRIVER_PACKAGE_VERSION}"
set(DRIVER_PACKAGE_VERSION "1.15.0.0")
set(DRIVER_PACKAGE_VERSION_COMMA_SEPARATED "1,15,0,0")
add_compile_definitions( OPENSEARCH_ODBC_VERSION="${DRIVER_PACKAGE_VERSION}"
# Comma separated version is required for odbc administrator's driver file.
ES_ODBC_DRVFILE_VERSION=${DRIVER_PACKAGE_VERSION_COMMA_SEPARATED} )
OPENSEARCH_ODBC_DRVFILE_VERSION=${DRIVER_PACKAGE_VERSION_COMMA_SEPARATED} )

# Extensions of base directories
set(PERFORMANCE_TESTS "${CMAKE_CURRENT_SOURCE_DIR}/PerformanceTests")
Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI\ODBC Drivers]
"ElasticsearchODBC"="Installed"
"OpenSearchODBC"="Installed"

[HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI\ElasticsearchODBC]
[HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBCINST.INI\OpenSearchODBC]
"Driver"="<project-root>\\bin64\\Release\\odfesqlodbc.dll"
"Setup"="<project-root>\\bin64\\Release\\odfesqlodbc.dll"

[HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI\ODBC Data Sources]
"test_aws_auth_dsn"="ElasticsearchODBC"
"test_aws_auth_dsn"="OpenSearchODBC"

[HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI\test_aws_auth_dsn]
"host"="https://search-sept-cdg-david-test-domain-gouok3seqeupz64smuvfxyddui.eu-west-3.es.amazonaws.com"
Expand Down
2 changes: 1 addition & 1 deletion sql-odbc/src/IntegrationTests/ITODBCAwsAuth/test_odbc.ini
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[ODBC Data Sources]
test_aws_auth_dsn = ElasticsearchODBC
test_aws_auth_dsn = OpenSearchODBC

[test_aws_auth_dsn]
host = https://search-sept-cdg-david-test-domain-gouok3seqeupz64smuvfxyddui.eu-west-3.es.amazonaws.com
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,19 @@

std::wstring dsn_name = L"test_aws_auth_dsn";
std::wstring aws_auth_conn_string =
L"Driver={Elasticsearch};DataBase=database_name;"
L"Driver={OpenSearch};DataBase=database_name;"
L"Host=https://"
L"search-bit-quill-cx3hpfoxvasohujxkllmgjwqde.us-west-2."
L"es.amazonaws.com;"
L"Auth=AWS_SIGV4;Region=us-west-2;LogLevel=1";
std::wstring aws_auth_conn_string_invalid_region =
L"Driver={Elasticsearch};DataBase=database_name;"
L"Driver={OpenSearch};DataBase=database_name;"
L"Host=https://"
L"search-bit-quill-cx3hpfoxvasohujxkllmgjwqde.us-west-2."
L"es.amazonaws.com;"
L"Auth=AWS_SIGV4;Region=us-west-3;LogLevel=1";
std::wstring aws_auth_conn_string_invalid_authtype =
L"Driver={Elasticsearch};DataBase=database_name;"
L"Driver={OpenSearch};DataBase=database_name;"
L"Host=https://"
L"search-bit-quill-cx3hpfoxvasohujxkllmgjwqde.us-west-2."
L"es.amazonaws.com;"
Expand Down
4 changes: 2 additions & 2 deletions sql-odbc/src/IntegrationTests/ITODBCAwsAuth/test_odbcinst.ini
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[ODBC Drivers]
ElasticsearchODBC = Installed
OpenSearchODBC = Installed

[ElasticsearchODBC]
[OpenSearchODBC]
Driver = <project-root>/lib64/libodfesqlodbc.dylib
Setup = <project-root>/lib64/libodfesqlodbc.dylib
Loading