Skip to content
This repository has been archived by the owner on Oct 7, 2022. It is now read-only.

Rename opendistro in strings, APIs and settings #26

Merged
merged 12 commits into from
May 24, 2021
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/dashboards-notebooks-release-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ on:

env:
PLUGIN_NAME: notebooksDashboards
OPENSEARCH_VERSION: 1.0.0-beta1
OPENSEARCH_PLUGIN_VERSION: 1.0.0.0-beta1
OPENSEARCH_DASHBOARDS_VERSION: 1.x
OPENSEARCH_PLUGIN_VERSION: 1.0.0.0-rc1

jobs:

Expand All @@ -33,7 +33,7 @@ jobs:
uses: actions/checkout@v1
with:
repository: opensearch-project/Opensearch-Dashboards
ref: ${{ env.OPENSEARCH_VERSION }}
ref: ${{ env.OPENSEARCH_DASHBOARDS_VERSION }}
path: dashboards-notebooks/OpenSearch-Dashboards

- name: Setup Node
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ on: [pull_request, push]

env:
PLUGIN_NAME: notebooksDashboards
OPENSEARCH_VERSION: 1.0.0-beta1
OPENSEARCH_PLUGIN_VERSION: 1.0.0.0-beta1
OPENSEARCH_DASHBOARDS_VERSION: 1.x
OPENSEARCH_PLUGIN_VERSION: 1.0.0.0-rc1

jobs:

Expand All @@ -22,7 +22,7 @@ jobs:
uses: actions/checkout@v1
with:
repository: opensearch-project/Opensearch-Dashboards
ref: ${{ env.OPENSEARCH_VERSION }}
ref: ${{ env.OPENSEARCH_DASHBOARDS_VERSION }}
path: dashboards-notebooks/OpenSearch-Dashboards

- name: Setup Node
Expand Down
8 changes: 7 additions & 1 deletion .github/workflows/opensearch-notebooks-release-workflow.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,16 @@
name: Release OpenSearch Notebooks Backend Plugin Artifacts
# This workflow is triggered on creating tags to master or an opendistro release branch
# This workflow is triggered on creating tags to main or an opensearch release branch
on:
push:
tags:
- "v*"

env:
PLUGIN_NAME: notebooksDashboards
OPENSEARCH_VERSION: 1.x
COMMON_UTILS_VERSION: main
OPENSEARCH_PLUGIN_VERSION: 1.0.0.0-rc1

jobs:
build:
runs-on: ubuntu-latest
Expand Down
16 changes: 11 additions & 5 deletions .github/workflows/opensearch-notebooks-test-and-build-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@ name: Test and Build OpenSearch Notebooks Backend Plugin

on: [push, pull_request]

env:
PLUGIN_NAME: notebooksDashboards
OPENSEARCH_VERSION: 1.x
COMMON_UTILS_VERSION: main
OPENSEARCH_PLUGIN_VERSION: 1.0.0.0-rc1

jobs:
build:
runs-on: ubuntu-latest
Expand All @@ -20,26 +26,26 @@ jobs:
with:
repository: 'opensearch-project/OpenSearch'
path: OpenSearch
ref: '1.0.0-beta1'
ref: ${{ env.OPENSEARCH_VERSION }}
- name: Build OpenSearch
working-directory: ./OpenSearch
run: ./gradlew publishToMavenLocal -Dbuild.version_qualifier=beta1 -Dbuild.snapshot=false
run: ./gradlew publishToMavenLocal -Dbuild.version_qualifier=rc1 -Dbuild.snapshot=false

# dependencies: common-utils
- name: Checkout common-utils
uses: actions/checkout@v2
with:
ref: '1.0.0-beta1'
ref: ${{ env.COMMON_UTILS_VERSION }}
repository: 'opensearch-project/common-utils'
path: common-utils
- name: Build common-utils
working-directory: ./common-utils
run: ./gradlew publishToMavenLocal -Dopensearch.version=1.0.0-beta1
run: ./gradlew publishToMavenLocal -Dopensearch.version=1.0.0-rc1

- name: Build with Gradle
run: |
cd opensearch-notebooks
./gradlew build -Dopensearch.version=1.0.0-beta1
./gradlew build -Dopensearch.version=1.0.0-rc1

- name: Create Artifact Path
run: |
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ We welcome you to get involved in development, documentation, testing the OpenSe

## Code of Conduct

This project has adopted an [Open Source Code of Conduct](https://opendistro.github.io/for-elasticsearch/codeofconduct.html).
This project has adopted an [Open Source Code of Conduct](https://docs-beta.opensearch.org/docs/opensearch-dashboards/notebooks/).

## Security issue notifications

Expand Down
2 changes: 1 addition & 1 deletion dashboards-notebooks/.cypress/utils/constants.js
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ export const MARKDOWN_TEXT = `%md
#### Code block
* Explain SQL
\`\`\`
POST _opensearch/_sql/_explain
POST _plugins/_sql/_explain
{
"query": "SELECT * FROM my-index LIMIT 50"
}
Expand Down
4 changes: 2 additions & 2 deletions dashboards-notebooks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Dashboards offer a solution for a few selected use cases, and are great tools if

## Documentation

Please see our technical [documentation](https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/notebooks/) to learn more about its features.
Please see our technical [documentation](https://docs-beta.opensearch.org/docs/opensearch-dashboards/notebooks/) to learn more about its features.

## Setup

Expand Down Expand Up @@ -36,7 +36,7 @@ Ultimately, your directory structure should look like this:

To build the plugin's distributable zip simply run `yarn build`.

Example output: `./build/opendistroNotebooks*.zip`
Example output: `./build/dashboardsNotebooks*.zip`


## Run
Expand Down
4 changes: 2 additions & 2 deletions dashboards-notebooks/common/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ export const SELECTED_BACKEND = 'DEFAULT'; // ZEPPELIN || DEFAULT
export const DATE_FORMAT = 'MM/DD/YYYY hh:mm A';
export const FETCH_SIZE = 1000;
export const CREATE_NOTE_MESSAGE = 'Enter a name to describe the purpose of this notebook.';
export const DOCUMENTATION_URL = 'https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/';
export const DOCUMENTATION_URL = 'https://docs-beta.opensearch.org/docs/opensearch-dashboards/notebooks/';

export const zeppelinURL = 'http://localhost:8080';

Expand All @@ -42,7 +42,7 @@ export const wreckOptions = {
headers: { 'Content-Type': 'application/json' },
};

const BASE_NOTEBOOKS_URI = '/_opensearch/_notebooks';
const BASE_NOTEBOOKS_URI = '/_plugins/_notebooks';
export const OPENSEARCH_NOTEBOOKS_API = {
GET_NOTEBOOKS: `${BASE_NOTEBOOKS_URI}/notebooks`,
NOTEBOOK: `${BASE_NOTEBOOKS_URI}/notebook`,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ OpenSearch Dashboards Notebooks enable data-driven, interactive data analytics a
- Zeppelin Backend will provide one stop shop for interpreters, runtime-environments and storage adaptors
- Cons:
- Users will not be free to customize Zeppelin Backend runtime-environment/storage adaptors
- Need to develop a new storage adaptor for Zeppelin to store notebooks as OpenSearch indices [POC details](../../poc/docs/Zeppelin_ODFE_Storage.md)
- Need to develop a new storage adaptor for Zeppelin to store notebooks as OpenSearch indices [POC details](../../poc/docs/Zeppelin_OpenSearch_Storage.md)
- Difficult to maintain releases, as we have to sync version currency/patches to Zeppelin code repository

**4.4.2 Version 2:** In this architecture, Backends are switchable with two options of Default backend (Markdown, Visualization support) or Apache Zeppelin Backend (25+ interpreter support)
Expand Down Expand Up @@ -207,7 +207,7 @@ OpenSearch Dashboards Notebooks enable data-driven, interactive data analytics a
- [Interpreter APIs](http://zeppelin.apache.org/docs/0.9.0-preview1/usage/rest_api/interpreter.html) Get interpreter settings, create/update/restart/delete interpreter setting
- [Notebook APIs](http://zeppelin.apache.org/docs/0.9.0-preview1/usage/rest_api/notebook.html) Create/update/restart/delete noteboooks and paragraphs
- Provides inter-paragraph communication capabilities
- Can be setup to provide OpenSearch and ODFE-SQL interpreters
- Can be setup to provide OpenSearch and OpenSearch-SQL interpreters
- Can be connected with python environment to use ML/plot libraries

### **5.3 Data Model**
Expand Down Expand Up @@ -242,7 +242,7 @@ OpenSearch Dashboards Notebooks enable data-driven, interactive data analytics a

### **7.1** POC: [Embeddable API & Usage](../../poc/docs/OpenSearch Dashboards_Embeddable_Documentation.md)

### **7.2** POC: [Zeppelin ODFE Storage](../../poc/docs/Zeppelin_ODFE_Storage.md)
### **7.2** POC: [Zeppelin OpenSearch Storage](../../poc/docs/Zeppelin_OpenSearch_Storage.md)

### **7.3** Screenshots:

Expand All @@ -263,8 +263,8 @@ OpenSearch Dashboards Notebooks enable data-driven, interactive data analytics a
![OpenSearch service UI](images/opensearch_ss.png)
- **Transmit Data between Interpreters**:

- Use output of an ODFE Query as save as a Zeppelin Context in a variable
![ODFE Query](images/odfe_ss_zepcontext.png)
- Use output of an OpenSearch Query as save as a Zeppelin Context in a variable
![OpenSearch Query](images/opensearch_ss_zepcontext.png)
- Use the Zeppelin Context vairable and import it in python
![Import Context](images/python_ss.png)

Expand Down
26 changes: 13 additions & 13 deletions dashboards-notebooks/docs/dev/Zeppelin_backend_adaptor.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
3. Also, [Notebooks](http://zeppelin.apache.org/docs/0.9.0-preview1/setup/security/notebook_authorization.html) can have access control based on Shiro defined users
3. **Deployment:**
1. Recommended way is to use stand alone docker
2. Create a **custom docker** with new Shiro & Zeppelin configs and set interpreter config for OpenSearch and ODFE-sql.
2. Create a **custom docker** with new Shiro & Zeppelin configs and set interpreter config for OpenSearch and OpenSearch-sql.
3. Sample scripts available in `scripts/docker/spark-cluster-managers`
4. **Storage:**
1. Apache Zeppelin has a pluggable notebook storage mechanism controlled by `zeppelin.notebook.storage` configuration option with multiple implementations.
Expand All @@ -54,10 +54,10 @@
- **[Optional] Setup OpenSearch Interpreter:**

- [Zeppelin OpenSearch interpreter Documentation](https://zeppelin.apache.org/docs/0.9.0-preview2/interpreter/elasticsearch.html)
- This interpreter can be used for ODFE:
- This interpreter can be used for OpenSearch:

- **Note: current issues with OpenSearch Interpreter in Zeppelin**
- User needs to remove ssl flag from the OpenSearch config as Zeppelin doesn’t support ssl request yet: https://issues.apache.org/jira/browse/ZEPPELIN-2031 so run the ODFE service without ssl enabled
- User needs to remove ssl flag from the OpenSearch config as Zeppelin doesn’t support ssl request yet: https://issues.apache.org/jira/browse/ZEPPELIN-2031 so run the OpenSearch service without ssl enabled
- Zeppelin has “no support for ssl” (only uses http) in elastic interpreter:
- [Code](https://github.com/apache/zeppelin/blob/0b8423c62ae52f3716d4bb63d60762fee6910788/elasticsearch/src/main/java/org/apache/zeppelin/elasticsearch/client/HttpBasedClient.java#L105)
- [Apache Issues](https://issues.apache.org/jira/browse/ZEPPELIN-2031)
Expand Down Expand Up @@ -94,32 +94,32 @@ index movies/default/1 {
}
```

- **[Optional] Setup ODFE-SQL JDBC Interpreter:**
- **[Optional] Setup OpenSearch-SQL JDBC Interpreter:**
- [Zeppelin JDBC Interpreter Documentation](https://zeppelin.apache.org/docs/0.9.0-preview2/interpreter/jdbc.html)
- Zeppelin has a generic JDBC interpreter, we can use this to add our ODFE-SQL Driver
- Download [ODFE-SQL Driver](https://opendistro.github.io/for-elasticsearch/downloads.html) Jar file
- Zeppelin has a generic JDBC interpreter, we can use this to add our OpenSearch-SQL Driver
- Download [OpenSearch-SQL Driver](https://opensearch.org/) Jar file
- To Use JDBC interpreter:
- **To add the JDBC interpreter settings for ODFE-SQL:**
- **To add the JDBC interpreter settings for OpenSearch-SQL:**
- Open Zeppelin in browser `localhost:8080`
- Please click the top right menu beside drop down and select interpreter option
- Click on "+ Create" Button
- Add ODFE-SQL interpreter with type JDBC **configure name: “odfesql”**
- Note: The name you assign to the interpreter is used later for accessing paragraphs in notebook - “%odfesql
- Edit config for with ODFE-SQL Driver details (Please refer to the [Github README](https://github.com/opensearch-project/sql/tree/main/sql-jdbc))
- Add OpenSearch-SQL interpreter with type JDBC **configure name: “opensearchsql”**
- Note: The name you assign to the interpreter is used later for accessing paragraphs in notebook - “%opensearchsql
- Edit config for with OpenSearch-SQL Driver details (Please refer to the [Github README](https://github.com/opensearch-project/sql/tree/main/sql-jdbc))
- **If using the default settings, add below mentioned changes in interpreter config:**
- Edit the url: `jdbc:elasticsearch://localhost:9200`
- Edit the driver class: `com.amazon.opendistroforelasticsearch.jdbc.Driver`
- Edit the driver class: `org.opensearch.jdbc.Driver`
- Edit the username to admin
- Edit the password to admin
- Add absolute path to the Jar in the last input box
- You’ll be prompted to restart the interpreter -> Click on ok
- Once configured the screen should look like this:
![SQL Interpreter](images/odfe-zeppelin.png)
![SQL Interpreter](images/opensearch-zeppelin-settings.png)

- Open a notebook and run below commands to check if interpreter settings are set correctly

```
%odfesql
%opensearchsql
SELECT * FROM movies
```

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# **Custom ODFE Storage in Zeppelin**
# **Custom OpenSearch Storage in Zeppelin**

### **Requirement:**

- Use Zeppelin as a backend service for OpenSearch Dashboards Notebooks and store notebooks as ODFE indices
- Use Zeppelin as a backend service for OpenSearch Dashboards Notebooks and store notebooks as OpenSearch indices
- Use Zeppelin’s storage adaptor interface and implement a new storage adaptor using OpenSearch Client

### **Design:**
Expand All @@ -14,7 +14,7 @@
### **Design Details:**

1. Implements the interface common for all Zeppelin Storage adaptors
2. Implementation of functions in ODFE Zeppelin storage adaptor:
2. Implementation of functions in OpenSearch Zeppelin storage adaptor:

- Init - Get all config params
- List - List all notebooks
Expand All @@ -26,9 +26,9 @@

### **Usage:**

1. POC for ODFE adapter is stored in branch 'zeppelin-odfe' of dashboards-notebooks
1. POC for OpenSearch adapter is stored in branch 'zeppelin-opensearch' of dashboards-notebooks
```
git checkout zeppelin-odfe
git checkout zeppelin-opensearch
```
2. Clone Apache Zeppelin and checkout to 'v0.9.0-preview2' branch in a separate folder
```
Expand All @@ -41,22 +41,22 @@ git checkout v0.9.0-preview2
```
git apply /path/to/zeppelin-patch.txt
```
4. Once, in this branch copy "odfe" storage adaptor to your zeppelin files
4. Once, in this branch copy "opensearch" storage adaptor to your zeppelin files
```
cp -r /path/to/dashboards-notebooks/zeppelin/zeppelin-plugins/notebookrepo/odfe path/to/your/zeppelin
cp -r /path/to/dashboards-notebooks/zeppelin/zeppelin-plugins/notebookrepo/opensearch path/to/your/zeppelin
```
4. Add ODFE storage property in zeppelin config file "conf/zeppelin-site.xml" and you should comment default git storage
4. Add OpenSearch storage property in zeppelin config file "conf/zeppelin-site.xml" and you should comment default git storage
```
<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.ODFENotebookRepo</value>
<value>org.apache.zeppelin.notebook.repo.OpenSearchNotebookRepo</value>
<description>versioned notebook persistence layer implementation</description>
</property>

<!--
<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.ODFENotebookRepo</value>
<value>org.apache.zeppelin.notebook.repo.OpenSearchNotebookRepo</value>
<description>versioned notebook persistence layer implementation</description>
</property>
-->
Expand Down
Loading