Skip to content

Commit

Permalink
Finish drafting content and send for tech review
Browse files Browse the repository at this point in the history
Signed-off-by: Melissa Vagi <[email protected]>
  • Loading branch information
vagimeli committed Oct 13, 2023
1 parent 225092d commit bf2ac7b
Show file tree
Hide file tree
Showing 15 changed files with 131 additions and 23 deletions.
15 changes: 7 additions & 8 deletions _dashboards/management/S3-data-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Connecting Amazon S3 to OpenSearch
parent: Data sources
nav_order: 15
has_children: false
has_children: true
---

# Connecting Amazon S3 to OpenSearch
Expand Down Expand Up @@ -39,7 +39,6 @@ To connect your Amazon S3 data source, follow these steps:
4. Select the **Review Configuration** button and verify the details.
5. Select the **Connect to Amazon S3** button.


## Manage your Amazon S3 data source

Once you've connected your Amazon S3 data source, you can explore that data through the **Manage data sources** tab. The following steps guide you through using this functionality:
Expand All @@ -49,19 +48,19 @@ Once you've connected your Amazon S3 data source, you can explore that data thro

![Manage data sources UI]({{site.url}}{{site.baseurl}}/images/dashboards/manage-data-source-UI.png)


3. (Optional) Explore the Amazon S3 use cases. Go to **Next steps** to learn more about each use case.

## Limitations

This feature is still under development, so there are some limitations:
This feature is still under development, including the data integration functionality. The following are some limitations:

- <SME: What are the limitations?>
- <SME: What is the GitHub link where users can leave feedback?>

## Next steps

- [Optimize query performance of your external data sources](), such as Amazon S3, through Query Workbench.
- [Query your data in Data Explorer]
- Learn about the [Amazon S3 and AWS Glue connector](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/s3glue_connector.rst), including configuration and queries.
- Learn about [Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) through OpenSearch Dashboards.
- Learn about [querying your data in Data Explorer]() through OpenSearch Dashboards.
- Learn about ways to [optimize query performance of your external data sources](), such as Amazon S3, through Query Workbench.
- Learn about the [Amazon S3 and AWS Glue connector](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/s3glue_connector.rst), including configuration settings and query examples.
- Learn about [managing your indexes]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) through OpenSearch Dashboards.
- <SME: What other links do we need to include?>
62 changes: 58 additions & 4 deletions _dashboards/management/accelerate-external-data.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,68 @@
---
layout: default
title: Optimize query performance of external data sources
parent: Data sources
nav_order: 30
title: Optimize query performance using an acceleration index
parent: Connecting Amazon S3 to OpenSearch
grand_parent: Data sources
nav_order: 15
has_children: false
---

# Optimize query performance of external data sources
# Optimize query performance using an acceleration index
Introduced 2.11
{: .label .label-purple }

Query performance can be slow when using external data sources for reasons such as network latency, data transformation, and data volume. You can optimize your query performance by using an acceleration index.

To get started with the **Accelerate performance** use case available under **Data sources**, follow these steps:

1. Go to **OpenSearch Dashboards** > **Query Workbench** and select your Amazon S3 data source from the **Data Sources** dropdown menu in the upper-left corner.
2. From the left-side navigation menu, select a database. An example using the `http_logs` database is shown in the following image:

![Query Workbench accelerate data UI]({{site.url}}{{site.baseurl}}/images/dashboards/query-workbench-accelerate-data.png)

3. View the results in the table.

To create an acceleration index, follow these steps:

1. Select the **Accelerate data** button. A pop-up window appears.
2. Enter details in the **Select data fields**. In the **Database** field, you will select the desired acceleration index, **Skipping index** or **Covering index**. An example is shown in the following image:

![Accelerate data pop-up window]({{site.url}}{{site.baseurl}}/images/dashboards/accelerate-data-popup.png)

3. Under **Index settings**, enter the details for your acceleration index. For information about naming, select **Help**. Note that an Amazon S3 table can only have one skipping index at a time. An example is shown in the following image:

![Skipping index settings]({{site.url}}{{site.baseurl}}/images/dashboards/skipping-index-settings.png)

### Define skipping index settings

1. Under **Skipping index definition**, select the **Add fields** button to define the skipping index acceleration method and choose the fields you want to add. An example is shown in the following image:

![Skipping index add fields]({{site.url}}{{site.baseurl}}/images/dashboards/add-fields-skipping-index.png)

2. Select the **Copy Query to Editor** button to apply your skipping index settings.
3. View the covering index query details in the table pane and then select the **Run** button. Your index is added the left-side navigation menu containing the list of your databases. An example is shown in the following image:



### Define cover index settings

1. Under **Index settings**, enter a valid index name. Note that each Amazon S3 table can have multiple covering indexes. An example is shown in the following image:

![Covering index settings]({{site.url}}{{site.baseurl}}/images/dashboards/covering-index-naming.png)


2. Once you have added the index name, define the covering index fields by selecting `(add fields here)` under **Covering index definition**. An example is shown in the following image:

![Covering index field naming]({{site.url}}{{site.baseurl}}/images/dashboards/covering-index-fields.png)

3. Select the **Copy Query to Editor** button to apply your covering index settings.
4. View the covering index query details in the table pane and then select the **Run** button. Your index is added the left-side navigation menu containing the list of your databases.

![Run index in Query Workbench]({{site.url}}{{site.baseurl}}/images/dashboards/run-index-query-workbench.png)

## Limitations

This feature is still under development, so there are some limitations:

-
-
15 changes: 4 additions & 11 deletions _dashboards/management/data-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,21 +64,14 @@ To make changes to **Connection Details**, edit one or both of the **Title** and
To delete the data source connection, select the delete icon ({::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/dashboards/trash-can-icon.png" class="inline-icon" alt="delete icon"/>{:/}).

## Create an index pattern
Once you've created a data source connection, you can create an index pattern for the data source. An index pattern is a template that OpenSearch uses to create indexes for data from the data source.

Learn how to load your own data and create an index pattern in the following steps. This tutorial uses the preconfigured index pattern `opensearch_dashboards_sample_data_ecommerce Default`.

1. In the Dashboards console, select **Index Patterns** > **Create index pattern**.
2. Select **Use external data source connection**.
3. Start typing in the **Search data sources** field to search for the data source you want to connect. Select the data source and **Next step**.
4. Add an **Index pattern name** to define the index pattern and then choose **Next step**.
5. Choose an option for the **Time** field and then select **Create index pattern**.

Now you can start indexing data from the data source.
Once you've created a data source connection, you can create an index pattern for the data source. An _index pattern_ is a template that OpenSearch uses to create indexes for data from the data source. See [Index patterns]() for more information and a tutorial.

## Next steps

- Learn about [indexing data using Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) in OpenSearch Dashboards.
- Learn about [managing index patterns]() through OpenSearch Dashboards.
- Learn about [indexing data using Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) through OpenSearch Dashboards.
- Learn about how to connect [multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/).
- Learn about how to [connect OpenSearch and Amazon S3 through OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/).
- Learn about the [Integrations]({{site.url}}{{site.baseurl}}/integrations/index/) tool that gives you flexibility to use various data ingestion methods and connect data from the Dashboards UI.

62 changes: 62 additions & 0 deletions _dashboards/management/query-data-source.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
layout: default
title: Query and visualize Amazon S3 data
parent: Connecting Amazon S3 to OpenSearch
grand_parent: Data sources
nav_order: 10
has_children: false
---

# Query and visualize Amazon S3 data
Introduced 2.11
{: .label .label-purple }

This tutorial guides you through querying and visualizing your Amazon S3 data using OpenSearch Dashboards. This tool uses **Data Explorer** or **Observability Logs** to query your data.

## Get started with querying

To get started, follow these steps:

1. From the **Manage data sources** page, select your data source from the list.
2. From the data source's detail page, select the **Query data** card. This option takes you to the **Observability** > **Logs** page, which is shown in the following image:

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/observability-logs-UI.png" alt="Observability Logs UI" width="700">

3. Select the **Event Explorer** button. This option creates and saves frequently searched queries and visualizations using [Piped Processing Language (PPL)]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) or [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/), which connects to Spark SQL.
4. Select the Amazon S3 data source from the dropdown menu in the upper-left corner. An example is shown in the following image:

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/query-data-sources-UI-2.png" alt="Observability Logs Amazon S3 dropdown menu" width="700">

5. Enter the query in the **Enter PPL query** field. Note that the default language is SQL. To change the language, select PPL from the dropdown menu.
6. Select the **Search** button. The **Query Processing** message is shown, confirming that your query is being processed.
7. View the results, which are listed in a table under the **Events** tab. From this page, details such as available fields, source, and time are shown in a table format.
8. (Optional) Create data visualizations.

## Create visualizations of your Amazon S3 data

To create visualizations, follow these steps:

1. From the Explorer page, select the **Visualizations** tab. An example is shown in the following image:

img src="{{site.url}}{{site.baseurl}}/images/dashboards/explorer-S3viz-UI.png" alt="Explorer Amazon S3 visualizations UI" width="700">

2. Select **Index data to visualize**. This option currently only creates [acceleration indexes]({{site.url}}{{site.baseurl}}/dashboards/management/accelerate-external-data/), which give you views of the data visualizations from within the **Visualizations** tab. To create a visualization of your Amazon S3 data, go to **Discover**. See the [**Discover** documentation]({{site.url}}{{site.baseurl}}/dashboards/discover/index-discover/) for information and a tutorial. <SME: How does the user create visualizations of their S3 data? Through Discover?>

## Use Query Workbench with your Amazon S3 data source

[Query Workbench]({{site.url}}{{site.baseurl}}/search-plugins/sql/workbench/) runs on-demand SQL queries, translates SQL into its REST equivalent, and views and saves results as text, JSON, JDBC, or CSV.

To use Query Workbench with your Amazon S3 data, follow these steps:

1. From the OpenSearch Dashboards main menu, select **OpenSearch Plugins** > **Query Workbench**.
2. From the **Data Sources** dropdown menu in the upper-left corner, choose your Amazon S3 data source. Your data begins loading the databases that are part of your data source. An example is shown in the following image:

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/query-workbench-S3.png" alt="Query Workbench Amazon S3 data loading UI" width="700">

3. View the databases listed in the left-side navigation menu and select a database to view its details. Any information about acceleration indexes is listed under **Acceleration index destination**.
4. Choose the **Describe Index** button to learn more about how data is stored in that particular index.
5. Choose the **Drop index** button to delete and clear both the OpenSearch index and the Amazon S3 Spark job that refreshes the data.

## Nest steps

- Learn about [accelerating query performance of your external data sources]({{site.url}}{{site.baseurl}}/dashboards/management/accelerate-external-data/).
Binary file added images/dashboards/accelerate-data-popup.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/add-fields-skipping-index.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/covering-index-fields.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/covering-index-naming.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/explorer-S3viz-UI.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/observability-logs-UI.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/query-data-sources-UI-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/query-workbench-S3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/run-index-query-workbench.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/dashboards/skipping-index-settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit bf2ac7b

Please sign in to comment.