-
Notifications
You must be signed in to change notification settings - Fork 504
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Finish drafting content and send for tech review
Signed-off-by: Melissa Vagi <[email protected]>
- Loading branch information
Showing
15 changed files
with
131 additions
and
23 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,14 +1,68 @@ | ||
--- | ||
layout: default | ||
title: Optimize query performance of external data sources | ||
parent: Data sources | ||
nav_order: 30 | ||
title: Optimize query performance using an acceleration index | ||
parent: Connecting Amazon S3 to OpenSearch | ||
grand_parent: Data sources | ||
nav_order: 15 | ||
has_children: false | ||
--- | ||
|
||
# Optimize query performance of external data sources | ||
# Optimize query performance using an acceleration index | ||
Introduced 2.11 | ||
{: .label .label-purple } | ||
|
||
Query performance can be slow when using external data sources for reasons such as network latency, data transformation, and data volume. You can optimize your query performance by using an acceleration index. | ||
|
||
To get started with the **Accelerate performance** use case available under **Data sources**, follow these steps: | ||
|
||
1. Go to **OpenSearch Dashboards** > **Query Workbench** and select your Amazon S3 data source from the **Data Sources** dropdown menu in the upper-left corner. | ||
2. From the left-side navigation menu, select a database. An example using the `http_logs` database is shown in the following image: | ||
|
||
![Query Workbench accelerate data UI]({{site.url}}{{site.baseurl}}/images/dashboards/query-workbench-accelerate-data.png) | ||
|
||
3. View the results in the table. | ||
|
||
To create an acceleration index, follow these steps: | ||
|
||
1. Select the **Accelerate data** button. A pop-up window appears. | ||
2. Enter details in the **Select data fields**. In the **Database** field, you will select the desired acceleration index, **Skipping index** or **Covering index**. An example is shown in the following image: | ||
|
||
![Accelerate data pop-up window]({{site.url}}{{site.baseurl}}/images/dashboards/accelerate-data-popup.png) | ||
|
||
3. Under **Index settings**, enter the details for your acceleration index. For information about naming, select **Help**. Note that an Amazon S3 table can only have one skipping index at a time. An example is shown in the following image: | ||
|
||
![Skipping index settings]({{site.url}}{{site.baseurl}}/images/dashboards/skipping-index-settings.png) | ||
|
||
### Define skipping index settings | ||
|
||
1. Under **Skipping index definition**, select the **Add fields** button to define the skipping index acceleration method and choose the fields you want to add. An example is shown in the following image: | ||
|
||
![Skipping index add fields]({{site.url}}{{site.baseurl}}/images/dashboards/add-fields-skipping-index.png) | ||
|
||
2. Select the **Copy Query to Editor** button to apply your skipping index settings. | ||
3. View the covering index query details in the table pane and then select the **Run** button. Your index is added the left-side navigation menu containing the list of your databases. An example is shown in the following image: | ||
|
||
|
||
|
||
### Define cover index settings | ||
|
||
1. Under **Index settings**, enter a valid index name. Note that each Amazon S3 table can have multiple covering indexes. An example is shown in the following image: | ||
|
||
![Covering index settings]({{site.url}}{{site.baseurl}}/images/dashboards/covering-index-naming.png) | ||
|
||
|
||
2. Once you have added the index name, define the covering index fields by selecting `(add fields here)` under **Covering index definition**. An example is shown in the following image: | ||
|
||
![Covering index field naming]({{site.url}}{{site.baseurl}}/images/dashboards/covering-index-fields.png) | ||
|
||
3. Select the **Copy Query to Editor** button to apply your covering index settings. | ||
4. View the covering index query details in the table pane and then select the **Run** button. Your index is added the left-side navigation menu containing the list of your databases. | ||
|
||
![Run index in Query Workbench]({{site.url}}{{site.baseurl}}/images/dashboards/run-index-query-workbench.png) | ||
|
||
## Limitations | ||
|
||
This feature is still under development, so there are some limitations: | ||
|
||
- | ||
- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
--- | ||
layout: default | ||
title: Query and visualize Amazon S3 data | ||
parent: Connecting Amazon S3 to OpenSearch | ||
grand_parent: Data sources | ||
nav_order: 10 | ||
has_children: false | ||
--- | ||
|
||
# Query and visualize Amazon S3 data | ||
Introduced 2.11 | ||
{: .label .label-purple } | ||
|
||
This tutorial guides you through querying and visualizing your Amazon S3 data using OpenSearch Dashboards. This tool uses **Data Explorer** or **Observability Logs** to query your data. | ||
|
||
## Get started with querying | ||
|
||
To get started, follow these steps: | ||
|
||
1. From the **Manage data sources** page, select your data source from the list. | ||
2. From the data source's detail page, select the **Query data** card. This option takes you to the **Observability** > **Logs** page, which is shown in the following image: | ||
|
||
<img src="{{site.url}}{{site.baseurl}}/images/dashboards/observability-logs-UI.png" alt="Observability Logs UI" width="700"> | ||
|
||
3. Select the **Event Explorer** button. This option creates and saves frequently searched queries and visualizations using [Piped Processing Language (PPL)]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) or [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/), which connects to Spark SQL. | ||
4. Select the Amazon S3 data source from the dropdown menu in the upper-left corner. An example is shown in the following image: | ||
|
||
<img src="{{site.url}}{{site.baseurl}}/images/dashboards/query-data-sources-UI-2.png" alt="Observability Logs Amazon S3 dropdown menu" width="700"> | ||
|
||
5. Enter the query in the **Enter PPL query** field. Note that the default language is SQL. To change the language, select PPL from the dropdown menu. | ||
6. Select the **Search** button. The **Query Processing** message is shown, confirming that your query is being processed. | ||
7. View the results, which are listed in a table under the **Events** tab. From this page, details such as available fields, source, and time are shown in a table format. | ||
8. (Optional) Create data visualizations. | ||
|
||
## Create visualizations of your Amazon S3 data | ||
|
||
To create visualizations, follow these steps: | ||
|
||
1. From the Explorer page, select the **Visualizations** tab. An example is shown in the following image: | ||
|
||
img src="{{site.url}}{{site.baseurl}}/images/dashboards/explorer-S3viz-UI.png" alt="Explorer Amazon S3 visualizations UI" width="700"> | ||
|
||
2. Select **Index data to visualize**. This option currently only creates [acceleration indexes]({{site.url}}{{site.baseurl}}/dashboards/management/accelerate-external-data/), which give you views of the data visualizations from within the **Visualizations** tab. To create a visualization of your Amazon S3 data, go to **Discover**. See the [**Discover** documentation]({{site.url}}{{site.baseurl}}/dashboards/discover/index-discover/) for information and a tutorial. <SME: How does the user create visualizations of their S3 data? Through Discover?> | ||
|
||
## Use Query Workbench with your Amazon S3 data source | ||
|
||
[Query Workbench]({{site.url}}{{site.baseurl}}/search-plugins/sql/workbench/) runs on-demand SQL queries, translates SQL into its REST equivalent, and views and saves results as text, JSON, JDBC, or CSV. | ||
|
||
To use Query Workbench with your Amazon S3 data, follow these steps: | ||
|
||
1. From the OpenSearch Dashboards main menu, select **OpenSearch Plugins** > **Query Workbench**. | ||
2. From the **Data Sources** dropdown menu in the upper-left corner, choose your Amazon S3 data source. Your data begins loading the databases that are part of your data source. An example is shown in the following image: | ||
|
||
<img src="{{site.url}}{{site.baseurl}}/images/dashboards/query-workbench-S3.png" alt="Query Workbench Amazon S3 data loading UI" width="700"> | ||
|
||
3. View the databases listed in the left-side navigation menu and select a database to view its details. Any information about acceleration indexes is listed under **Acceleration index destination**. | ||
4. Choose the **Describe Index** button to learn more about how data is stored in that particular index. | ||
5. Choose the **Drop index** button to delete and clear both the OpenSearch index and the Amazon S3 Spark job that refreshes the data. | ||
|
||
## Nest steps | ||
|
||
- Learn about [accelerating query performance of your external data sources]({{site.url}}{{site.baseurl}}/dashboards/management/accelerate-external-data/). |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.