-
Notifications
You must be signed in to change notification settings - Fork 57
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
(cherry picked from commit 31b0a94) Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Nathan Bower <[email protected]>
- Loading branch information
1 parent
67a5952
commit 65f8eaa
Showing
21 changed files
with
756 additions
and
1,326 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,33 +8,21 @@ export const dedupCmd = `## dedup | |
### Description | ||
Using \'dedup\' command to remove identical document defined by field from | ||
the search result. | ||
Use the \'dedup\' command to remove identical documents from the search results, based on the specified field. | ||
### Syntax | ||
dedup \[int\] <field-list> \[keepempty=<bool>\] | ||
\[consecutive=<bool>\] | ||
- int: optional. The \'dedup\' command retains multiple events for each | ||
combination when you specify <int>. The number for <int> | ||
must be greater than 0. If you do not specify a number, only the | ||
first occurring event is kept. All other duplicates are removed from | ||
the results. **Default:** 1 | ||
- keepempty: optional. if true, keep the document if the any field in | ||
the field-list has NULL value or field is MISSING. **Default:** | ||
false. | ||
- consecutive: optional. If set to true, removes only events with | ||
duplicate combinations of values that are consecutive. **Default:** | ||
false. | ||
- field-list: mandatory. The comma-delimited field list. At least one | ||
field is required. | ||
- \`field-list\`: Required. The comma-delimited field list. At least one field is required. | ||
- \`consecutive\`: Optional. If set to \`true\`, removes duplicate events, where the duplicate events have consecutive timestamps. Default is \`false\`. | ||
- \`int\`: Optional. The \'dedup\' command retains multiple events for each combination when you specify \`<int>\`. The number for \`<int>\` must be greater than 0. If you do not specify a number, only the first occurring event is kept. All other duplicates are removed from the results. Default is \`1\`. | ||
- \`keepempty\`: Optional. If set to \`true\`, keeps the document if any field in the \`field-list\` is null or missing. Default is \`false\`. | ||
### Example 1: Dedup by one field | ||
#### Example 1: Dedup by one field | ||
The example show dedup the document with gender field. | ||
PPL query: | ||
The following example PPL query shows how to use \`dedup\` to remove duplicate documents based on the \`gender\` field: | ||
os> source=accounts | dedup gender | fields account_number, gender; | ||
fetched rows / total rows = 2/2 | ||
|
@@ -45,12 +33,9 @@ PPL query: | |
| 13 | F | | ||
+------------------+----------+ | ||
### Example 2: Keep 2 duplicates documents | ||
The example show dedup the document with gender field keep 2 | ||
duplication. | ||
#### Example 2: Keep two duplicate documents | ||
PPL query: | ||
The following example PPL query shows how to use \`dedup\` to remove duplicate documents based on the \`gender\` field while keeping two duplicates: | ||
os> source=accounts | dedup 2 gender | fields account_number, gender; | ||
fetched rows / total rows = 3/3 | ||
|
@@ -62,11 +47,9 @@ PPL query: | |
| 13 | F | | ||
+------------------+----------+ | ||
### Example 3: Keep or Ignore the empty field by default | ||
The example show dedup the document by keep null value field. | ||
#### Example 3: Keep or ignore empty fields by default | ||
PPL query: | ||
The following example PPL query shows how to use \`dedup\` to remove duplicate documents while keeping documents with null values in the specified field: | ||
os> source=accounts | dedup email keepempty=true | fields account_number, email; | ||
fetched rows / total rows = 4/4 | ||
|
@@ -79,9 +62,7 @@ PPL query: | |
| 18 | [email protected] | | ||
+------------------+-----------------------+ | ||
The example show dedup the document by ignore the empty value field. | ||
PPL query: | ||
The following example PPL query shows how to use \`dedup\` to remove duplicate documents while ignoring documents with empty values in the specified field: | ||
os> source=accounts | dedup email | fields account_number, email; | ||
fetched rows / total rows = 3/3 | ||
|
@@ -93,11 +74,9 @@ PPL query: | |
| 18 | [email protected] | | ||
+------------------+-----------------------+ | ||
#### Example 4: Dedup in consecutive document | ||
The example show dedup the consecutive document. | ||
#### Example 4: Remove duplicate consecutive documents | ||
PPL query: | ||
The following example PPL query shows how to use \`dedup\` to remove duplicate consecutive documents: | ||
os> source=accounts | dedup gender consecutive=true | fields account_number, gender; | ||
fetched rows / total rows = 3/3 | ||
|
@@ -109,6 +88,6 @@ PPL query: | |
| 18 | M | | ||
+------------------+----------+ | ||
#### Limitation | ||
The \`dedup\` command is not rewritten to OpenSearch DSL, it is only executed on the coordination node. | ||
`; | ||
### Limitation | ||
The \`dedup\` command is not rewritten to [query domain-specific language (DSL)](https://opensearch.org/docs/latest/query-dsl/index/). It is only run on the coordinating node. | ||
`; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.