Skip to content

Commit

Permalink
Added advanced index actions API guide & sample.
Browse files Browse the repository at this point in the history
Signed-off-by: Djcarrillo6 <[email protected]>

Updated CHANGELOG

Signed-off-by: Djcarrillo6 <[email protected]>

Updated CHANGELOG & link to sample.

Signed-off-by: Djcarrillo6 <[email protected]>

updated changelog (opensearch-project#522)

Signed-off-by: saimedhi <[email protected]>

Bump version to 2.3.2 (opensearch-project#524)

Signed-off-by: saimedhi <[email protected]>

Fix: typos. (opensearch-project#526)

* Fix: typo.

Signed-off-by: dblock <[email protected]>

* Fix: typo.

Signed-off-by: dblock <[email protected]>

* Fixed its.

Signed-off-by: dblock <[email protected]>

* Added Visual Code settings to .gitignore.

Signed-off-by: dblock <[email protected]>

* Added loop type for async client.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>

Modified generator to generate api deprecation warnings (opensearch-project#527)

Signed-off-by: saimedhi <[email protected]>

Generate cat client from API specs (opensearch-project#529)

Signed-off-by: saimedhi <[email protected]>

Generate cluster client from API specs (opensearch-project#530)

Signed-off-by: saimedhi <[email protected]>

Added new guide & sample module for using index templates. (opensearch-project#531)

Added index_template guide and sample

Signed-off-by: Djcarrillo6 <[email protected]>

Removed EOL Python3.5 & bumped urllib3 version to patch security vulnerability (opensearch-project#533)

Updated CHANGELOG with pull #

Updated CHANGELOG with pull #

Updated CHANGELOG removed section.

Updated CHANGELOG removed section again

Signed-off-by: Djcarrillo6 <[email protected]>

Align pool_maxsize for different connection pool implementations. (opensearch-project#535)

* Align pool_maxsize for different connection pool implementations.

Signed-off-by: dblock <[email protected]>

* Document connection classes and settings.

Signed-off-by: dblock <[email protected]>

* Undo change in async for backwards compatibility.

Signed-off-by: dblock <[email protected]>

* Fix: typo.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>

Add micro benchmarks. (opensearch-project#537)

* Align pool_maxsize for different connection pool implementations.

Signed-off-by: dblock <[email protected]>

* Added benchmarks.

Signed-off-by: dblock <[email protected]>

* Multi-threaded vs. async benchmarks.

Signed-off-by: dblock <[email protected]>

* Set pool size to the number of threads.

Signed-off-by: dblock <[email protected]>

* Added sync/async benchmark.

Signed-off-by: dblock <[email protected]>

* Report client-side latency.

Signed-off-by: dblock <[email protected]>

* Various updates to benchmarks, demonstrating threading improves throughput.

Signed-off-by: dblock <[email protected]>

* Bench info.

Signed-off-by: dblock <[email protected]>

* Fixup format.

Signed-off-by: dblock <[email protected]>

* Undo async maxsize.

Signed-off-by: dblock <[email protected]>

* Moved benchmarks folder.

Signed-off-by: dblock <[email protected]>

* Updated documentation and project description.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
  • Loading branch information
Djcarrillo6 committed Oct 14, 2023
1 parent 80b9681 commit 5edf6c0
Show file tree
Hide file tree
Showing 52 changed files with 3,683 additions and 911 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
fail-fast: false
matrix:
opensearch_version: [ '1.0.1', '1.1.0', '1.2.4', '1.3.7', '2.0.1', '2.1.0', '2.2.1', '2.3.0', '2.4.0', '2.5.0', '2.6.0', '2.7.0', '2.8.0' ]
opensearch_version: [ '1.0.1', '1.1.0', '1.2.4', '1.3.7', '2.0.1', '2.1.0', '2.2.1', '2.3.0', '2.4.0', '2.5.0', '2.6.0', '2.7.0', '2.8.0', '2.9.0', '2.10.0' ]
secured: [ "true", "false" ]

steps:
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ jobs:
strategy:
matrix:
entry:
- { os: 'ubuntu-20.04', python-version: "3.5" }
- { os: 'ubuntu-20.04', python-version: "3.6" }
- { os: 'ubuntu-latest', python-version: "3.7" }
- { os: 'ubuntu-latest', python-version: "3.8" }
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unified-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
fail-fast: false
matrix:
stack_version: ['2.3.1']
stack_version: ['2.3.2']

steps:
- name: Checkout
Expand Down
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,9 @@ test_opensearch/cover
test_opensearch/local.py
.ci/output

#Vi text editor
# vi text editor
.*.swp
*~

# Visual Studio Code
.vscode
32 changes: 24 additions & 8 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,33 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
### Added
- Added generating imports and headers to API generator ([#467](https://github.com/opensearch-project/opensearch-py/pull/467))
- Added point-in-time APIs (create_pit, delete_pit, delete_all_pits, get_all_pits) and Security Client APIs (health and update_audit_configuration) ([#502](https://github.com/opensearch-project/opensearch-py/pull/502))
- Added new guide for using index templates with the client ([#531](https://github.com/opensearch-project/opensearch-py/pull/531))
- Added `pool_maxsize` for `Urllib3HttpConnection` ([#535](https://github.com/opensearch-project/opensearch-py/pull/535))
- Added benchmarks ([#537](https://github.com/opensearch-project/opensearch-py/pull/537))
### Changed
- Integrated generated `tasks client` APIs into the existing module, ensuring alignment with the server and maintaining backward compatibility ([#508](https://github.com/opensearch-project/opensearch-py/pull/508))
- Integrated generated `ingest client` APIs into the existing module, ensuring alignment with the server and maintaining backward compatibility ([#513](https://github.com/opensearch-project/opensearch-py/pull/513))
- Integrated generated `dangling_indices client` APIs into the existing module, ensuring alignment with the server and maintaining backward compatibility ([#511](https://github.com/opensearch-project/opensearch-py/pull/511))
- Integrated generated `nodes client` APIs into the existing module, ensuring alignment with the server and maintaining backward compatibility ([#514](https://github.com/opensearch-project/opensearch-py/pull/514))
- Generate `tasks` client from API specs ([#508](https://github.com/opensearch-project/opensearch-py/pull/508))
- Generate `ingest` client from API specs ([#513](https://github.com/opensearch-project/opensearch-py/pull/513))
- Generate `dangling_indices` client from API specs ([#511](https://github.com/opensearch-project/opensearch-py/pull/511))
- Generate `cluster` client from API specs ([#530](https://github.com/opensearch-project/opensearch-py/pull/530))
- Generate `nodes` client from API specs ([#514](https://github.com/opensearch-project/opensearch-py/pull/514))
- Generate `cat` client from API specs ([#529](https://github.com/opensearch-project/opensearch-py/pull/529))
### Deprecated
- Deprecated point-in-time APIs (list_all_point_in_time, create_point_in_time, delete_point_in_time) and Security Client APIs (health_check and update_audit_config) ([#502](https://github.com/opensearch-project/opensearch-py/pull/502))
### Removed
### Fixed
### Security
### Dependencies
- Bumps `sphinx` from <7.1 to <7.3
- Bumps `urllib3` from >=1.21.1, <2 to >=1.21.1 ([#466](https://github.com/opensearch-project/opensearch-py/pull/466))
- Bumps `urllib3` from >=1.21.1 to >=1.26.9 ([#518](https://github.com/opensearch-project/opensearch-py/pull/518))

## [2.3.2]
### Added
### Changed
### Deprecated
### Removed
### Fixed
### Security
### Dependencies
- Bumps `urllib3` from >=1.21.1, <2 to >=1.26.9 ([#518](https://github.com/opensearch-project/opensearch-py/pull/518))

## [2.3.1]
### Added
Expand All @@ -29,6 +42,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Fixed race condition in AWSV4SignerAuth & AWSV4SignerAsyncAuth when using refreshable credentials ([#470](https://github.com/opensearch-project/opensearch-py/pull/470))
### Security
### Dependencies
- Bumps `urllib3` from >= 1.26.9 to >= 1.26.17 [#533](https://github.com/opensearch-project/opensearch-py/pull/533)

## [2.3.0]
### Added
Expand All @@ -40,7 +54,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Added support for the security plugin ([#399](https://github.com/opensearch-project/opensearch-py/pull/399))
- Supports OpenSearch 2.1.0 - 2.6.0 ([#381](https://github.com/opensearch-project/opensearch-py/pull/381))
- Added `allow_redirects` to `RequestsHttpConnection#perform_request` ([#401](https://github.com/opensearch-project/opensearch-py/pull/401))
- Enhanced YAML test runner to use OpenSearch `rest-api-spec` YAML tests ([#414](https://github.com/opensearch-project/opensearch-py/pull/414)
- Enhanced YAML test runner to use OpenSearch `rest-api-spec` YAML tests ([#414](https://github.com/opensearch-project/opensearch-py/pull/414))
- Added `Search#collapse` ([#409](https://github.com/opensearch-project/opensearch-py/issues/409))
- Added support for the ISM API ([#398](https://github.com/opensearch-project/opensearch-py/pull/398))
- Added `trust_env` to `AIOHttpConnection` ([#398](https://github.com/opensearch-project/opensearch-py/pull/438))
Expand All @@ -55,6 +69,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
### Deprecated
### Removed
- Removed support for Python 2.7 ([#421](https://github.com/opensearch-project/opensearch-py/pull/421))
- Removed support for Python 3.5 ([#533](https://github.com/opensearch-project/opensearch-py/pull/533))
### Fixed
- Fixed flaky CI tests by replacing httpbin with a simple http_server ([#395](https://github.com/opensearch-project/opensearch-py/pull/395))
- Fixed import cycle when importing async helpers ([#311](https://github.com/opensearch-project/opensearch-py/pull/311))
Expand Down Expand Up @@ -129,10 +144,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Fixed Wrong return type hint in `async_scan` ([520](https://github.com/opensearch-project/opensearch-py/pull/520))
### Security

[Unreleased]: https://github.com/opensearch-project/opensearch-py/compare/v2.3.1...HEAD
[Unreleased]: https://github.com/opensearch-project/opensearch-py/compare/v2.3.2...HEAD
[2.0.1]: https://github.com/opensearch-project/opensearch-py/compare/v2.0.0...v2.0.1
[2.1.0]: https://github.com/opensearch-project/opensearch-py/compare/v2.0.1...v2.1.0
[2.1.1]: https://github.com/opensearch-project/opensearch-py/compare/v2.1.0...v2.1.1
[2.2.0]: https://github.com/opensearch-project/opensearch-py/compare/v2.1.1...v2.2.0
[2.3.0]: https://github.com/opensearch-project/opensearch-py/compare/v2.2.0...v2.3.0
[2.3.1]: https://github.com/opensearch-project/opensearch-py/compare/v2.3.0...v2.3.1
[2.3.2]: https://github.com/opensearch-project/opensearch-py/compare/v2.3.1...v2.3.2
11 changes: 6 additions & 5 deletions COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,12 @@ The below matrix shows the compatibility of the [`opensearch-py`](https://pypi.o
| --- | --- | --- |
| 1.0.0 | 1.0.0-1.2.4 | |
| 1.1.0 | 1.3.0-1.3.7 | |
| 2.0.x | 1.0.0-2.8.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.1.x | 1.0.0-2.8.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.2.0 | 1.0.0-2.8.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.3.0 | 1.0.0-2.8.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.3.1 | 1.0.0-2.8.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.0.x | 1.0.0-2.10.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.1.x | 1.0.0-2.10.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.2.0 | 1.0.0-2.10.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.3.0 | 1.0.0-2.10.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.3.1 | 1.0.0-2.10.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
| 2.3.2 | 1.0.0-2.10.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |

## Upgrading

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ For more information, see [opensearch.org](https://opensearch.org/) and the [API

## User Guide

To get started with the OpenSearch Python Client, see [User Guide](https://github.com/opensearch-project/opensearch-py/blob/main/USER_GUIDE.md).
To get started with the OpenSearch Python Client, see [User Guide](https://github.com/opensearch-project/opensearch-py/blob/main/USER_GUIDE.md). This repository also contains [working samples](https://github.com/opensearch-project/opensearch-py/tree/main/samples) and [benchmarks](https://github.com/opensearch-project/opensearch-py/tree/main/benchmarks).

## Compatibility with OpenSearch

Expand Down
2 changes: 2 additions & 0 deletions USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,8 @@ print(response)
- [Search](guides/search.md)
- [Point in Time](guides/point_in_time.md)
- [Using a Proxy](guides/proxy.md)
- [Index Templates](guides/index_template.md)
- [Connection Classes](guides/connection_classes.md)

## Plugins

Expand Down
63 changes: 63 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
- [Benchmarks](#benchmarks)
- [Start OpenSearch](#start-opensearch)
- [Install Prerequisites](#install-prerequisites)
- [Run Benchmarks](#run-benchmarks)

## Benchmarks

Python client benchmarks using [richbench](https://github.com/tonybaloney/rich-bench).

### Start OpenSearch

```
docker run -p 9200:9200 -e "discovery.type=single-node" opensearchproject/opensearch:latest
```

### Install Prerequisites

Install [poetry](https://python-poetry.org/docs/), then install package dependencies.

```
poetry install
```

Benchmarks use the code in this repository by specifying the dependency as `opensearch-py = { path = "..", develop=true, extras=["async"] }` in [pyproject.toml](pyproject.toml).

### Run Benchmarks

Run all benchmarks available as follows.

```
poetry run richbench . --repeat 1 --times 1
```

Outputs results from all the runs.

```
Benchmarks, repeat=1, number=1
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Min ┃ Max ┃ Mean ┃ Min (+) ┃ Max (+) ┃ Mean (+) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 1 client vs. more clients (async) │ 1.640 │ 1.640 │ 1.640 │ 1.102 (1.5x) │ 1.102 (1.5x) │ 1.102 (1.5x) │
│ 1 thread vs. 32 threads (sync) │ 5.526 │ 5.526 │ 5.526 │ 1.626 (3.4x) │ 1.626 (3.4x) │ 1.626 (3.4x) │
│ 1 thread vs. 32 threads (sync) │ 4.639 │ 4.639 │ 4.639 │ 3.363 (1.4x) │ 3.363 (1.4x) │ 3.363 (1.4x) │
│ sync vs. async (8) │ 3.198 │ 3.198 │ 3.198 │ 0.966 (3.3x) │ 0.966 (3.3x) │ 0.966 (3.3x) │
└───────────────────────────────────┴─────────┴─────────┴─────────┴─────────────────┴─────────────────┴─────────────────┘
```

Run a specific benchmark, e.g. [bench_sync.py](bench_sync.py) by specifying `--benchmark [name]`.

```
poetry run richbench . --repeat 1 --times 1 --benchmark sync
```

Outputs results from one benchmark.

```
Benchmarks, repeat=1, number=1
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Benchmark ┃ Min ┃ Max ┃ Mean ┃ Min (+) ┃ Max (+) ┃ Mean (+) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 1 thread vs. 32 threads (sync) │ 6.804 │ 6.804 │ 6.804 │ 3.409 (2.0x) │ 3.409 (2.0x) │ 3.409 (2.0x) │
└────────────────────────────────┴─────────┴─────────┴─────────┴─────────────────┴─────────────────┴─────────────────┘
```
101 changes: 101 additions & 0 deletions benchmarks/bench_async.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
#!/usr/bin/env python

# SPDX-License-Identifier: Apache-2.0
#
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.

import asyncio
import uuid

from opensearchpy import AsyncHttpConnection, AsyncOpenSearch

host = "localhost"
port = 9200
auth = ("admin", "admin")
index_name = "test-index-async"
item_count = 100


async def index_records(client, item_count):
await asyncio.gather(
*[
client.index(
index=index_name,
body={
"title": f"Moneyball",
"director": "Bennett Miller",
"year": "2011",
},
id=uuid.uuid4(),
)
for j in range(item_count)
]
)


async def test_async(client_count=1, item_count=1):
clients = []
for i in range(client_count):
clients.append(
AsyncOpenSearch(
hosts=[{"host": host, "port": port}],
http_auth=auth,
use_ssl=True,
verify_certs=False,
ssl_show_warn=False,
connection_class=AsyncHttpConnection,
pool_maxsize=client_count,
)
)

if await clients[0].indices.exists(index_name):
await clients[0].indices.delete(index_name)

await clients[0].indices.create(index_name)

await asyncio.gather(
*[index_records(clients[i], item_count) for i in range(client_count)]
)

await clients[0].indices.refresh(index=index_name)
print(await clients[0].count(index=index_name))

await clients[0].indices.delete(index_name)

await asyncio.gather(*[client.close() for client in clients])


def test(item_count=1, client_count=1):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(test_async(item_count, client_count))
loop.close()


def test_1():
test(1, 32 * item_count)


def test_2():
test(2, 16 * item_count)


def test_4():
test(4, 8 * item_count)


def test_8():
test(8, 4 * item_count)


def test_16():
test(16, 2 * item_count)


def test_32():
test(32, item_count)


__benchmarks__ = [(test_1, test_8, "1 client vs. more clients (async)")]
Loading

0 comments on commit 5edf6c0

Please sign in to comment.