Add support for Async I/O via asyncio #1230
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Splitting this PR off into their own separate PRs for reviewability:
Going to continue splitting the individual commits into their own PRs
Async Notes
The work in this PR right now accomplishes:
AIOHttpConnection
which usesaiohttp
as an HTTP clientAsyncTransport
which is capable of background sniffing with anasyncio.Task
AsyncConnectionPool
which is basically only needed for an async.close()
method for connections.[async]
extra which will install dependencies required forAsyncElasticsearch
to be available. Can be used for upstream dependencies (likeelasticsearch-dsl[async]
requireselasticsearch[async]
)async_parallel_bulk()
isn't implemented currently because it'd require it's own custom implementation, doesn't play nicely withunasync
.unasync
to generate sync client APIs and sync helpers. This guarantees that APIs and helpers are in sync with each other.Things that are still todo but will be accomplished outside this PR:
elasticsearch-async
repository to point toelasticsearch[async]
for 7.x+async_parallel_bulk()
Currently RequestsHttpConnection tests are being directly replaced with AIOHttpConnection tests meaning RequestsHttpConnection isn't tested on Python 3.6+. Isn't too big of an issue.
PYTHON_CONNECTION_CLASS=AIOHttpConnection
test async instead of relying on versions.Planning on taking a similar approach with adding async support to
elasticsearch-dsl
except will probably be a lot less code, only needsunasync
in a few places. :)cc @vEpiphyte