Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Async I/O via asyncio #1230

Closed
wants to merge 9 commits into from
Closed

Add support for Async I/O via asyncio #1230

wants to merge 9 commits into from

Conversation

sethmlarson
Copy link
Contributor

@sethmlarson sethmlarson commented May 13, 2020

Splitting this PR off into their own separate PRs for reviewability:

Going to continue splitting the individual commits into their own PRs

Async Notes

The work in this PR right now accomplishes:

  • Switches the primary test runner and some fixtures to be pytest. Eventually would like to switch everything to be pytest.
  • Add AIOHttpConnection which uses aiohttp as an HTTP client
  • Add AsyncTransport which is capable of background sniffing with an asyncio.Task
  • Add AsyncConnectionPool which is basically only needed for an async .close() method for connections.
  • Add the [async] extra which will install dependencies required for AsyncElasticsearch to be available. Can be used for upstream dependencies (like elasticsearch-dsl[async] requires elasticsearch[async])
  • Add async implementations of helpers. async_parallel_bulk() isn't implemented currently because it'd require it's own custom implementation, doesn't play nicely with unasync.
  • Change the API generator to generate async APIs first then use unasync to generate sync client APIs and sync helpers. This guarantees that APIs and helpers are in sync with each other.
  • Adds docs for async client and helpers.

Things that are still todo but will be accomplished outside this PR:

  • Update elasticsearch-async repository to point to elasticsearch[async] for 7.x+
  • Implement async_parallel_bulk()
  • Switch sync tests to use pytest fixtures
  • Figure out a better way to "stripe" the async/sync tests across Python 3.6+
    Currently RequestsHttpConnection tests are being directly replaced with AIOHttpConnection tests meaning RequestsHttpConnection isn't tested on Python 3.6+. Isn't too big of an issue.
  • Make PYTHON_CONNECTION_CLASS=AIOHttpConnection test async instead of relying on versions.

Planning on taking a similar approach with adding async support to elasticsearch-dsl except will probably be a lot less code, only needs unasync in a few places. :)

cc @vEpiphyte

@sethmlarson sethmlarson mentioned this pull request May 13, 2020
13 tasks
@sethmlarson
Copy link
Contributor Author

I'm planning on doing a refactor of helpers without unasync as it doesn't make the actual changes easy to review. Planning on pulling out the sync components not tied to either making an API call or handling an iterator which can be async. That'll hopefully make the helpers easier to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant