Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync connector 225 #237

Merged
merged 12 commits into from
Jun 16, 2020
Merged

Sync connector 225 #237

merged 12 commits into from
Jun 16, 2020

Conversation

ewjoachim
Copy link
Member

@ewjoachim ewjoachim commented Jun 6, 2020

Closes #225

Haven't touched the tests yet.

Successful PR Checklist:

  • Tests
    • (not applicable?)
  • Documentation
    • (not applicable?)
  • Had a good time contributing?

procrastinate/app.py Outdated Show resolved Hide resolved
@ewjoachim ewjoachim force-pushed the sync-connector-225 branch from 7db8821 to 0016349 Compare June 11, 2020 12:57
@ewjoachim
Copy link
Member Author

ewjoachim commented Jun 11, 2020

Ok, I've done a 2nd pass.

@ewjoachim ewjoachim force-pushed the sync-connector-225 branch from 0016349 to c6232f1 Compare June 11, 2020 13:49
procrastinate/store.py Outdated Show resolved Hide resolved
docs/howto/sync_defer.rst Outdated Show resolved Hide resolved
@ewjoachim ewjoachim force-pushed the sync-connector-225 branch 4 times, most recently from 728ac84 to f9ce79c Compare June 15, 2020 16:45
@ewjoachim ewjoachim marked this pull request as ready for review June 15, 2020 16:45
@codecov
Copy link

codecov bot commented Jun 15, 2020

Codecov Report

Merging #237 into master will not change coverage.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##            master      #237    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           24        25     +1     
  Lines         1100      1226   +126     
  Branches       135       140     +5     
==========================================
+ Hits          1100      1226   +126     
Impacted Files Coverage Δ
procrastinate/app.py 100.00% <ø> (ø)
procrastinate/__init__.py 100.00% <100.00%> (ø)
procrastinate/admin.py 100.00% <100.00%> (ø)
procrastinate/aiopg_connector.py 100.00% <100.00%> (ø)
procrastinate/cli.py 100.00% <100.00%> (ø)
procrastinate/connector.py 100.00% <100.00%> (ø)
procrastinate/exceptions.py 100.00% <100.00%> (ø)
procrastinate/healthchecks.py 100.00% <100.00%> (ø)
procrastinate/jobs.py 100.00% <100.00%> (ø)
procrastinate/psycopg2_connector.py 100.00% <100.00%> (ø)
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10c3171...756f4fd. Read the comment docs.

This mode is necessary in multi-threaded cases.
- "mixed" I/O (synchronously launching an event loop, and have asynchronous coroutine
run under the hood).
This mode will use less connections when used within another job.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been thinking about this whole thing over the week-end 😄 And I must say that I am not a big fan of supporting two ways of doing synchronous I/O. The paragraph above, where we explain that there's a "classic" way and a "mixed" way, is, to me, a clear sign of complexity and poor design. I think what I'd like to see a very clear separation between the sync API and the async API. I'd even consider removing the add_sync_api decorator, and making the async API async only. And introducing a separate sync API for sync applications. I know my words may sound a bit theoretic… but that's all I have at this point.

Copy link
Member Author

@ewjoachim ewjoachim Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but what about deferring from within a task ? You already have an app and you most probably wish not to make a new pool for this.

@ewjoachim ewjoachim force-pushed the sync-connector-225 branch from 7ea5973 to d4e896e Compare June 16, 2020 07:48
@elemoine elemoine force-pushed the sync-connector-225 branch 2 times, most recently from f30697c to d812171 Compare June 16, 2020 08:16
This mode is necessary in multi-threaded cases.
- "mixed" I/O (synchronously launching an event loop, and have asynchronous coroutine
run under the hood). This mode is adapted for synchronously defering jobs from other
job, from within the workers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
job, from within the workers.
jobs, from within the workers.

- "classic" synchronous I/O (using synchronous database drivers such as ``Psycopg2``).
This mode is necessary in multi-threaded cases.
- "mixed" I/O (synchronously launching an event loop, and have asynchronous coroutine
run under the hood). This mode is adapted for synchronously defering jobs from other
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
run under the hood). This mode is adapted for synchronously defering jobs from other
run under the hood). This mode is adapted for synchronously deferring jobs from other

@@ -265,11 +292,15 @@ this? Here are a few resources:
with sync **and** async callers: "Just add await" from Andrew Godwin:
https://www.youtube.com/watch?v=oMHrDy62kgE

That being said, synchronous defers (see `discussion-sync-defer`) rely on a few
specific methods that have been duplicated by hand (one async and one sync version).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ewjoachim is this comment sill valid? I am wondering about "duplicated by hand".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've clarified it, but it's still valid.

Copy link
Contributor

@elemoine elemoine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few new comments and suggestions, otherwise looks good to me.

@ewjoachim ewjoachim force-pushed the sync-connector-225 branch from d812171 to 756f4fd Compare June 16, 2020 10:54
@ewjoachim ewjoachim merged commit 7ed0f3d into master Jun 16, 2020
@ewjoachim ewjoachim deleted the sync-connector-225 branch June 16, 2020 11:15
@ewjoachim
Copy link
Member Author

This closed #152 and #149

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR type: feature ⭐️ Contains new features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants