Skip to content

Commit

Permalink
Documentation regarding purely sync defer
Browse files Browse the repository at this point in the history
  • Loading branch information
ewjoachim authored and Joachim Jablon committed Jun 11, 2020
1 parent 1d9b9c4 commit c6232f1
Show file tree
Hide file tree
Showing 4 changed files with 72 additions and 0 deletions.
25 changes: 25 additions & 0 deletions docs/discussions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,31 @@ either serial or parallel (see `howto/async`, `howto/concurrency`).

The rest of this section will discuss the specifics of asynchronous parallel workers.

.. _discussion-async-sync-defer:

Synchronous deferring
^^^^^^^^^^^^^^^^^^^^^

Procrastinate gets to be called in two very different contexts:

- When deferring a task, in **your** process, where Procrastinate is being used as a
library
- When running a worker, where Procrastinate itself controls the process.

Workers will always be asynchronous, and will support both synchronous and asynchronous
jobs. When deferring a task, on the other hand, Procrastinate needs to play nice with
your program.

Synchronous APIs in Procrastinate will synchronously launch an asynchronous event loop
and run inside this loop. In some cases, especially if your program is multithreaded,
you may prefer procrastinate defers to use standard synchronous I/O.

If you use an `AiopgConnector`, then, by default, synchronous calls to ``.defer()`` will
use the synchronous wrapper described above. You can request the ``AiopgConnector`` to
use a realy ``psycopg2``-based implementation that will be nicer to multithreaded
programs by passing ``real_sync_defer=True`` when creating the ``AiopgConnector``. See
`howto/sync_defer`.

Don't mix sync and async
^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
40 changes: 40 additions & 0 deletions docs/howto/sync_defer.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
Control the way synchronous calls to defer are handled
======================================================

In some cases, usually linked to multithreading (see `discussion-async-sync-defer`), you
may want to defer tasks purely synchronously. This can be achieving by creating your
`AiopgConnector` with ``real_sync_defer=True``::

import procrastinate

app = procrastinate.App(
connector=procrastinate.AiopgConnector(
host="somehost",
real_sync_defer=True
),
)

This will will create a ``psycopg2.pool.ThreadedConnectionPool`` (see psycopg2
documentation__), which should fit most workflows.

.. __: https://www.psycopg.org/docs/pool.html#psycopg2.pool.ThreadedConnectionPool

Note: the ``ThreadedConnectionPool`` will be instanciated with mostly the same
parameters as the ``aiopg.Pool``. That being said, there are small differences here and
there between the two parameter sets. If you find a parameter that is not well
supported, feel free to warn us through an issue__.

.. __: https://github.com/peopledoc/procrastinate/issues

Pros and Cons of ``real_sync_defer``
------------------------------------

Pros:

- Works better with multithreaded programs
- Might be a bit faster because you don't create an event loop every time

Cons:

- Uses its own connection pool, which will increase the total number of connections
used by procrastinate.
1 change: 1 addition & 0 deletions docs/howto_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ How to...
howto/middleware
howto/events
howto/async
howto/sync_defer
howto/logging
howto/testing
howto/deployment
Expand Down
6 changes: 6 additions & 0 deletions procrastinate/aiopg_connector.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,12 @@ def __init__(
to the function used by psycopg2. See the `psycopg2 doc`_. Unused if the
pool is externally created and set into the connector through the
`AiopgConnector.set_pool` method.
real_sync_defer :
If a synchronous call to a defer operation is issued, whether to call a
really synchronous psycopg2 implementation (``True``) which will use its own
connection pool, or a synchronous wrapper around this asynchronous
connector, which may not play as nicely with multithreaded programs but will
use connections from this connector's pool (``False``).
dsn : ``Optional[str]``
Passed to aiopg. Default is "" instead of None, which means if no argument
is passed, it will connect to localhost:5432 instead of a Unix-domain
Expand Down

0 comments on commit c6232f1

Please sign in to comment.