Do notation

[ckp95]

With coroutines we can implement a Haskell-ish do-notation. Call them "doroutines":

from pymonad.operators.maybe import Maybe, Just, Nothing
from pymonad.tools import curry

@curry(2)
def safe_div(x, y):
    if y == 0:
        return Nothing
    else:
        return Just(x / y)


@do
def something(a, b):
    x = yield Just(a)
    y = yield Just(b)
    divided = yield safe_div(x, y)
    return Just(divided + 10)


first = something(5, 0)
second = something(12, 4)

print(f"{first=}")
print(f"{second=}")

Nothing
Just 13.0

We yield monad values out of the coroutine, and get unwrapped values back. The decorator handles the bookkeeping logic. Here's a rough implementation:

def do(generator):
    def new_func(*args, **kwargs):
        gen = generator(*args, **kwargs)
        
        def recurse_bind(value):
            return gen.send(value) >> recurse_bind
        
        try:
            return next(gen) >> recurse_bind
        except StopIteration as e:
            return e.value
    
    return new_func

The main deficiency here is that we are limited by the Python interpreter's maximum recursion depth of about 1000. I tried to convert it to an iteration but it gave me a headache since we're returning a function each time rather than a value :S.

It currently relies on the generator raising a StopIteration exception to get the return value, which is incompatible with the Either monad which is supposed to capture exceptions and halt their journey up the stack. I see two ways of dealing with this: first, keep the do implementation the same and have the Either monad log all exceptions other than StopIteration; second, create some kind of wrapper/decorator for generators so that they use an Either monad to signal their return value instead of an exception, then alter do so that it exits when it sees this type of error. I think the second one is preferable.

I've only tested it with Maybe so far. I had to edit Maybe so that it doesn't swallow errors as I mentioned in #1. For that I just commented out some parts in the maybe.py file:

    def bind(self: 'Maybe[S]', kleisli_function: 'Callable[[S], Maybe[T]]') -> 'Maybe[T]':
        """ See Monad.bind """
        if self.monoid is False: #pylint: disable=no-else-return
            return self
        else:
            # try:
            return kleisli_function(self.value)
            # except: # pylint: disable=bare-except
            #     return Nothing
...
    def map(self: 'Maybe[S]', function: Callable[[S], T]) -> 'Maybe[T]':
        """ See Monad.map """
        if self.is_nothing(): #pylint: disable=no-else-return
            return self
        else:
            # try:
            return self.__class__(function(self.value), True) # pytype: disable=not-callable
            # except: # pylint: disable=bare-except
            #     return Nothing

I'm not familiar enough with the codebase to figure out a more robust solution yet.

[jasondelaat]

I've been playing with this a bit. There are a few problems with the implementation but I think there are problems with the concept of do-notation in Python more generally. I've made attempts at do-notation in the past and never been particularly satisfied with them. That said, I'm not actually against the idea so if we can find something that works I'll definitely consider it.

With that said, I'll walk you through the problems that I see so far:

First a minor issue. The old version of PyMonad had operators by default but they're very un-pythonic. I've kept them around in the operators sub-package but for general purpose stuff like your 'do' function it's best to stick to the 'bind' instead of '>>' since that will be available everywhere.

More problematic, this won't work as written for other monads. Consider Writer.

from pymonad.writer import Writer

@do
def log():
    yield do_a(None)
    yield do_a(None)
    return do_b(None)

def do_a(_):
    return Writer(None, 'Doing A, ')

def do_b(_):
    return Writer(None, 'Doing B, ')

print(log())

(None, Doing B, )

Even though the functions don't do anything, I want the full log but this will only give me back the log from the last line of log(). Changing the return to a yield means I end up getting nothing back at all.

We can actually fix that and the recursion problem at the same time.

from pymonad.writer import Writer

def do(generator):
    def new_func(*args, **kwargs):
        gen = generator(*args, **kwargs)
        
        val = next(gen)
        while True:
            try:
                val = val.then(lambda x: gen.send(x))
            except StopIteration:
                return val
            
    return new_func

@do
def log():
    yield do_a(None)
    yield do_a(None)
    yield do_b(None)

def do_a(_):
    return Writer(None, 'Doing A, ')

def do_b(_):
    return Writer(None, 'Doing B, ')

print(log())

(None, Doing A, Doing A, Doing B, )

This gives us what we want but we have to ban 'return' from any coroutines that we use with do because using return will end up throwing away the log. A similar problem would happen with State, etc.

Except now that implementation of 'do' won't work with your original example. When we try to divide by zero we get back a Nothing (as expected). The next iteration of the loop then calls 'bind' on Nothing which simply ignores the function passed to it. As a result gen.send() is never called, we never raise StopIteration and we're stuck in an infinite loop.

There are a couple ways we can solve that but neither is ideal.

The first is that we can modify 'do' to check the type of the monad it's working with and if it's a Maybe (or Either) we treat it slightly differently. This is brittle though since any new monads, whether added to PyMonad directly or added by users, might also need special conditions and 'do' becomes a mess and stops being general.

The second solution isn't really a solution, it just looks like one in the right light.

def do(generator):
    def new_func(*args, **kwargs):
        gen = generator(*args, **kwargs)
        
        val = next(gen)
        next_val = val
        while True:
            try:
                next_val = gen.send(val.value)
                val = val.bind(lambda _: next_val)
            except StopIteration as e:
                return val
            except:
                return next_val
    
    return new_func

Basically, we pull the gen.send() call out to get the next value and then we bind to a lambda, ignoring the input and returning the next_val. This works exactly the same for the logging example but it would actually fail for the Maybe example because we'd end up with 'divided' being assigned the value None and then attempting to add None and 10, raising a TypeError.

The example seems to work because the final catch-all except clause catches the TypeError and "backs up" to the last valid value and returns that instead. In your example that happens to be Nothing and so we seem to get the correct behaviour. And this is why it's not really a solution. Suppose we do something like this to the log example:

@do
def log():
    yield do_a(None)
    yield do_a(None)
    x = None + 10
    yield do_b(None)

(None, Doing A, )

That's an actual error that needs to be fixed but the program appears to work fine. In actual fact we're getting a truncated log and masking an error, not unlike the problem with Maybe in Issue #1.

[jasondelaat]

More generally, I think do-notation in Python is just inherently problematic.

Do-notation in Haskell gives an imperative-like syntax to chain or monadic binds. It looks imperative but, under the hood, it's actually not.

Python, on the other hand, actually is imperative so what we're trying to accomplish with do-notation is to basically replace Pythons actual imperative semantics with our own. Which gets messy because we have to yield every line in our functions, or run into the problems above.

On the other hand, using the apply, bind, map, and then methods still gives us a way to order our operations while working with Pythons imperative model. The results aren't always as nice as we might like but at least we're not fighting the language quite so hard.

So, for instance, your original example I would probably write something like this:

def something(a, b):
    x = Just(a)
    y = Just(b)
    return (Maybe
            .apply(safe_div).to_arguments(x, y)
            .then(lambda just: just.then(lambda divided: divided + 10))
    )

And actually, if I add a join method (which I should because it's useful for exactly these situations) we could make it a bit cleaner:

def something(a, b):
    x = Just(a)
    y = Just(b)
    return (Maybe
            .apply(safe_div).to_arguments(x, y) # result: Just Just (x/y)
            .join()                                                 # result: Just (x/y)
            .then(lambda divided: divided + 10) # result: (x/y) + 10
    )

This still isn't perfect but it's not bad.

Note that join() isn't implemented (yet) so this example won't actually work if you try it. The call to apply().to_arguments() will result in a Maybe inside of a Maybe: join() unpacks that one level.

Anyway, as I said I'm not actually against do-notation, I'll probably come back to it periodically. I'll leave the issue open for now because I absolutely welcome any further thoughts on the problem.

[ckp95]

More generally, I think do-notation in Python is just inherently problematic.
Do-notation in Haskell gives an imperative-like syntax to chain or monadic binds. It looks imperative but, under the hood, it's actually not.
Python, on the other hand, actually is imperative so what we're trying to accomplish with do-notation is to basically replace Pythons actual imperative semantics with our own.

That's fair. The reason I want to be able to use do-notation is monadic IO which composes cleanly with other monads.

There is a library called effect which lets you write purely functional IO code. It has its own do-notation decorator for imperative-looking IO code, but it also lets you hook into it and interrogate what side effects it's trying to do, and substitute your own fake inputs back in for testing purposes. That's something you can't easily do with normal Python imperative code.

However, effect's do-notation is hardcoded to only work with its own Effect objects, so it doesn't compose with other monads (via monad transformers or some other compositional mechanism).

A Python do-notation that works on arbitrary monads would be quite useful in this context.

[jasondelaat]

I wasn't aware of effect, thanks for pointing that out. I haven't dug into it very deeply but I took a quick look and had some thoughts. I've not testing this with all monads yet so there might be problems but I think this might actually work. I'll detail the implementation in the next comment, for now I'll just talk about the syntax.

One thing I noticed about effect was that the do blocks didn't return functions/generators but actually returned an Effect value (I think, I may have misunderstood exactly how it was being used), which makes sense since that what a do block should return. So my implementation requires that the coroutine takes zero arguments. You can always wrap it in a function to send it arguments if necessary. The second thing is that, in Haskell, we just return the value and not an already wrapped value. In other words, in Haskell:

do
    x <- Just 1
    y <- Just 2
    return x + y

For that to work we're going to need to know what monad we're dealing with so we end up with something like this. So we get this:

@Maybe.do
def add_1_2():
    x = yield Just(1)
    y = yield Just(2)
    return x + y

print(add_1_2)
print(add_1_2.then(lambda x: x + 1))

Just 3
Just 4

Notice that add_1_2 is an actual Maybe value and we can treat it as such and continue using it with other monad operations.

We can even use your safe_div function from upthread:

@Maybe.do
def div_fail():
    x = yield Just(1)
    y = yield Just(0)
    z = yield safe_div(x, y)
    return z
print(div_fail)

Nothing

We can do a similar thing with Writer:

@Writer.do
def write_1_2():
    x = yield Writer(1, '-got 1-')
    y = yield Writer(2, '-got 2-')
    return x + y
print(write_1_2)

(3, -got 1--got 2-)

Here the return appends an empty string as the message so but we still get the log from the other two Writer values. If we wanted a message we could do this instead:

@Writer.do
def write_1_2():
    x = yield Writer(1, '-got 1-')
    y = yield Writer(2, '-got 2-')
    z = yield Writer(x + y, f'-adding {x} and {y}-')
    return z
print(write_1_2)

(3, -got 1--got 2--adding 1 and 2-)

So far so good. But this also makes it possible to do things like this:

@Maybe.do
def stacked():
    x = yield Just(1)
    y = yield Just(2)
    @Writer.do
    def write():
        z = yield Writer(x + y, f'adding {x} and {y}')
        return z
    return write

Just (3, adding 1 and 2)

So we've got a Writer inside of a Maybe. There aren't proper monad transformers in pymonad so at the moment you'd have to deal with these kinds of values manually but that would've been the case anyway. You can, of course, create values like this manually or using the monad methods without a do-syntax but this is not horrible, at least from my perspective.

[jasondelaat]

The implementation is actually simple and not too different from what we had up above:

In the monad module:

class Monad:
        @classmethod
        def do(cls, coroutine):
            def _do_internal():
                gen = coroutine()
                val = next(gen)
                next_val = val
                while True:
                    try:
                        next_val = gen.send(val.value)
                        val = val.bind(lambda _: next_val)
                    except StopIteration as final_result:
                        return val.bind(lambda _: cls.insert(final_result.value))
            return _do_internal()

I'll have to do some thorough testing before I'm willing to add it as an actual feature but feel free to play around with it and suggest any changes/additions you think would help with your specific use case and we'll see what we can do.

[ckp95]

Can't it just be

def do(cls, coroutine):
    def _do_internal(*args, **kwargs):
        gen = coroutine(*args, **kwargs)
        ...

So that the coroutine can take arguments?

[jasondelaat]

You could. My thinking is that if you do that though then what you get back is a new function/generator and not a Maybe (or whatever) value. Or a function/generator wrapped in a monad. Not really sure which is best.

You can always wrap the do block to inject values.

def a_function(a, b):
    @Maybe.do
    def do_block():
        x = yield Just(a)
        y = yield Just(b)
        return x + y
    return do_block

Maybe that's less clear/convenient though.

We'd also have to change the return value from do:

@classmethod
def do(cls, coroutine):
   # ...
   return _do_internal   # was return _do_internal()

Otherwise we can't actually pass it arguments at all. If we change nothing else then this doesn't work quite right.

@Maybe.do
def stacked():
    x = yield Just(1)
    y = yield Just(2)
    @Writer.do
    def write():
        z = yield Writer(x + y, f'adding {x} and {y}')
        return z
    return write

print(stacked)
print(stacked())
print(stacked().bind(lambda g: g())

<function Monad.do.<locals>._do_internal at 0x10d0fc268>
Just <function Monad.do.<locals>._do_internal at 0x10d0fcb70>
(3, adding 1 and 2)

The original seems clearer to me but I don't have a clear idea of your use case so your idea might be better. Do you have an example of how you want to use it by any chance?

[jleonard-r7]

Did anything ever come of this? I'd like to use do-notation rather than deeply nested if/else or a huge flatMap chain of lambdas. I found this: https://github.com/miiohio/ziopy but not particularly enamored with it given all the ZIO-specific stuff and the lack of clarity around how the do magic actually works (mostly the ZIO naming, environment, types, etc).

[jleonard-r7]

??

[jasondelaat]

Oh hey, sorry, I somehow missed your previous comment. But to answer your question: unfortunately no, nothing ever came of it. I've not done much with pymonad at all for quite a while because I've just not had the time or energy to do so. The only other thought I've had but not really explored is somehow using the 'with' statement to create a monad context to work within. But, like I said, I haven't really explored it.

…

On Mon, Mar 6, 2023 at 11:40 AM jleonard-r7 ***@***.***> wrote: ?? — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALD4GMTNYDI5TNKOFD3VQ6TW2YHQLANCNFSM4RST5VTQ> . You are receiving this because you commented.Message ID: ***@***.***>

[jleonard-r7]

Yea, something like the ExitStack could work but the problem would be getting the individual bindings. Otherwise, a normal context per binding would still result in "nesting hell" (which is the whole syntactic problem i'm mainly trying to solve).

[jasondelaat]

Yeah, that's pretty much the problem I run up against any time I take another look at this idea. There's always some sort of trade-off that makes it more of a pain than it's worth. I'm reasonably convinced that there's just not a good solution to the problem unfortunately.

…

On Mon, Mar 6, 2023 at 2:14 PM jleonard-r7 ***@***.***> wrote: Yea, something like the ExitStack could work but the problem would be getting the individual bindings. Otherwise, a normal context per binding would still result in "nesting hell" (which is the whole syntactic problem i'm mainly trying to solve). — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALD4GMUMCGBZ5KH5TK2DW3LW2YZS3ANCNFSM4RST5VTQ> . You are receiving this because you commented.Message ID: ***@***.***>

[jleonard-r7]

Well, at least not until this is passed:
https://peps.python.org/pep-0638/

[jasondelaat]

Indeed. Wasn't aware of that PEP, thanks for pointing it out. I guess it's true, everything really does become LISP sooner or later!

…

On Mon, Mar 6, 2023 at 3:16 PM jleonard-r7 ***@***.***> wrote: Well, at least not until this is passed: https://peps.python.org/pep-0638/ — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALD4GMWM5ND56DW4KQYDAOLW2ZA27ANCNFSM4RST5VTQ> . You are receiving this because you commented.Message ID: ***@***.***>

[jleonard-r7]

Well except that LISP’s homoiconocity would come in really nicely for this stuff.

But yea I’ve long considered these scripting languages to be mere imperfect reflections of LISP.

[MichaelSchneeberger]

I implemented a do decorator that works for a general Python object (that implements a flat_map or bind method) based on AST manipulation.

The following nested code runs (surprisingly) without monad transformers:

from donotation import do

from pymonad.maybe import Just
from pymonad.writer import Writer

pymonad_do = do(attr='bind')

@pymonad_do
def stacked():
    x = yield Just(1)
    y = yield Just(2)

    @pymonad_do
    def inner_write():
        yield Writer(x + y, f'adding {x} and {y}')

    return inner_write()

# Output will be (3, adding 1 and 2)
print(stacked())

https://github.com/MichaelSchneeberger/do-notation

Let me know what you think.