Runtime errors when launching many shell subprocesses.

[ghost]

Here's the code to reproduce those issues:

import uvloop
from asyncio import set_event_loop_policy, create_subprocess_shell, gather, get_event_loop
from asyncio.subprocess import PIPE

async def run_cmd():
    length = 1000000
    process = await create_subprocess_shell("head -c %s /dev/urandom" % length, stdout=PIPE)
    output, err = await process.communicate()
    assert len(output) == length

set_event_loop_policy(uvloop.EventLoopPolicy())
loop = get_event_loop()
loop.set_debug(True)
runners = [run_cmd() for i in range(50)]
loop.run_until_complete(gather(*runners))

The exact runtime error raised depends on where I run it:

This raises multiple RuntimeError: Cannot pause_reading() when closing:

• uvloop version: 0.8.0
• Python version: 3.6.1
• Platform: linux 4.7.8-1 i686 pae, backport, Debian 8
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: yes

On the same system with python 3.5.3 this runtime error isn't raised but the assertion fails.

This raises multiple exceptions such as RuntimeError: unable to perform operation on <ReadUnixTransport closed=True reading=False 0x7efe0d848d00>; the handler is closed when using /dev/zero instead of /dev/urandom and only needs 2 parallel subprocesses (1 works):

• uvloop version: 0.8.0
• Python version: 3.6.1
• Platform: linux 3.16.0-4-amd64, Debian 8
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: yes

It also raises multiple times RuntimeError: Cannot pause_reading() when closing if run_cmd is launched 200 times instead of 50. Note that this system is on a server with 16G of RAM, while the first one only has 4G (if that makes any difference).

Running the same code with asyncio's default loop works in any case.

[1st1]

Interesting. Thanks for providing a code to reproduce this, I'll take a look.

[kura]

Might be worth doing a couple of things;

1. Check what your max processes and max open file descriptors are set to. You can do this with ulimit -a.
   I'd say while it's unlikely you'll be hitting max processes. if you're running this quite frequently during testing you may easily hit the open files limit, especially if it's causing unclean process shutdown.
2. Check how much entropy you have in urandom. That may be doing things that cause bad behaviour or hanging for a process. I've seen weird cases where distros mess with urandom that are really not nice.

FYI, I tested this code on a high spec test rig, found it failed fairly frequently when set to length of 1000000, so I tested it hundreds more times with a random delay between test periods with different length values. All of my tests had zero errors up to and including a length of 100000, I tried 250000, 500000, and 750000, all three of these lengths resulting errors with the default range(50) provided in your example.

• length 10000 fails 0
• length 50000 fails 0
• length 100000 fails 0
• length 250000 fails 91/100
• length 500000 fails 89/100
• length 1000000 fails 92/100

[ghost]

@kura Well using /dev/urandom was just a cheap way to open file descriptors and generate data, the same problems appear on my computer with the command "yes | head -c %s" and a length of 10,000,000, even after removing the limits on the number of open file descriptors.

[1st1]

Fixed in master. A simple race between transports and high-level asyncio streams. A new uvloop version with a fix will be released soon.