StreamReader.read truncates content

[nkaretnikov]

Long story short

Performing a GET request. Switched from read() to content.read(n=NUM_BYTES) (the streaming API) in order to limit the number of bytes read. The content gets truncated: the number of bytes returned is less than NUM_BYTES.

Expected behavior

Read up to NUM_BYTES.

Actual behavior

The number of bytes returned is less than NUM_BYTES.

Steps to reproduce

Run this script:

import aiohttp
import asyncio

NUM_BYTES = 10_000_000  # 10 MB

async def f(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            # content = await response.read()  # returns 22212
            content = await response.content.read(n=NUM_BYTES)  # returns 511
            print(len(content))


URL = 'https://upload.wikimedia.org/wikipedia/commons/thumb/7/71/2010-kodiak-bear-1.jpg/320px-2010-kodiak-bear-1.jpg'

asyncio.run(f(URL))

Your environment

• macOS
• Python 3.7.3
• aiohttp 3.5.4

Related issues

• There is no way to set limit size for response.read() #2638
• await content.read() reads partial content under high load #3881

[nkaretnikov]

The number of read bytes varies (test.py is the above script):

% for i in `seq 1 50`; do python3 test.py; sleep 0.1; done | sort | uniq -c
   2 1810
   1 1811
   1 1813
  24 510
   6 511
   9 513
   4 514
   2 515
   1 5710

Could be related to my environment, let me know if you can reproduce. Or maybe it's just that I'm using the API incorrectly.

[socketpair]

It's not a bug. It's by design. It reads no more than n bytes. It blocks (asynchronously) only if no data available at the moment.

[socketpair]

@asvetlov, I had PR that fixes that issue. We can introduce parameter or another function like read_greedy(n).