HTTP GET + compress=True = 501

[a-urth]

There are some strange behavior with proxies. Trying to do HTTP GET request, requests package gives correct result, asyncio - 501.

Response headers:
SERVER - squid/3.1.4
MIME-VERSION - 1.0
DATE - Mon, 14 Jul 2014 20:20:15 GMT
CONTENT-TYPE - text/html
CONTENT-LENGTH - 3136
X-SQUID-ERROR - ERR_UNSUP_REQ 0
VIA - 1.0 host (squid/3.1.4)

Proxy - 173.234.226.20:8800

Seems that GET method for such proxy is incorrect, even if request done through http.
Any suggestions?

[popravich]

Can you add code showing how you are making the request?

I've just tryied the following code and both requests and aiohttp got 403 (consider this is ok):

>>> import asyncio
>>> import aiohttp
>>> import requests

>>> loop = asyncio.get_event_loop()
>>> resp = loop.run_until_complete(
...    aiohttp.request('GET', 'http://httpbin.org',
...                    connector=aiohttp.ProxyConnector('http://173.234.226.20:8800')))
>>> resp
... <ClientResponse [403 Forbidden]>

>>> requests.get('http://httpbin.org', proxies={'http': '173.234.226.20:8800'})
... <Response [403]>

[a-urth]

Hm... the thing was not in proxy, its about compress=True
Sorry for such inconvenience, is it possible because of firewall?

@asyncio.coroutine
def get(url):
    proxy = random.choice(PROXIES)
    response = yield from aiohttp.request(
        'GET', url, compress=True, connector=aiohttp.ProxyConnector(proxy=proxy)
    )
    print("{0} - {1} - {2}".format(url, response.status))
    return (yield from response.read_and_close())

if __name__ == '__main__':
    # Proxies initializing

    loop = asyncio.get_event_loop()
    tasks = [get(url) for url in get_records()]
    loop.run_until_complete(asyncio.wait([get(url) for url in get_records()]))

[popravich]

Aha, I see.

Making 'GET'-request with compress=True causes aiohttp to add extra request headers --
Content-encoding: deflate and Transfer-encoding: chunked.
It seems that GET + Transfer-encoding: chunked pair makes squid nervous so its might be buggy aiohttp behavior.

We will into this.

[fafhrd91]

well, nervous is wrong word, buggy is right :)
we can check in client if there is no body and omit content-encoding and transfer-encoding headers

[fafhrd91]

is it still the problem?

[danielnelson]

I think it must be still an issue based on comments in #835.

[fafhrd91]

@danielnelson what to fix it?

[danielnelson]

sure

[asvetlov]

Fixed by #891

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.