aiohttp.client_exceptions.ClientResponseError: 400, message='invalid character in header' #5443

Evanyifan · 2021-01-27T14:53:54Z

🐞 Describe the bug

When I want to use the put method to access a public network service, because the target operating system is windows, there is a problem of garbled code in the reason field of the response packet, which causes the above error. Stable reproduced in the aiohttp-3.7.3 version。

💡 To Reproduce

import aiohttp
import asyncio


async def fetch(session, url):
    async with session.put(url) as response:
        return await response.text()


async def main(url):
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, url)
        print(html)


loop = asyncio.get_event_loop()

# 生成多个请求方法
url ="http://118.25.92.249:8161/fileserver/840f12/425ba"
tasks = [main(url)]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

💡 Expected behavior

📋 Logs/tracebacks

📋 Your version of the Python

$ python --version
3.8
...

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
3.7.3
...

$ python -m pip show multidict
...

$ python -m pip show yarl
...

📋 Additional context

andrey-tikhov · 2021-02-26T09:10:46Z

Same issue here.

aiohttp==3.7.3
Python 3.6.4

$ python -m pip show multidict
Name: multidict
Version: 5.0.2
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /Users/andrey/Documents/am_foundations_service_bot/env/lib/python3.6/site-packages
Requires: 
Required-by: yarl, aiohttp

$ python -m pip show yarl
Name: yarl
Version: 1.6.3
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /Users/andrey/Documents/am_foundations_service_bot/env/lib/python3.6/site-packages
Requires: multidict, idna, typing-extensions
Required-by: aiohttp

Using requests for the same web-address works successfully.

webknjaz · 2021-03-06T22:12:59Z

@Evanyifan your server at 118.25.92.249:8161 produces a corrupted HTTP response. And since it's invalid, it's impossible to parse it as an HTTP message. That is why an appropriate exception is raised:

$ telnet 118.25.92.249 8161                 
Trying 118.25.92.249...
Connected to 118.25.92.249.
Escape character is '^]'.
PUT /fileserver/840f12/425ba HTTP/1.1
Host: 118.25.92.249:8161
User-Agent: curl/7.73.0
Accept: */*

0)TP/1.1 500 E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (~
Content-Type: text/html; charset=iso-8859-1
Content-Length: 3466
Server: Jetty(6.1.9)

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 500 E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (???????????)</title>
</head>
<body><h2>HTTP ERROR: 500</h2><pre>E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (???????????)</pre>
<p>RequestURI=/fileserver/840f12/425ba</p><h3>Caused by:</h3><pre>java.io.FileNotFoundException: E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (???????????)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
	at org.apache.activemq.util.RestFilter.doPut(RestFilter.java:176)
	at org.apache.activemq.util.RestFilter.doFilter(RestFilter.java:89)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
	at org.apache.activemq.util.FilenameGuardFilter.doFilter(FilenameGuardFilter.java:46)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:726)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:205)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
	at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)
</pre>
<p><i><small><a href="http://jetty.mortbay.org/">Powered by Jetty://</a></small></i></p><br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                

</body>
</html>

Specifically,

0)TP/1.1 500 E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (~

looks like a leak from a corrupted in-memory object or something like that.

It's not a problem in aiohttp: the solution is to make your server return an actual valid HTTP response that does not have any garbage bytes in the status-line. If you need to just ignore such cases in your code, use try/except.

andrey-tikhov · 2021-03-07T06:56:49Z

It’s a good question. In my case sync python requests are parsing such response properly while aiohttp fails. So we can stay on the point that aiohttp is doing ok or we can try to make implement a solution that follow the most popular sync tool. Andrey Tikhov

…

6 марта 2021 г., в 23:13, Sviatoslav Sydorenko ***@***.***> написал(а): @Evanyifan your server at 118.25.92.249:8161 produces a corrupted HTTP response. And since it's invalid, it's impossible to parse it as an HTTP message. That is why an appropriate exception is raised: $ telnet 118.25.92.249 8161 Trying 118.25.92.249... Connected to 118.25.92.249. Escape character is '^]'. PUT /fileserver/840f12/425ba HTTP/1.1 Host: 118.25.92.249:8161 User-Agent: curl/7.73.0 Accept: */* 0)TP/1.1 500 E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (~ Content-Type: text/html; charset=iso-8859-1 Content-Length: 3466 Server: Jetty(6.1.9) <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> <title>Error 500 E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (???????????)</title> </head> <body><h2>HTTP ERROR: 500</h2><pre>E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (???????????)</pre> RequestURI=/fileserver/840f12/425ba<h3>Caused by:</h3><pre>java.io.FileNotFoundException: E:\apache-activemq-5.3.0\webapps\fileserver\840f12\425ba (???????????) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.activemq.util.RestFilter.doPut(RestFilter.java:176) at org.apache.activemq.util.RestFilter.doFilter(RestFilter.java:89) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at org.apache.activemq.util.FilenameGuardFilter.doFilter(FilenameGuardFilter.java:46) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:726) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:828) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:205) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450) </pre> <a href="http://jetty.mortbay.org/">Powered by Jetty://</a> </body> </html> It's not a problem in aiohttp: the solution is to make your server return an actual valid HTTP response that does not have any garbage bytes in the status-line. If you need to just ignore such cases in your code, use try/except. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

webknjaz · 2021-03-07T09:26:51Z

It's not a real response status but a synthetic one. The server is broken and the exception is a natural way of communicating this. It would be weird if the client pretended that everything is fine. It would give a false impression that the HTTP request is constructed incorrectly which is simply not true. The current design is the best way of communicating this situation unambiguously.
It sounds like requests does the wrong thing or maybe you misunderstood what it actually does. FWIW there's not enough info to make such a judgment because you didn't provide any proof.

andrey-tikhov · 2021-03-07T19:27:03Z

Nice. Now blame that everyone is lying but you. I can’t provide proof because my calls are made to private API and contain privacy restricted data which I obviously can’t share in the public github. But I will see what I can do because the situation when requests are working just fine and aiohttp throws exceptions doesn’t look normal to me. Andrey Tikhov

…

7 марта 2021 г., в 10:27, Sviatoslav Sydorenko ***@***.***> написал(а): It's not a real response status but a synthetic one. The server is broken and the exception is a natural way of communicating this. It would be weird if the client pretended that everything is fine. It would give a false impression that the HTTP request is constructed incorrectly which is simply not true. The current design is the best way of communicating this situation unambiguously. It sounds like requests does the wrong thing or maybe you misunderstood what it actually does. FWIW there's not enough info to make such a judgment because you didn't provide any proof. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

Dreamsorcerer · 2021-03-07T21:30:01Z

Because another framework will ignore and workaround the error doesn't mean that it should (or would) work in all other platforms.

For example, if I put invalid HTML on a web page, most browsers will be able to ignore and work around the error. But, different browsers may do different things and render completely different content, and other tools may error completely, because there is no defined behaviour for invalid HTML.

The same thing applies here, maybe other frameworks make a best guess at the correct behaviour when encountering this error, and it works well enough for you. But, aiohttp simply throws an exception as the response is not valid, rather than taking a best guess approach (which may or may not produce the desired outcome).

devblack · 2021-04-14T22:16:05Z

AIOHTTP_NO_EXTENSIONS=1 python3 my_script.py

Evanyifan added the bug label Jan 27, 2021

Evanyifan mentioned this issue Jan 27, 2021

Server returns 400: "invalid character in chunk size header" for valid request #5220

Closed

webknjaz removed the bug label Mar 6, 2021

webknjaz closed this as completed Mar 6, 2021

webknjaz added invalid This doesn't seem right wontfix labels Mar 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aiohttp.client_exceptions.ClientResponseError: 400, message='invalid character in header' #5443

aiohttp.client_exceptions.ClientResponseError: 400, message='invalid character in header' #5443

Evanyifan commented Jan 27, 2021 •

edited by webknjaz

Loading

andrey-tikhov commented Feb 26, 2021 •

edited by webknjaz

Loading

webknjaz commented Mar 6, 2021 •

edited

Loading

andrey-tikhov commented Mar 7, 2021 via email

webknjaz commented Mar 7, 2021

andrey-tikhov commented Mar 7, 2021 via email

Dreamsorcerer commented Mar 7, 2021

devblack commented Apr 14, 2021

aiohttp.client_exceptions.ClientResponseError: 400, message='invalid character in header' #5443

aiohttp.client_exceptions.ClientResponseError: 400, message='invalid character in header' #5443

Comments

Evanyifan commented Jan 27, 2021 • edited by webknjaz Loading

andrey-tikhov commented Feb 26, 2021 • edited by webknjaz Loading

webknjaz commented Mar 6, 2021 • edited Loading

andrey-tikhov commented Mar 7, 2021 via email

webknjaz commented Mar 7, 2021

andrey-tikhov commented Mar 7, 2021 via email

Dreamsorcerer commented Mar 7, 2021

devblack commented Apr 14, 2021

Evanyifan commented Jan 27, 2021 •

edited by webknjaz

Loading

andrey-tikhov commented Feb 26, 2021 •

edited by webknjaz

Loading

webknjaz commented Mar 6, 2021 •

edited

Loading