-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
403 "Please enable cookies." when Request a Special Cloudflare Website #10141
Comments
It is a quite strange problem. It must be Aiohttp's internal defect. I do not think you would like to solve this "small" problem. It is time to use HTTPX and FastAPI. I guess I will never use Aiohttp more. |
Because you expect your issue to be fixed in under 2 hours...? It's not like you've paid for an SLA... To debug, you'll probably want to compare what actual data is being sent between the working and non-working code and see what the difference is. A simple way to do that is to use netcat (e.g. on Ubuntu: Compare the 2 requests and let us know what the difference is. Maybe there's some difference in cookie quoting or similar which confuses Cloudflare. |
Yes, I agree with you that there is a difference in actual behavior between Aiohttp and others. I have moved my business to HTTPX in recent hours. I am not calling an SLA here. It is not my obligation to guarantee the availability of Aiohttp. What do I want to do is just reporting it. |
Well, if you're not going to look at the issue and provide the details needed for us to fix it, then there's not much we can do... |
I cannot reproduce it with your script, so it may not be aiohttp's problem. I suspect it is caused by your proxy or dns. aiohappyeyeballs may prefer a different dns record compared to other http clients, which can make aiohttp connect to a host that happens to be blocking your ip. |
According to my tests, this problem only occurs when using Aiohttp. Other clients, including Requests, HTTPX, Chrome, Postman and cURL, work fine. It should not be related to the proxy. Maybe because I installed Aiohttp with AioDNS. The problem might be caused by AioDNS, but I don't understand why Cloudflare says "please enable cookies". By the way, how does aiohappyeyeballs work in Aiohttp? Will it cause some DNS leaks when using proxy? The website is a Cloudflare protected website. Though aiohappyeyeballs prefers a different dns record, but is there any difference between Cloudflare Edge Nodes? |
DNS and aiohappyeyeballs shouldn't be a factor here as the proxy will be doing the resolution and connection decisions. It's very likely a difference in how cookie quoting is handled or lack of partitioned cookie support |
Additional InformationWhether I set the param |
I'm assuming this because I cannot reproduce your problem with or without a proxy... Both aiohttp and requests works well. And it is possible to have difference between ipv4 and ipv6 nodes when cf is blocking your ipv6 address while accepting ipv4. By the way I don't think "please enable cookies" is a valid error message. It seems to be just a html class attribute used in cf page. If you try to manually trigger a rate limit or something, you can also see this in response. |
Which is what I asked for. If you can tell us the difference between a working request and a non-working one, we can figure out what is happening. Without that, there's nothing we can do here... |
@CodingMoeButa I finally reproduced your problem and did a network traffic analysis. I believe your problem is because one of the hosts in dns records (not all of them) is blocking requests without alpn. Please try aiohttp with a custom ssl_context: import ssl
ssl_context = ssl.create_default_context()
ssl_context.set_alpn_protocols(['http/1.1'])
ssl_context.check_hostname = False
...
async with session.get(url, headers=headers, ssl=ssl_context) as response:
... |
Thanks for your work. Why does Aiohttp not set ALPN by default? |
Because ssl.create_default_context() doesn't. It's something that has come up before though, so we should probably enable it (unless anyone knows why the ssl module doesn't do it by default?). |
For simplicity, we'll track this in #10152. |
Describe the bug
When I access the website using aiohttp, a 403 "Please enable cookies" error occurs. But when I use other clients to access it, it works fine.
To Reproduce
403
Expected behavior
200
Logs/tracebacks
Python Version
aiohttp Version
multidict Version
propcache Version
yarl Version
OS
Windows 11
Related component
Client
Additional context
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: