Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix multi-header compatibility with Scrapy #14

Merged
merged 1 commit into from
Jun 7, 2024
Merged

Conversation

gg
Copy link
Contributor

@gg gg commented Jun 7, 2024

ImpersonateDownloadHandler._download_request() currently returns a scrapy.http.Response where multiple headers with the same key are concatenated into a single comma separated value. This means that for example, response.headers.getlist('Set-Cookie') returns a list with 1 item even if multiple Set-Cookie headers were present in the HTTP response.

To be compatible with Scrapy's built-in HTTP11DownloadHandler and H2DownloadHandler, response.headers.getlist(key) should return multiple values when multiple headers with the same key are present in the HTTP response.

To fix this, we can pass curl_cffi.requests.Headers.multi_items() to Scrapy's Headers() constructor.

@jxlil jxlil merged commit 6b43bd0 into jxlil:master Jun 7, 2024
@jxlil
Copy link
Owner

jxlil commented Jun 7, 2024

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants