Fix multi-header compatibility with Scrapy #14
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ImpersonateDownloadHandler._download_request()
currently returns ascrapy.http.Response
where multiple headers with the same key are concatenated into a single comma separated value. This means that for example,response.headers.getlist('Set-Cookie')
returns a list with 1 item even if multipleSet-Cookie
headers were present in the HTTP response.To be compatible with Scrapy's built-in
HTTP11DownloadHandler
andH2DownloadHandler
,response.headers.getlist(key)
should return multiple values when multiple headers with the samekey
are present in the HTTP response.To fix this, we can pass
curl_cffi.requests.Headers.multi_items()
to Scrapy'sHeaders()
constructor.