Decoding problem / Malformed Response Data

[xtekky]

I saw that several issues were raised, and most solved using the new additional_decode parameter, however this does not work for me, but everything works fine using the normal requests library.

I tried removing / setting the "accept-encoding" header to combinations of gzip, deflate, br also while using the additional_decode parameter. All of those did not work.

correct data (using requests):

08a48c9081041002280432b001b87f0e66703ee53e57472fe9bdce1240438204fd036b7726390d4d9d8492d70bed6715e2be7a02283179bedc6eb19451c3cdaa3aeb054581723fc1dc62019471c676df6f33b2ee39589c1316cfdcc43da4c451026c6e432e9a76c1a4f87abb27167ae9da16831f22eacc81e08ef71dd0d8242ca0b865c3121dffaea6857e09b36322f7f213689f52130c919d8497bf230e3c5f4e889c4da0120e39fe70e2f880a5064d6de50a6003884fa5d4ce7ae646

incorrect:

08efbfbdefbfbdefbfbdefbfbd041002280432efbfbd01efbfbdefbfbdefbfbd3fefbfbd6dc2bfefbfbd443074efbfbd3a66efbfbdefbfbd31efbfbdefbfbd08c3bf1a42efbfbd26efbfbd083e2defbfbd5f6cefbfbdefbfbdefbfbdefbfbd1eefbfbd6169efbfbdefbfbdefbfbdefbfbd743befbfbdefbfbd4fefbfbdefbfbdcbb45a500cefbfbd5e4befbfbd49efbfbd4a1c7fefbfbd353defbfbd5fefbfbd1defbfbdefbfbdefbfbdefbfbdda8fefbfbd6f494e1430efbfbd42eaaf9fdf9e06efbfbdefbfbd79346cefbfbd4635efbfbdef809f2eefbfbdefbfbdefbfbd6a29efbfbdefbfbd4e0945efbfbd671148efbfbd70efbfbd7f17462cdc9eefbfbd68efbfbdefbfbd38046632585cefbfbd463900226340efbfbd0befbfbd16efbfbdefbfbdefbfbd79efbfbd192077713e35efbfbd576c4662efbfbd2eefbfbd0defbfbd2752efbfbd45

there is a repetition of efbfbd which corresponds to the Unicode Replacement Character in UTF-8encoding (U+FFFD).
Replacing it with nothing wont solve the issue

any fixes ?

p.s.: the request made was to tiktokv.com and the response is a protobuf (in bytes).
The response header indicates that the content is application/octet-stream, which i suppose requests decodes and this library does not

[shellcmd89]

the response.content contains many repeat special characters \xef\xbf\xbd, what I thought is if we can delete them.

append: success delete \xef\xbf\xbd from response.content, but the value change too much, still can't parse. so I think there's something wrong in handling raw binary data in tls-client since standard requests/curl-cffi/httpx all give the correct response. what a pity.

[AmericanY]

@xtekky did you reached any solution?

[xtekky]

unfortunately no, but it does work with curl_cffi, httpx, requests etc.. so you may try those libraries.

[enodr]

I can confirm having exactly the same bug on binary content

[iamtorsten]

Fixed here
https://github.com/iamtorsten/tls-client

Py Package tls-client2