Feature/impersonate 6.0

[perklet]

TODOs

• add chrome 116 - 120
• support websockets

ETA:

https://github.com/users/yifeikong/projects/1/views/3

Update:

Delayed other build options, none of them can be built directly...

[T-256]

I think this section could be improved

[T-256]

you could be more explicit here to why not supported others.

[Kwsswart]

I agree, it would be great to have some documentation explaining why we don't support firefox or other ones supported by the curl_impersonate

[Kwsswart]

Possibly simply adding a link to #59 (comment) will be enough

[perklet]

Appologies for the missunderstanding, I'll make sure it's well documented in the new version.

[Kwsswart]

Please never apologize! Thank you for the work on this mate its an amazing repository! Only reviewing to try help somewhat ^^

[T-256]

nit: remove commented print

[T-256]

You need also update repo address at here:
https://github.com/yifeikong/curl_cffi/blob/e53593e0d7f660479a8b18aaa6e1b18decbb9fbd/Makefile#L18

[T-256]

FYI I added win32 build for testing in my fork

[Snusmumr1000]

FYI, WebSocket support doesn't seem to work properly.

import asyncio
from time import sleep

from curl_cffi import requests

with requests.Session() as s:
    w = s.connect("ws://localhost:8765")
    w.send(b"Foo")
    reply = w.recv()
    print(reply)
    assert reply == b"Hello Foo!"

    w.send(b"Bar")
    reply = w.recv()
    print(reply)

    assert reply == b"Hello Bar!"


async def async_examples():
    async with requests.AsyncSession() as s:
        w = await s.connect("ws://localhost:8765")
        await w.asend(b"Bar")
        reply = await w.arecv()
        print(reply)
        await w.asend(b"Test")
        sleep(5)
        reply2 = await w.arecv()
        print(reply2)
        assert reply == b"Hello Bar!"
        assert reply2 == b"Hello Test!"


asyncio.run(async_examples())

consider this test, it seems there are some problems with reading incoming WebSocket messages after the first one was accepted.

[perklet]

@Snusmumr1000 That's why it's still a draft.

[Snusmumr1000]

@yifeikong ok, but that wasn't mentioned anywhere, so probably somebody will find this precaution useful.

[Kwsswart]

Currently curl_impersonate only supports until version 116, are we not worried already providing support to 120 when it doesn't handle this yet?

Ref. https://github.com/lwthiker/curl-impersonate?tab=readme-ov-file#supported-browsers

On that note would it not make more sense to offer support for firefox?

[perklet]

According to the usage table here, most users are using the latest versions of Chrome and Safari. For strict blocking strategy, it's reasonable to just block users with any older versions of browsers.

116 is mucher new than 110, but it does not make things significantly better, let alone that their fingerprints are actually the same. The insteresting part is in 117, when ECH was added.

Actually I have been working on this in my fork of curl-impersonate. Hopefully I could get it landed before Chrome 120 is main stream. I'm just too busy on other stuff recently.

[perklet]

As of firefox, it's really challenging to pack an addtional .so file in a python wheel. There are two options to bypass this:

1. release another package, i.e. curl_cffi_ff, as suggested by one of our users
2. Try to use boringssl(chrome) to emulate nss(firefox)

At least one of them should work, just haven't had time to try them out. Maybe I can experiment them during the Chinese New Year.

[Kwsswart]

I do agree that it would be nice to have the Chrome version117+ available sooner as this will help a lot more with the more challenging sites. (Already saw you are well on the way through all the different versions there on your fork.)

As for firefox probably the easier of the two options you mentioned would be to simply release a new package for firefox curl, but this would require maintenance of both packages simultaneously which seems a lot more effort on your part.

I would love to get closer to this project although I am extremely new to it, if there are any smaller issues for me to explore and help out on let me know and will try tackle it in my free time

[bjia56]

I would be open to trying out the multiple package option. The opencv_python project builds 4 different packages (each with slightly different configurations) out of the same base repo, so I think it should be possible to minimize the maintenance overhead.

[bjia56]

A related option could possibly be to factor out the ffi binding portion into its own standalone package, build chrome/firefox versions of that, and have curl_cffi import the bindings packages. This way, the requests/async interfaces that curl_cffi provides don't need be duplicated

[T-256]

At here, IMO, simulating NSS on BoringSSL could have more priority than maintaining multiple packages.
This may need to have some patches on BoringSSL, but I think it's worth to try investigating on it.

[perklet]

Try whatever you like, I'm open to merge them both since there is no conflict, actually.

[perklet]

@Snusmumr1000 It seems that the local echo server impelmentation is behaving wrongly. Our implementation is actully correct. See the latest commit for fixed server and updated tests and examples.