[Feature request] Data streaming

[acheong08]

Similar to Python requests: https://stackoverflow.com/questions/60343944/how-does-requests-stream-true-option-streams-data-one-block-at-a-time

[GunGunGun]

Streaming would be great to have because it allows user to make decision about skipping body data or buffer fetching to print result to client, which is great in many situations:

• Content is too big, like 2GB and memory limitation like 2GB RAM, then if you fetch everything into memory you'll get OOM error
• In proxy environment, you may want to abort the body data if the response header doesn't match some of your criterias

I'll try to look into the code and implement this feature, shouldn't be that hard considering we're fetching data from Go TLS Client which probably allow data streaming.

This line going forward should be what we need to modify: https://github.com/FlorianREGAZ/Python-Tls-Client/blob/master/tls_client/response.py#L40

[GunGunGun]

Actually after researching the API of tls-client by bogdanfinn, I can't find any information about stream being made available, so it looks like to have this stream feature we first need to do something with the core first, which is the Go Client https://github.com/bogdanfinn/tls-client-api

And... tls-client actually sends back JSON data instead of standard HTTP response with headers \r\n\r\n then body, which is ways easier to handle streaming (the same way requests and urllib3 are doing). Something like this, which is pretty pointless to stream data if you want to send buffer to something like web browsers, in this case you need to parse JSON data anyways, then send to web browsers:

{....,"headers":{"Access-Control-Allow-Credentials":["true"],"Access-Control-Allow-Origin":["https://rr6-sn-8qj-i5o6k.gooqlevideo.com"],"Age":["72715"],"Alt-Svc":["h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400"],"Cache-Control":["max-age=31536000"],"Cf-Cache-Status":["HIT"],"Cf-Ray":["79f028e9cf569e28-SIN"],"Content-Encoding":["br"],"Content-Security-Policy":["default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;form-action 'self';frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests"],"Content-Type":["text/html; charset=utf-8"],"Cross-Origin-Embedder-Policy":["require-corp"],"Cross-Origin-Opener-Policy":["same-origin"],"Date":["Sat, 25 Feb 2023 11:39:19 GMT"],"Expect-Ct":["max-age=0"],"Origin-Agent-Cluster":["?1"],"Referrer-Policy":["no-referrer"],"Server":["cloudflare"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"],"Vary":["Accept-Encoding","Origin"],"X-Content-Type-Options":["nosniff"],"X-Dns-Prefetch-Control":["off"],"X-Download-Options":["noopen"],"X-Frame-Options":["SAMEORIGIN"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Powered-By":["centminmod"],"X-Xss-Protection":["0","1; mode=block"]},"cookies":{"__cf_bm":"yxyOVhUi38n1UfQujprACZl5yxXyhYZMYl6boMzKKh0-1677324536-0-AdwLPV4avEeaFZNNLGA7VKzwHM3C2rbm2jL/SPwv1WNoRDaahAKij0hWwlApbK52ErrqT3pGG+pCYWIZlOzkM9o="}}'

[FlorianREGAZ]

Hey, I don't think it'll be possible to add this soon sadly. :/