Ability to use a proxy for http testing

[spirillen]

Is your feature request related to a problem? Please describe.
IP getting marked as bots/open proxies by external sources like cloudflare

Describe the solution you'd like
This can be overcome by appending an external proxy-server for the HTTP response code

This should all ready be supported by the urllib3 and should be possible to put into the .pyfunceble-env as it should be supporting --user and --password, there could easily be a pre-configured proxy server list either locally or from URI.

Example string

--proxy protocol(s)?://<USER-NAME>:<PASSWORD>@ip|host:port
OR
--proxy-list list-of-proxy-servers-pre-configured

Another thing, for which we might wan't to wait implanting the proxy a bit, is we should actually append conditional rules such as:

socks5://127.0.0.1:9050, TLD={onion]
http://localhost:3128,DOMAINS={amarican-fakenews,someother.non.eu.gdpr-blocked.site}

Additional context
Pros:
Ofload the traffic from own network
Tests of the tor network *.onion*
Removes any cookies
Enhancing privacy

Cons:
User might choose unwisely and uses "open" proxies which manipulate results.

Internal PROS:
We can put up a "public" proxy server from which we scrape the log for uri's that later can be tested for track-, ads- and phishing (unwanted) code

[spirillen]

APPENDIX

A lists of proxies should probably support some round-robin/randomness and weight (preferred) e.g. re-use the weight + order syn-taxing from PowerDNS's dnsdist https://dnsdist.org/reference/config.html?highlight=weight#servers & https://dnsdist.org/guides/serverselection.html?highlight=weight#built-in-policies

[funilrys]

Note about this: The requests module follow the HTTP_PROXY and HTTPS_PROXY environment variable.
Basically putting it into one of those files should be enough:

$cur_dir/.env
$cur_dir/.pyfunceble-env
$pyf_config_dir/.env
$pyf_config_dir/.pyfunceble-env

But, I agree, we should be able to read a proxy and choose one randomly before any query though the HTTP requester ...

[funilrys]

Note about the implementation: It should be better to have the --http-proxy and --https-proxy arguments.

But, 2 new environment variables will be introduced: PYFUNCEBLE_HTTP_PROXY and PYFUNCEBLE_HTTPS_PROXY which will be also parsed into the system.

[spirillen]

Note about the implementation: It should be better to have the --http-proxy and --https-proxy arguments.

But, 2 new environment variables will be introduced: PYFUNCEBLE_HTTP_PROXY and PYFUNCEBLE_HTTPS_PROXY which will be also parsed into the system.

And if you are using a socks5 proxy like tor socks5://127.0.0.1:9050 ? 😃

[spirillen]

Updated OT

Another thing, for which we might wan't to wait implanting the proxy a bit, is we should actually append conditional rules such as:

socks5://127.0.0.1:9050, TLD={onion]
http://localhost:3128,DOMAINS={amarican-fakenews,someother.non.eu.gdpr-blocked.site}

See also; #51

[funilrys]

Note to self: Since the introduction of the .PyFunceble.overwrite.yaml file, it is not necessary nor wanted to create an extra file for this. We shouldn't add some "hidden" or "secret" files.

We should implement this feature through a new configuration section/index/key. It's way easier to maintain persistency now than earlier thanks to the .PyFunceble.overwrite.yaml configuration file that is never maintained by PyFunceble (itself).

[funilrys]

Note to self after thinking about this .... With the complex schema described in #132 (comment), the implementation shouldn't provide any --argument. Everything regarding that complex this should be done from the configuration file (cf: previous comment).

The reason behind this is the complexity that is going up. For most people, the HTTP_PROXY and HTTPS_PROXY environment variables should be sufficient. For those "normal" people, a --http-proxy and --https-proxy (single input) should be sufficient.

[spirillen]

Note to self after thinking about this .... With the complex schema described in #132 (comment), the implementation shouldn't provide any --argument. Everything regarding that complex this should be done from the configuration file (cf: previous comment).

The reason behind this is the complexity that is going up. For most people, the HTTP_PROXY and HTTPS_PROXY environment variables should be sufficient. For those "normal" people, a --http-proxy and --https-proxy (single input) should be sufficient.

You are still missing the --SOCK5-proxy to support i.e. the *.onion and other country specific domain That can't be access (nor should) do to the GDPR, by which many American sites have blocked access from EU IP-adresses.

[funilrys]

Hey @spirillen, with the requests python module, a sock:// URL can be passed as HTTP_PROXY or HTTPS_PROXY if you install requests[sock] 😋

[spirillen]

Super @funilrys didn't know that. I'm used to be a bit more specific about the protocols 😏

[spirillen]

As I'm sitting here and have just done some clean up the last few days, and Tor is switching to the TorV3 protocol for hidden services, it strikes me, could we make PyFunceble skip testing of onions?? 🧅 till a proxy module is finally implanted.

[funilrys]

Hey @spirillen, the TLD thing is implemented and will be deployed as soon as I'm done testing it.

Is the domain rules/condition that you are requesting really a necessary thing ? I couldn't find a real use case for that..

Stay safe and healthy!

[funilrys]

@spirillen, feel free to let me know if the domain rules/condition is necessary - from your standing point.

[spirillen]

Thanks a lot for this huge add to the code, One question comes to mind, can you set a sock5 proxy too, this would open up for testing .onions and yes the rules are actually requested if we will be able to fully test IE onion/russian/Chinese domains as well

Found the answer here... c30c466#diff-4ac672a24850174ba4915519c73ac7797ff76b93f0255c2d8b9f9abce88a7ba9R75-R78