FEATURE: user_agent: random

[spirillen]

Description

As I'm working my way through the new PyFunceble.yaml I stumbled on the fact we are hard coding the use of browser and OS, Somehow this just seems wrong, as we know, then the big5 and their wannabees, are logging the IP/Browser/OS etc, and sharing these data, to fetch bots and gives us FP-responses on http codes, as we receives false 4xx, 5xx HTTPS codes.

Possible Solution

One of two

1. Randomize browser and OS per -n records (Optimal)
2. Randomize browser and OS per test starts

Considered Alternative

Manual labor, bad, the system is MY slave, not the other way around.

Additional context

UPDATE: typos

[funilrys]

Yeah, but what does random means for you?

Our user_agents are getting rotated every week - when not every day... And we choose a random one from one of the latest / most used ones:

PyFunceble/PyFunceble/dataset/user_agent.py

Lines 225 to 226 in fd2ce92

	if isinstance(result, (list, tuple)):
	return secrets.choice(result)

[spirillen]

And we choose a random one from one of the latest / most used ones:

I see tree questions here

1. Selection among browser + OS
2. How to control
3. You(me) ask because

---

1. Automatically rotate among all the available ones from the list (forgot the url), but you probably remember it. to the list of options
2. by a switch of Random vs chrome,Linux in the pyf.owerwrite.yml
3. Because as of now, 1 I didn't know it was rotated by default, 2. in pyf.owerwrite.yml. you can only set a firm value

[spirillen]

Come to think of... and YES it is a though only, can you spoof the mac addresses used for querying?

IF == True

Would it be an idea then to periodically with hourly randomness generate a set of "Clients, running its own MAC + Browser + OS"?
... IF == TRUE
Could this help making it looks like a random number of users on the same network vs one client from the same network?

[funilrys]

Well, look here under the @modern datasets. What you give PyFunceble is the keys to a list of user-agents.

I'll have to think of a way to implement randomness. That won't be the default though.

[funilrys]

Let's clarify ...

1. The reference file is rotated every few days: https://github.com/PyFunceble/user_agents/blob/master/user_agents.json
2. Every PyFunceble instance in the world tries to fetch that file; if the local version is older than 1 hour.

In the end-choice, user have to choices:

1. Set their own user agent through the user_agent.custom key:

PyFunceble/PyFunceble/data/infrastructure/.PyFunceble_production.yaml

Line 630 in 296a613

custom: null

2. Set a preferred user_agent.platform and user_agent.browser:

PyFunceble/PyFunceble/data/infrastructure/.PyFunceble_production.yaml

Lines 610 to 621 in 296a613

	# Set the browser to pickup.
	#
	# WARNING:
	# This parameter will be deprecated (soon?).
	#
	# Available Values: chrome, edge, firefox, ie, opera, safari
	browser: chrome
	# Set the platform to pickup.
	#
	# Available Values: linux, macosx, win10
	platform: linux

When end-user choose the later, the engine takes the given user_agent.platform as a subkey of the @modern key and the user_agent.browser as a subkey of the resulting dataset of the previous query. The result will be a list of user-agents. From that list, the engine will always select a random one.

PyFunceble/PyFunceble/dataset/user_agent.py

Line 255 in 296a613

secrets.choice(result),

That's how it works - right now.

So, you are suggesting, that we should an extra random layer / option ?

[spirillen]

Let me see if I understood this, for dummies.

Even I set a browser.agentX and platform.OXY, then the engine are mixing any of the @morern browsers, versions, etc

Then what about the OS.platform and screenSize?