Add automated CLI demonstrations #262

veracioux · 2021-10-17T00:14:42Z

This is a proof-of-concept implementation of my suggestion in #260.

I recreated the demos from the landing page of the website, each in its own script examples/demo/landing*.tut. They can be reproduced in the terminal as (using the first demo as an example):

tuterm landing1.tut -m demo

(tuterm is a dependency)

You can record them using asciinema as:

asciinema rec -c 'tuterm landing1.tut -m demo'

I tried to make the demos the same as they already are. But there are some inconsistencies with the output styles/colors and the --no-whois option is unrecognized.

I uploaded an asciicast of the last demo (landing3.tut) for your preview.

Please let me know if I can make improvements or add additional demos.

Signed-off-by: Haris Gušić <[email protected]>

spirillen · 2021-10-17T01:07:11Z

examples/demo/landing1.tut

+
+run() {
+    echo
+    c PyFunceble -a -d github.com


I would suggest using example.org (for good domains), example.com (for bad domains and example.org (for neutral domains) this would follow the practice in the docs. (https://github.com/funilrys/PyFunceble/blame/dev/docs/usage/terminal.rst#L9-L10)

In one of these examples I will suggest adding two domains.

Example

c pyfunceble -a -d example.com example.org

I personally think microsoft_google.com is better as a "bad" domain because it's more intuitive. No one expects microsoft and google to share a domain. But example.com wouldn't strike anyone as a bad domain until they actually check it. I do agree that example.org a suitable "good" domain.

This is just my opinion, and I will implement it the way you and the other maintainers want it of course.

You nailed it @veracioux! That's actually why I choose microsoft_google.com haha.

I can't say I in anyway disagree with any of you. It was just for the sake of the continually flow.

It also could be a help to new users, if naming is related to the docs, so people better can compare the readden word with the video.

spirillen · 2021-10-17T01:13:07Z

examples/demo/landing3.tut

+    c PyFunceble -u https://github.com/funilrys
+    c PyFunceble -u https://github.com/microsoft_google
+    c PyFunceble -u https://microsoft_google.com
+    c PyFunceble -u https://microsoft_google.com --local


Again, use the example.(com|net|org) #L8-L10

Actually, this is okay.

See #262 (comment)

For this demo specifically, I don't know how to use example.org and achieve the same effect as the existing demo does. I don't know how example.net handles URL paths.

You will get a 404 so my first though is to append the example. to the number of lines.

See also #262 (comment) for answer

spirillen · 2021-10-17T01:16:58Z

examples/demo/landing1.tut

+run() {
+    echo
+    c PyFunceble -a -d github.com
+    c PyFunceble -a -d github.com --no-whois # TODO doesn't work


The --no-whois is deprecated and replaces by --whois-lookup as also mentioned here #260 (comment) there have been several changes since the lasts recordings.

@spirillen @funilrys I did see the comment. I was just a bit confused that the --whois-lookup option actually disables whois. The option name suggests otherwise. Personally I think that --no-whois is a better name for the option. Maybe I'm doing something wrong. (The version of pyfunceble I used for the demos is 4.0.1)

@veracioux, it's because --whois-lookup will switch what people define in their configuration file (lookup.whois).

So if the lookup.whois index is set to true, using --whois-lookup will switch the internal value to false.
If the lookup.whois index is set to false, using --whois-lookup will switch the internal value to true.

spirillen · 2021-10-17T01:23:29Z

Kind a looks good for a first spin, In comment to your own comment for --no-whois is to run the pyfunceble --version as first command, this will mark which version was recorded. It can also help in the future if/when another argument it deprecated or replaced.

For example, there are some things going on in the background which will be available only from a sub version release and not put in a major release.

funilrys · 2021-10-17T11:04:38Z

This actually great!

Here is what is missing:

Just fix the --no-whois thing.
Is it possible to integrate this within our deployment workflow (AKA GitHub Actions Pipeline) ?

About the subject to use, I'm expecting the current one to be tested (cf: What can actually be seen in https://pyfunceble.github.io/#/). But using at least one example.(org|net|com) may be a good idea to start with (first input in demo).

Thank you for your time and contribution @veracioux !
Can't wait to see this working!!!

Stay safe and healthy!

funilrys · 2021-10-17T11:07:27Z

Note: I put this in the planning of 4.1.0 (due date: end of this year) but we may release this with another minor version if we are done before the end of the year!

veracioux · 2021-10-17T11:21:16Z

@spirillen

run the pyfunceble --version as first command, this will mark which version was recorded. It can also help in the future if/when another argument it deprecated or replaced.

I agree.

@funilrys

Is it possible to integrate this within our deployment workflow (AKA GitHub Actions Pipeline) ?

Quite possibly. I don't know very much about GitHub actions, but since this is just a glorified bash script, I currently see no reason why it shouldn't work.

Thank you for your time and contribution @veracioux ! Can't wait to see this working!!!

You're welcome.

Stay safe and healthy!

Thanks, you too.

Signed-off-by: Haris Gušić <[email protected]>

veracioux · 2021-10-17T18:06:03Z

I applied most of the suggestions.

One thing that still needs taking care of are the ipv4 addresses. They don't work right. No status is printed with them. Take a look at the TODO comments in landing_page_demo_2.tut. You could give me a new set of IP addresses to replace them with.

I added two scripts: record.sh and upload.sh in the same directory. record.sh will replay the commands in your actual terminal and create asciicasts from them, while upload.sh will upload them. You need to specify the demo files as arguments to these scripts.

You can run both scripts at once:

./record.sh *.tut && ./upload.sh *.tut

This will create and upload the scripts to asciinema and print the uploaded URLs to stdout.

Regarding CI, my guess is that in the CI pipeline you would have to:

install asciinema and tuterm
authenticate with asciinema somehow
upload the demos
delete the old demos from asciinema (or maybe you don't want to?)
update your website somehow to use the new URLs

You'll have to investigate yourselves, because I'm underqualified. I will assist wherever I can though.

Here's an idea for a side-project: create a GitHub action to do all of the above 😄 I might actually have started working on this, but again I'm underqualified.

spirillen · 2021-10-17T18:57:14Z

One thing that still needs taking care of are the ipv4 addresses.

Just a quicky, how about 9.9.9.9?

For the example.org+/path you'll receive a HTTP code 404 with this page

pyfunceble -u http://example.org/about

Subject                                                                                              Status      Source    
---------------------------------------------------------------------------------------------------- ----------- ----------
http://example.org/about                                                                             INACTIVE    STDLOOKUP

funilrys · 2021-10-27T02:46:54Z

Note to self: asciinema (CLI) can read an API token that is located inside ~/.config/asciinema/install-id.

WARNING: ~/.config/asciinema/install-id shouldn't end with a new line.

funilrys · 2021-10-27T02:52:25Z

Note to self: asciinema (CLI) output should be grep-ed to get the new URL. As far as I could test, there is no way to update the URL within asciinema.org.

From the perspective of this project, an automated GitHub Action Job, should:

clone the PyFunceble/pyfunceble.github.io project
switch to the react-src branch
rewrite the content of the src/data/asciinema_demo.json file.
deploy to GitHub pages.
push changes - made to the react-src branch.

veracioux · 2021-10-27T09:10:11Z

Sorry, I kind of pushed this PR aside. Can you tell me which IP addresses I should add into landing_page_demo_2.tut? The first IP is fine (added per @spirillen 's suggestion. But for the others I don't know what to use. Can you fill in the question marks for me?

PyFunceble -d 9.9.9.9
PyFunceble -d ?
PyFunceble -d ?

After this change, I think my work with this PR is done.

spirillen · 2021-10-28T21:16:42Z

Sorry, I kind of pushed this PR aside. Can you tell me which IP addresses I should add into landing_page_demo_2.tut? The first IP is fine (added per @spirillen 's suggestion. But for the others I don't know what to use. Can you fill in the question marks for me?

PyFunceble -d 9.9.9.9 # Neutral stabel
PyFunceble -d 2620:fe::fe # quad9 IPv6 (As I recall GHA now supports IPv6)
PyFunceble -d 194.187.99.221 # Adult CDN: https://mypdns.org/my-privacy-dns/porn-records/-/issues?scope=all&state=all&label_name[]=IP%3A%3ABlackListing
PyFunceble -d 45.136.204.40 # 23 phis: https://github.com/mitchellkrogza/phishing/pull/67
PyFunceble -u http://18.191.88.103/login.html # As we can do urls to, this is a inactive Phis

Just a few findings from blacklists projects

Signed-off-by: Haris Gušić <[email protected]>

veracioux · 2021-10-29T12:00:16Z

I added the missing commands per @spirillen's suggestion. The last two commands are a bit slow though. Could be due to my machine.

Let me know if there's more that I can do.

spirillen · 2021-10-30T19:59:58Z

examples/demo/landing_page_demo_2.tut

+    c PyFunceble -d 194.187.99.221                  # Adult CDN: https://mypdns.org/my-privacy-dns/porn-records/-/issues?scope=all&state=all&label_name[]=IP%3A%3ABlackListing
+    c PyFunceble -d 45.136.204.40                   # 23 phis: https://github.com/mitchellkrogza/phishing/pull/67


Why not merge these two lines to demonstrate how multiply values are committed at once?

~~c PyFunceble -d 194.187.99.221 45.136.204.40~~ 💭 🤔 , could that be an idea?? @funilrys and @veracioux

See https://github.com/funilrys/PyFunceble/pull/262/files/aa75d8886d6196b4dead17b815e04e6908b6306b..f7430a448961eb63856e3bc00f358d9b9176bc2f#r739693178 as the valid one ⏫

spirillen · 2021-10-30T20:09:02Z

examples/demo/landing_page_demo_2.tut

-    c PyFunceble -d 192.168.178.60 # TODO doesn't print the status
-    c PyFunceble -d 192.168.178.250 # TODO doesn't print the status
-    c PyFunceble -d 192.168.178.259
+    c PyFunceble -d 9.9.9.9                         # Neutral stable


Would actually rather suggest to to do https://github.com/funilrys/PyFunceble/pull/262/files/aa75d8886d6196b4dead17b815e04e6908b6306b..f7430a448961eb63856e3bc00f358d9b9176bc2f#r739692268 here as. 2 IP to same service.

c PyFunceble -d 9.9.9.9 149.112.112.112

@spirillen I think it's a good idea to show two domains at once, but I don't like using this as the very first example. I like #262 (comment) better.

Hmm, my second thought was based on the stability on the quad9's IP addresses, how about reordering them then?

(The comment line would also be kept shorter 😉 ) Just throwing some ball's into the air here.

I think it's best to keep a single IP as the first example, especially 9.9.9.9 because it's so simple. And I think that it's better that two IPs in a single command have different statuses, i.e. one of them is active and the other inactive - makes it more noticable IMO. But these are just nitpicks, I don't feel as strongly about this as I may have made it seem.

I buy your argument with two diff result status... it's not bad...

@funilrys Your thoughts?

Hi there, single IP as the first example is good for me.

Later, when the generation and upload to the gh-pages are automated, we could document all possible cases @spirillen.

I'm open for making further documentation, there are just the "just", if you remember I was kind of starting making examples for various use cases (which I then deleted again before it got merged into 4.0) as making all that inside the documentation became more messy than good constructive use examples.

I can't help thinking on @T145 comment in #265 (comment), not that I'll dare to say I fully understand it yet, but the is obviously room for some rethinking. Please do throw anything into the pot :cauldron:

Non the less as always, please make all docs related issues over there... (NOT 🇺🇸 but https://mypdns.org/spirillen/PyFunceble) so i'm in control of the related issues work-boards etc.

PS: sorry for typos, as I switches between EN and DK keyboard layout.

funilrys

LGTM!

@avatartw

commit 0582a27 Author: funilrys <[email protected]> Date: Sat Jul 16 17:50:29 2022 +0200 Bump version to v4.1.2.dev commit 1ddc3a0 Author: funilrys <[email protected]> Date: Sat Jul 16 14:38:45 2022 +0200 Fix issue with encoding. This patch fixes #300. Indeed, before this patch, we were using the system-wide encoding instead of the defacto default UTF-8. Contributors: * @avatartw commit 9a5fada Author: funilrys <[email protected]> Date: Mon Mar 21 19:40:47 2022 +0100 Bump version to v4.1.1 commit 07ad55f Author: funilrys <[email protected]> Date: Mon Mar 21 19:35:53 2022 +0100 Squashed commit of the following: commit 8320b72 Author: funilrys <[email protected]> Date: Mon Mar 21 19:29:53 2022 +0100 Upgrade contributor list. This patch adds @bigdargon to the list of contributors. commit a448673 Author: funilrys <[email protected]> Date: Mon Mar 21 19:24:48 2022 +0100 Better handling of subjects without extension. This patch fixes #297. This patch fixes dead-hosts/dev-center#23. This patch touches the proxy matcher. Indeed, before this patch, I didn't assume that it may be possible to have subject without any extension. This error lead some session to crash shortly before the end. This patch fixes the issue by handling the case that the subject without any extension is given. Contributors: * @spirillen * @bigdargon commit a6f8b9c Author: funilrys <[email protected]> Date: Sat Mar 12 12:41:30 2022 +0100 Bump version to v4.1.0 commit 476a942 Author: funilrys <[email protected]> Date: Sat Mar 12 09:13:37 2022 +0100 Bump version to v4.1.0b18. commit 5cae08e Author: funilrys <[email protected]> Date: Sat Mar 12 09:06:38 2022 +0100 Restrict python-box. This patch fixes #294. Contributors: * @cdgriffith commit 3688974 Author: funilrys <[email protected]> Date: Sun Mar 6 16:55:12 2022 +0100 Bump version to v4.1.0b17 commit 7770336 Merge: 47a4e4b 06398ca Author: funilrys <[email protected]> Date: Sun Mar 6 16:51:56 2022 +0100 Merge branch 'maintenance' into dev commit 06398ca Author: funilrys <[email protected]> Date: Sun Mar 6 16:51:19 2022 +0100 Add missing variable. commit 47a4e4b Author: funilrys <[email protected]> Date: Sun Mar 6 14:41:01 2022 +0100 Bump version to v4.1.0b16. commit 513a61f Author: funilrys <[email protected]> Date: Sun Mar 6 14:22:26 2022 +0100 Better handling of proxy usage. This patch touches #132. This patch fixes #293. Indeed, before this patch a special case related to the test on onions subject wasn't handled properly. This patch also replace the socks5 with socks5h protocol within the documentation. User that are testing onions and do not have a DNS resolver/server that can handle them are advised to use the socks5h protocol instead of socks5. That will ensure that the DNS lookup is not handled locally. Contributors: * @spirillen commit 7a52279 Author: funilrys <[email protected]> Date: Sat Mar 5 23:55:19 2022 +0100 Bump version to v4.1.0b15. commit d721eef Author: funilrys <[email protected]> Date: Sat Mar 5 23:53:14 2022 +0100 fixup! Add sock capabilities. commit 2f2eddb Author: funilrys <[email protected]> Date: Sat Mar 5 21:53:33 2022 +0100 Bump version to v4.1.0b14. commit 03f9446 Author: funilrys <[email protected]> Date: Sat Mar 5 21:47:48 2022 +0100 Add sock capabilities. commit fdb9a27 Author: funilrys <[email protected]> Date: Sat Mar 5 19:50:13 2022 +0100 Bump version to v4.1.0b13. commit 5b813d7 Author: funilrys <[email protected]> Date: Sat Mar 5 19:45:29 2022 +0100 Introduction of the *.altervista.org SPECIAL rule. This patch fixes #292. From now on, any subjects matching *.altervista.org that return the 403 status codes are supplied as `INACTIVE`. Contributors: * @spirillen commit d73c097 Author: funilrys <[email protected]> Date: Sat Mar 5 19:41:46 2022 +0100 Fix deployment to AUR. commit 824afd9 Author: funilrys <[email protected]> Date: Sat Mar 5 19:23:51 2022 +0100 Add undocumented SPECIAL rules. commit b393b85 Author: funilrys <[email protected]> Date: Sat Mar 5 19:20:17 2022 +0100 Update Copyright year. commit 0fc29e6 Author: funilrys <[email protected]> Date: Sat Mar 5 18:54:38 2022 +0100 Bump version to v4.1.0b12 commit 8738648 Author: funilrys <[email protected]> Date: Sat Mar 5 18:49:04 2022 +0100 Introduction of the proxy settings. This patch fixes #132. Indeed, before this patch, one could use the HTTP_PROXY and HTTPS_PROXY settings. But, PyFunceble wouldn't actually handle some special rules. This patch introduce the `--http-proxy` and `https-proxy` arguments. This patch introduce the ability to define some extra rules - within the configuration file. Indeed, this is something that was missing. Therefore, from now on, you can define some extra rules around your proxies. For example, if you want to use `example.com` as main proxy but want all `.com` and `.org` subjects to be requested through `example.dev`, this is now possible through the configuration: ```yaml proxy: global: http: http://example.com:8080 https: http://example.com:8080 rules: - http: http://example.dev:8080 https: http://example.dev:8080 tld: - com - org ``` Also: * Fix Codeblock syntax in the documentation. Contributors: * @spirillen commit 152dc4e Author: funilrys <[email protected]> Date: Sun Feb 13 18:53:58 2022 +0100 Bump version to v4.1.0b11. commit 22d39cf Author: funilrys <[email protected]> Date: Sun Feb 13 18:28:25 2022 +0100 Add forgotten explicit encoding. This patch fixes #291. Indeed, before this patch there may have been some encoding collision because I forgot to put the encoding explicitely before opening a CSV file. Contributors: * @avatartw commit ccbf6fe Author: funilrys <[email protected]> Date: Sat Feb 12 13:07:03 2022 +0100 Bump version to v4.1.0b10. commit ae46748 Author: funilrys <[email protected]> Date: Sat Feb 12 13:01:48 2022 +0100 Introduction of the '*-lookup-only' argument. This patch fixes #276. Indeed, before this patch, there way no way to tell - from the CLI - that we only want the method X or Z for the lookup. Contributors: * @spirillen commit 8ebfde6 Author: funilrys <[email protected]> Date: Sat Jan 1 17:36:31 2022 +0100 Bump version to v4.1.0b9. commit ab8f5e7 Author: funilrys <[email protected]> Date: Sat Jan 1 17:30:58 2022 +0100 Improvement of the Support of IPv6 DNS. Indeed, before this patch, the support was not complete. This patch complete the support by expecting the following format for any IPv6 with explicit address: [ip]:port If no braket is found, we will provide the default port. commit 2640455 Author: funilrys <[email protected]> Date: Sat Jan 1 16:11:09 2022 +0100 Bump version to v4.1.0b8. commit 16f3898 Author: funilrys <[email protected]> Date: Sat Jan 1 16:07:07 2022 +0100 fixup! Do DNS Lookup before querying URL. commit b01376b Author: funilrys <[email protected]> Date: Sat Jan 1 14:37:52 2022 +0100 fixup! Fix issue with deployment to the AUR. commit 2393ff3 Author: funilrys <[email protected]> Date: Sat Jan 1 13:47:32 2022 +0100 Fix issue with deployment to the AUR. commit 408e277 Author: funilrys <[email protected]> Date: Fri Dec 31 18:40:50 2021 +0100 Bump version to v4.1.0b7. commit cd14dc8 Author: funilrys <[email protected]> Date: Fri Dec 31 18:21:33 2021 +0100 Do DNS Lookup before querying URL. This patch fixes #164. Indeed, before this patch, we were trying to query URL even if the domain didn't have an IP address. This patch fixes the issue by doing a classic (NS, CNAME, A, AAA) query before trying to query any URL. Note: The DNS Lookup is the same as the one we do for a standard domain/ip test. Contributors: * @mitchellkrogza * @spirillen commit 86e2558 Author: spirillen <[email protected]> Date: Fri Dec 31 01:38:33 2021 +0100 Slight change to the DNS IP Have changed the Quad9 IP address to 9.9.9.10 as we should promoted unfiltered DNS services In general, as that hopefully is the most needed in combination with @PyFunceble commit 8505492 Author: funilrys <[email protected]> Date: Fri Dec 31 16:30:48 2021 +0100 Bump version to v4.1.0b6. commit 088947b Author: funilrys <[email protected]> Date: Fri Dec 31 16:11:24 2021 +0100 Workaround Y2K38. This patch fixes #261. Indeed, before this patch, there was no proper solution to the Y2K38 issue - within PyFunceble. This patch try to workaround the issue by rounding the dates to a few weeks before Y2K38 - when the database vendor didn't implement a solution. Contributors: * @spirillen commit 2e2c517 Author: funilrys <[email protected]> Date: Sat Dec 25 22:32:25 2021 +0100 Bump version to v4.1.0b5. commit 2c4ea7d Merge: b3c4442 8514ca5 Author: funilrys <[email protected]> Date: Sat Dec 25 22:28:20 2021 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit 8514ca5 Author: funilrys <[email protected]> Date: Sat Dec 25 20:29:11 2021 +0100 Review Syntax checker. This patch fixes #282. Our testing module was too close to standards. If we consider blogspot.com a Public Suffix and therefore a effective Top-Level domain (eTLD), hello_world.blogspot.com shouldn't be valid. But the really behind the public suffixes is different. Therefore, this patch relaxes the rules regarding sub-domains behind a known public suffix. commit a346154 Author: funilrys <[email protected]> Date: Sat Dec 25 19:09:14 2021 +0100 Fix issue regarding false-positive invalid. This patch touches #282. Indeed, before this patch our self-healing false-positive lookup through DNS may have been incomplete. This patch fixes the issue by making the ruleset for the continuation to the next test method more transparent. commit b3c4442 Merge: f0292cc f537119 Author: Nissar Chababy <[email protected]> Date: Sat Dec 25 17:28:34 2021 +0100 Merge pull request #262 from veracioux:tuterm_demo Add automated CLI demonstrations Contributors: * @veracioux * @spirillen commit f537119 Merge: f6c666e f0292cc Author: funilrys <[email protected]> Date: Sat Dec 25 17:26:45 2021 +0100 Merge remote-tracking branch 'upstream/dev' into tuterm_demo commit f6c666e Author: funilrys <[email protected]> Date: Sat Dec 25 16:56:52 2021 +0100 Add @veracioux to the list of contributors. commit f0292cc Author: spirillen <[email protected]> Date: Sat Dec 25 16:45:03 2021 +0100 Merge: #284 This patch merges #284. This patch fixes #283. Contributors: * @spirillen commit bb80fa5 Author: funilrys <[email protected]> Date: Wed Dec 22 19:06:15 2021 +0100 Fix sphinx error. commit 7b7ca75 Author: funilrys <[email protected]> Date: Wed Dec 22 18:59:02 2021 +0100 Url with scheme and port shouldn't be INVALID. commit ae9333a Author: funilrys <[email protected]> Date: Wed Dec 22 17:19:04 2021 +0100 Bump version to v4.1.0b4. commit 869cab6 Merge: 6a66e71 d4d42bf Author: funilrys <[email protected]> Date: Wed Dec 22 17:13:31 2021 +0100 Merge remote-tracking branch 'origin/chancy-tester' into dev commit 6a66e71 Merge: a8bea19 177a2f0 Author: funilrys <[email protected]> Date: Wed Dec 22 17:13:05 2021 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit d4d42bf Author: funilrys <[email protected]> Date: Wed Dec 22 01:11:45 2021 +0100 Introduction of the chancy tester mode. After discussion with some people and reading some emails of people privately making a comparison between PyFunceble 3 and 4, I decided to write the code for a "chancy" testing mode. Origin: The chancy mode is just a reviving of what I do not agree with - anymore. In PyFunceble 3, the test feedback is almost instant, meaning that the feeling of "speed" is there. But if you read the source code, you will see that it's possible because I chose to ignore a lot of safety to play with my luck, people's luck and machine's luck. Indeed, under PyFunceble 3, it is theoretically possible to have a "line overflow". That means that when: - You are using the multiprocessing mode - You are testing `example.org` and `example.net`, it may have been possible to get the following line as a result: examexampleple.org.net or this: example.orgexample.net That's the fact. If that rare case happens, you may have a corrupted output that you will later parse into your workflow. To solve the problem, I decided to change the workflow design. That's PyFunceble 4. Current State: Under PyFunceble 4, two main things changed: 1. No more single and multiprocessing. 2. No file production from the testing workers. What we used to call the `single-mode` went away and was replaced with a multiprocessing mode that's always there. The idea was that nowadays multiprocessing is the way. On the other hand, I killed the possible "overflow" of PyFunceble 3 by following a clear data workflow that makes output safe. This is an approximate overview of the actual workflow: --- Read Subject ---> Tester Queue \ ---> N Tester Process ---> Producer Queue / 1 Producer Process <--- / Output File | Graphical CLI Output <--- With this workflow, I'm sure that only ONE single process is reporting into any of the generated files. That means that the issue I previously described can't happen. That's the safe workflow. The downside of this workflow is that people think that everything is slow. It is. But for your safety. If you look at some very verbose logs, you will see that the actual testing is as quick as possible (for your hardware) but it's the file generation that is slow. That's because it goes one tested subject at a time - for your safety. Chancy Mode: The chancy mode brings back a PyFunceble 3 "like" way of handling data. That means that if used, no safety is given. An overflow can happen, but the feeling of "speed" is back. This mode exists but won't be actively maintained. To reduce long term maintenance of this mode, it has been built on top of the maintained and developed components of the workflow. Meaning that as long as no new component is built, this mode is "self-maintained". commit 177a2f0 Author: funilrys <[email protected]> Date: Tue Dec 21 16:55:27 2021 +0100 Add @gwarser to the list of contributors. commit a8bea19 Author: funilrys <[email protected]> Date: Tue Dec 21 16:04:03 2021 +0100 Bump version to v4.1.0b3. commit 1806e10 Author: funilrys <[email protected]> Date: Tue Dec 21 15:38:39 2021 +0100 Handle DNS name too long. Indeed before this patch, some DNS name which were longer than 255 octets may produce some fatal error. commit 62d799c Author: funilrys <[email protected]> Date: Sun Dec 19 12:11:15 2021 +0100 fixup! Bump version to v4.1.0b2. commit dad28cc Merge: 1310461 b5ecdc2 Author: funilrys <[email protected]> Date: Sat Dec 18 22:59:43 2021 +0100 Bump version to v4.1.0b2. Also: Fix #272. commit 1310461 Author: funilrys <[email protected]> Date: Sat Dec 18 22:53:09 2021 +0100 Bump version to v4.1.0b1. commit 213a087 Author: funilrys <[email protected]> Date: Sat Dec 18 19:59:12 2021 +0100 Add max internal max rescursion depth. Indeed, before this patch, the CNAME follower when doing HTTP request may habe been able to run without a limit. This patch fixes it by allowing a max depth of 60. Any depth above 60 will just be skipped. commit ebc02c6 Author: funilrys <[email protected]> Date: Sat Dec 18 20:11:18 2021 +0100 Introduction of a the max-http-retries arg. Indeed, before this patch, there was no way to control the maximum number of retries PyFunceble is allowed to perform when doing an HTTP request. This patch fixes that hole by adding the `--max-http-retries` argument. commit fdc4f9c Author: funilrys <[email protected]> Date: Sat Dec 18 22:31:46 2021 +0100 Logical fix. Indeed, before this patch, we didn't actively checked if the inactive and autocontinue dataset lookup was deactivated before doing some backup or update of the dataset. This patch fixes the issue by checking if we are authorized to process such dataset before even trying to do so. commit d31b9b1 Author: funilrys <[email protected]> Date: Sat Dec 18 22:31:16 2021 +0100 Fix issue with counter. commit a85eafb Author: funilrys <[email protected]> Date: Sat Dec 18 22:29:58 2021 +0100 Review requester initialization. Indeed, before this patch, the way the requester was initialized wasn't safe enough for a multiprocessing environment. commit c487a48 Author: funilrys <[email protected]> Date: Sat Dec 18 21:44:22 2021 +0100 Add a way to differs worker startup and creation. Indeed, before this patch, as soon as something is put inside an input queue of a worker, the worker will be automatically created. But, at sometime, for example for worker that should only run at the end, we do not want that behavior. Therefore, this patch introduce a way to differ the creation and startup of a process - when necessary. This patch apply for example for the file sorter that is actually only necessary at the end of a test. commit cb19031 Author: funilrys <[email protected]> Date: Sun Dec 5 19:05:36 2021 +0100 Bump version ot v4.0.2. commit b5ecdc2 Author: funilrys <[email protected]> Date: Sun Dec 5 18:18:30 2021 +0100 Extract the registrar from the WHOIS record. This patch fixes #272. Indeed, before this patch, there was no way to get access to the registrar from the WHOIS record. This patch tries to solve the issue by adding a registrar extractor. This patch also introduce the `--registrar` and `--max-registrar` arguments that can be used to display the top N registrar that were extracted. Using the `--registrar` argument also generate a `logs/registrar.txt` file that can be used to list the all registrar found while testing the current file. Necessary Migration: -- All migration will be automatically triggered. * csv file: The whois.csv file needs to be migrated in order for it to have the `registrar` column. * mariadb: The whois_record tables need to be migrated in order for it to have the `registrar` column. Contributors: * @ryanbr commit d9f93ea Author: funilrys <[email protected]> Date: Sun Dec 5 17:34:48 2021 +0100 Make regex case insensitive. commit f7430a4 Author: Haris Gušić <[email protected]> Date: Fri Oct 29 13:57:02 2021 +0200 Change delay times Signed-off-by: Haris Gušić <[email protected]> commit 42286da Author: Haris Gušić <[email protected]> Date: Fri Oct 29 13:56:47 2021 +0200 Add remaining IP addresses Signed-off-by: Haris Gušić <[email protected]> commit aa75d88 Author: Haris Gušić <[email protected]> Date: Sun Oct 17 18:39:43 2021 +0200 Apply suggestions and create record/upload scripts Signed-off-by: Haris Gušić <[email protected]> commit 8c79508 Author: Haris Gušić <[email protected]> Date: Sun Oct 17 01:54:11 2021 +0200 Add automated CLI demos (proof of concept) Signed-off-by: Haris Gušić <[email protected]>

@spirillen

commit 77d7250 Author: funilrys <[email protected]> Date: Sun Nov 20 12:05:27 2022 +0100 Bump version to v4.1.3a7 commit 9d59070 Author: funilrys <[email protected]> Date: Sun Nov 20 11:54:10 2022 +0100 fixup! Adjust timeout checker. commit cf9688e Author: smed79 <[email protected]> Date: Tue Nov 8 04:13:32 2022 +0100 Fix typo #317 commit b0c5f7b Author: funilrys <[email protected]> Date: Thu Nov 3 17:23:49 2022 +0100 Bump version to v4.1.3a6 commit 618620e Author: funilrys <[email protected]> Date: Thu Nov 3 17:03:34 2022 +0100 Adjust timeout checker. This patch touches #316. Indeed, before this patch, it may have been impossible to set the timeout to 0. commit 1a84b35 Author: funilrys <[email protected]> Date: Sun Oct 30 16:38:42 2022 +0100 fixup! Conditional Check: Timeout. commit 942bf2c Author: funilrys <[email protected]> Date: Sun Oct 30 16:30:27 2022 +0100 Fix linting issues. commit 89b34dd Author: funilrys <[email protected]> Date: Sun Oct 30 16:20:40 2022 +0100 Bump version to v4.1.3a5 commit cd01b10 Merge: fd13855 60a6506 Author: funilrys <[email protected]> Date: Sun Oct 30 16:17:03 2022 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit 60a6506 Author: funilrys <[email protected]> Date: Sun Oct 30 15:56:07 2022 +0100 Conditional Check: Timeout. This patch fixes #316. Indeed, before this patch, it may have been possible to crash pyfunceble by setting nagative timeout. This patch fixes the issue by checking the timeout value before injecting the configuration keys into the mechanism. Contributors: * @spirillen commit b465290 Author: funilrys <[email protected]> Date: Sun Oct 30 15:52:59 2022 +0100 Fix issue when configuration key is missing. Indeed before this patch, it may have been possible to let pyfunceble crash through a missing key. This patch fixes the issue by checking if the read keys exists. commit 4056525 Author: funilrys <[email protected]> Date: Sun Oct 30 15:43:21 2022 +0100 Decrease logging level for urllib3. This patch fixes #315. Indeed, before this patch the urllib3 package was "allowed" of logging some warnings to STDOUT. This patch fixes the issues by only allowind CRITICAL logs. Contributors: * @spirillen commit 8c74caf Author: funilrys <[email protected]> Date: Sun Oct 30 11:39:18 2022 +0100 Fix issue with exception handling. This patch fixes #314. Indeed, our exception handling wasn't broad enough. This patch fixes the issue by handling all possible yaml parsing error. Contributors: * @bigdargon commit fe717b2 Author: funilrys <[email protected]> Date: Sun Oct 30 11:06:35 2022 +0100 Fix issue while loading configuration files. Indeed, if one provides an invalid yaml file, PyFunceble may crash. This patch fixes the issue by checking if the given inputs are really in the expected format. commit 32d0a97 Author: funilrys <[email protected]> Date: Tue Oct 25 13:25:01 2022 +0200 Adjust windows requirements file. This patch touches #310. Indeed, the windows requirements file wasn't matching the standard one. This is the first step in trying to solve the issue described in #310. Contributors: * @DandelionSprout commit fd13855 Author: funilrys <[email protected]> Date: Tue Oct 25 13:16:58 2022 +0200 Update version.yaml. This patch fixes #312. Contributors: * @spirillen commit bfd6120 Author: funilrys <[email protected]> Date: Mon Oct 17 19:06:58 2022 +0200 Bump version to v4.1.3a4. commit e94f2ac Author: funilrys <[email protected]> Date: Mon Oct 17 19:06:37 2022 +0200 fixup! Improvement of exception handling around proxies. commit 974b194 Author: funilrys <[email protected]> Date: Sun Oct 16 21:25:23 2022 +0200 Bump version to v4.1.3a3. commit a2f1594 Author: funilrys <[email protected]> Date: Sat Oct 15 14:02:35 2022 +0200 Improvement of exception handling around proxies. This patch fixes #303. Indeed, before this patch, we were rising exceptions to prevent unknown safe. This patch simply remove those exceptions because the following conditions already handle a broad scopes of possible problems. Contributors: * @spirillen commit 20d4232 Author: funilrys <[email protected]> Date: Fri Oct 7 22:38:58 2022 +0200 Fix typo. commit e5e4552 Author: funilrys <[email protected]> Date: Wed Oct 5 20:16:30 2022 +0200 Introduction of a note about the switch mechanism. This patch touches #265. This patch also introduces some basic examples at in the epilog of the CLI. commit ad9a7f4 Author: funilrys <[email protected]> Date: Wed Oct 5 19:20:05 2022 +0200 Bump version to v4.1.3a2 commit b8a9a61 Author: funilrys <[email protected]> Date: Wed Oct 5 18:51:41 2022 +0200 Better handling of requests errors. This patch (potentially) fixes #303. Indeed, before this patch, we didn't handle all possible requests exceptions. This may have caused some issues similar to #303. This patch solves the issue by handling a broader exception. Contributors: * @spirillen commit 3d209f9 Author: funilrys <[email protected]> Date: Wed Oct 5 18:31:41 2022 +0200 Bump version to 4.1.3a1 commit 62671df Author: funilrys <[email protected]> Date: Sat Sep 24 02:55:43 2022 +0200 Fix FUNDING.yml file. commit 24c9628 Author: funilrys <[email protected]> Date: Sat Sep 24 02:52:43 2022 +0200 Maintenance: GitHub Workflows. * Add Tests with Python 3.10. * Upgrade setup-python action. commit d9a78f4 Author: funilrys <[email protected]> Date: Sat Sep 24 02:41:30 2022 +0200 Remove .pylintrc. From now on the setup has to made inside the setup.cfg file. commit 114d397 Author: funilrys <[email protected]> Date: Sat Sep 24 02:37:27 2022 +0200 Adjust requirements. commit 2ccc39d Merge: de8d27b 0582a27 Author: funilrys <[email protected]> Date: Sun Jul 17 17:01:19 2022 +0200 Merge remote-tracking branch 'origin/dev' into maintenance commit 0582a27 Author: funilrys <[email protected]> Date: Sat Jul 16 17:50:29 2022 +0200 Bump version to v4.1.2.dev commit 1ddc3a0 Author: funilrys <[email protected]> Date: Sat Jul 16 14:38:45 2022 +0200 Fix issue with encoding. This patch fixes #300. Indeed, before this patch, we were using the system-wide encoding instead of the defacto default UTF-8. Contributors: * @avatartw commit de8d27b Author: funilrys <[email protected]> Date: Mon May 30 22:35:49 2022 +0200 Disable too-many-function-args. commit 62f989e Author: funilrys <[email protected]> Date: Sun Apr 17 14:14:57 2022 +0200 Fix linting issues. commit da4f27f Author: funilrys <[email protected]> Date: Sun Apr 17 13:53:05 2022 +0200 Fix possible issue with latest version of git. commit 9a5fada Author: funilrys <[email protected]> Date: Mon Mar 21 19:40:47 2022 +0100 Bump version to v4.1.1 commit 07ad55f Author: funilrys <[email protected]> Date: Mon Mar 21 19:35:53 2022 +0100 Squashed commit of the following: commit 8320b72 Author: funilrys <[email protected]> Date: Mon Mar 21 19:29:53 2022 +0100 Upgrade contributor list. This patch adds @bigdargon to the list of contributors. commit a448673 Author: funilrys <[email protected]> Date: Mon Mar 21 19:24:48 2022 +0100 Better handling of subjects without extension. This patch fixes #297. This patch fixes dead-hosts/dev-center#23. This patch touches the proxy matcher. Indeed, before this patch, I didn't assume that it may be possible to have subject without any extension. This error lead some session to crash shortly before the end. This patch fixes the issue by handling the case that the subject without any extension is given. Contributors: * @spirillen * @bigdargon commit 2f759d9 Author: funilrys <[email protected]> Date: Mon Mar 21 19:14:45 2022 +0100 fixup! Apply suggestions and create record/upload scripts commit d969fef Author: funilrys <[email protected]> Date: Mon Mar 21 19:12:33 2022 +0100 Maintenance: .github commit a6f8b9c Author: funilrys <[email protected]> Date: Sat Mar 12 12:41:30 2022 +0100 Bump version to v4.1.0 commit 476a942 Author: funilrys <[email protected]> Date: Sat Mar 12 09:13:37 2022 +0100 Bump version to v4.1.0b18. commit 5cae08e Author: funilrys <[email protected]> Date: Sat Mar 12 09:06:38 2022 +0100 Restrict python-box. This patch fixes #294. Contributors: * @cdgriffith commit 3688974 Author: funilrys <[email protected]> Date: Sun Mar 6 16:55:12 2022 +0100 Bump version to v4.1.0b17 commit 7770336 Merge: 47a4e4b 06398ca Author: funilrys <[email protected]> Date: Sun Mar 6 16:51:56 2022 +0100 Merge branch 'maintenance' into dev commit 06398ca Author: funilrys <[email protected]> Date: Sun Mar 6 16:51:19 2022 +0100 Add missing variable. commit 47a4e4b Author: funilrys <[email protected]> Date: Sun Mar 6 14:41:01 2022 +0100 Bump version to v4.1.0b16. commit 513a61f Author: funilrys <[email protected]> Date: Sun Mar 6 14:22:26 2022 +0100 Better handling of proxy usage. This patch touches #132. This patch fixes #293. Indeed, before this patch a special case related to the test on onions subject wasn't handled properly. This patch also replace the socks5 with socks5h protocol within the documentation. User that are testing onions and do not have a DNS resolver/server that can handle them are advised to use the socks5h protocol instead of socks5. That will ensure that the DNS lookup is not handled locally. Contributors: * @spirillen commit 7a52279 Author: funilrys <[email protected]> Date: Sat Mar 5 23:55:19 2022 +0100 Bump version to v4.1.0b15. commit d721eef Author: funilrys <[email protected]> Date: Sat Mar 5 23:53:14 2022 +0100 fixup! Add sock capabilities. commit 2f2eddb Author: funilrys <[email protected]> Date: Sat Mar 5 21:53:33 2022 +0100 Bump version to v4.1.0b14. commit 03f9446 Author: funilrys <[email protected]> Date: Sat Mar 5 21:47:48 2022 +0100 Add sock capabilities. commit fdb9a27 Author: funilrys <[email protected]> Date: Sat Mar 5 19:50:13 2022 +0100 Bump version to v4.1.0b13. commit 5b813d7 Author: funilrys <[email protected]> Date: Sat Mar 5 19:45:29 2022 +0100 Introduction of the *.altervista.org SPECIAL rule. This patch fixes #292. From now on, any subjects matching *.altervista.org that return the 403 status codes are supplied as `INACTIVE`. Contributors: * @spirillen commit d73c097 Author: funilrys <[email protected]> Date: Sat Mar 5 19:41:46 2022 +0100 Fix deployment to AUR. commit 824afd9 Author: funilrys <[email protected]> Date: Sat Mar 5 19:23:51 2022 +0100 Add undocumented SPECIAL rules. commit b393b85 Author: funilrys <[email protected]> Date: Sat Mar 5 19:20:17 2022 +0100 Update Copyright year. commit 0fc29e6 Author: funilrys <[email protected]> Date: Sat Mar 5 18:54:38 2022 +0100 Bump version to v4.1.0b12 commit 8738648 Author: funilrys <[email protected]> Date: Sat Mar 5 18:49:04 2022 +0100 Introduction of the proxy settings. This patch fixes #132. Indeed, before this patch, one could use the HTTP_PROXY and HTTPS_PROXY settings. But, PyFunceble wouldn't actually handle some special rules. This patch introduce the `--http-proxy` and `https-proxy` arguments. This patch introduce the ability to define some extra rules - within the configuration file. Indeed, this is something that was missing. Therefore, from now on, you can define some extra rules around your proxies. For example, if you want to use `example.com` as main proxy but want all `.com` and `.org` subjects to be requested through `example.dev`, this is now possible through the configuration: ```yaml proxy: global: http: http://example.com:8080 https: http://example.com:8080 rules: - http: http://example.dev:8080 https: http://example.dev:8080 tld: - com - org ``` Also: * Fix Codeblock syntax in the documentation. Contributors: * @spirillen commit 152dc4e Author: funilrys <[email protected]> Date: Sun Feb 13 18:53:58 2022 +0100 Bump version to v4.1.0b11. commit 22d39cf Author: funilrys <[email protected]> Date: Sun Feb 13 18:28:25 2022 +0100 Add forgotten explicit encoding. This patch fixes #291. Indeed, before this patch there may have been some encoding collision because I forgot to put the encoding explicitely before opening a CSV file. Contributors: * @avatartw commit ccbf6fe Author: funilrys <[email protected]> Date: Sat Feb 12 13:07:03 2022 +0100 Bump version to v4.1.0b10. commit ae46748 Author: funilrys <[email protected]> Date: Sat Feb 12 13:01:48 2022 +0100 Introduction of the '*-lookup-only' argument. This patch fixes #276. Indeed, before this patch, there way no way to tell - from the CLI - that we only want the method X or Z for the lookup. Contributors: * @spirillen commit 8ebfde6 Author: funilrys <[email protected]> Date: Sat Jan 1 17:36:31 2022 +0100 Bump version to v4.1.0b9. commit ab8f5e7 Author: funilrys <[email protected]> Date: Sat Jan 1 17:30:58 2022 +0100 Improvement of the Support of IPv6 DNS. Indeed, before this patch, the support was not complete. This patch complete the support by expecting the following format for any IPv6 with explicit address: [ip]:port If no braket is found, we will provide the default port. commit 2640455 Author: funilrys <[email protected]> Date: Sat Jan 1 16:11:09 2022 +0100 Bump version to v4.1.0b8. commit 16f3898 Author: funilrys <[email protected]> Date: Sat Jan 1 16:07:07 2022 +0100 fixup! Do DNS Lookup before querying URL. commit b01376b Author: funilrys <[email protected]> Date: Sat Jan 1 14:37:52 2022 +0100 fixup! Fix issue with deployment to the AUR. commit 2393ff3 Author: funilrys <[email protected]> Date: Sat Jan 1 13:47:32 2022 +0100 Fix issue with deployment to the AUR. commit 408e277 Author: funilrys <[email protected]> Date: Fri Dec 31 18:40:50 2021 +0100 Bump version to v4.1.0b7. commit cd14dc8 Author: funilrys <[email protected]> Date: Fri Dec 31 18:21:33 2021 +0100 Do DNS Lookup before querying URL. This patch fixes #164. Indeed, before this patch, we were trying to query URL even if the domain didn't have an IP address. This patch fixes the issue by doing a classic (NS, CNAME, A, AAA) query before trying to query any URL. Note: The DNS Lookup is the same as the one we do for a standard domain/ip test. Contributors: * @mitchellkrogza * @spirillen commit 86e2558 Author: spirillen <[email protected]> Date: Fri Dec 31 01:38:33 2021 +0100 Slight change to the DNS IP Have changed the Quad9 IP address to 9.9.9.10 as we should promoted unfiltered DNS services In general, as that hopefully is the most needed in combination with @PyFunceble commit 8505492 Author: funilrys <[email protected]> Date: Fri Dec 31 16:30:48 2021 +0100 Bump version to v4.1.0b6. commit 088947b Author: funilrys <[email protected]> Date: Fri Dec 31 16:11:24 2021 +0100 Workaround Y2K38. This patch fixes #261. Indeed, before this patch, there was no proper solution to the Y2K38 issue - within PyFunceble. This patch try to workaround the issue by rounding the dates to a few weeks before Y2K38 - when the database vendor didn't implement a solution. Contributors: * @spirillen commit 2e2c517 Author: funilrys <[email protected]> Date: Sat Dec 25 22:32:25 2021 +0100 Bump version to v4.1.0b5. commit 2c4ea7d Merge: b3c4442 8514ca5 Author: funilrys <[email protected]> Date: Sat Dec 25 22:28:20 2021 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit 8514ca5 Author: funilrys <[email protected]> Date: Sat Dec 25 20:29:11 2021 +0100 Review Syntax checker. This patch fixes #282. Our testing module was too close to standards. If we consider blogspot.com a Public Suffix and therefore a effective Top-Level domain (eTLD), hello_world.blogspot.com shouldn't be valid. But the really behind the public suffixes is different. Therefore, this patch relaxes the rules regarding sub-domains behind a known public suffix. commit a346154 Author: funilrys <[email protected]> Date: Sat Dec 25 19:09:14 2021 +0100 Fix issue regarding false-positive invalid. This patch touches #282. Indeed, before this patch our self-healing false-positive lookup through DNS may have been incomplete. This patch fixes the issue by making the ruleset for the continuation to the next test method more transparent. commit b3c4442 Merge: f0292cc f537119 Author: Nissar Chababy <[email protected]> Date: Sat Dec 25 17:28:34 2021 +0100 Merge pull request #262 from veracioux:tuterm_demo Add automated CLI demonstrations Contributors: * @veracioux * @spirillen commit f537119 Merge: f6c666e f0292cc Author: funilrys <[email protected]> Date: Sat Dec 25 17:26:45 2021 +0100 Merge remote-tracking branch 'upstream/dev' into tuterm_demo commit f6c666e Author: funilrys <[email protected]> Date: Sat Dec 25 16:56:52 2021 +0100 Add @veracioux to the list of contributors. commit f0292cc Author: spirillen <[email protected]> Date: Sat Dec 25 16:45:03 2021 +0100 Merge: #284 This patch merges #284. This patch fixes #283. Contributors: * @spirillen commit bb80fa5 Author: funilrys <[email protected]> Date: Wed Dec 22 19:06:15 2021 +0100 Fix sphinx error. commit 7b7ca75 Author: funilrys <[email protected]> Date: Wed Dec 22 18:59:02 2021 +0100 Url with scheme and port shouldn't be INVALID. commit ae9333a Author: funilrys <[email protected]> Date: Wed Dec 22 17:19:04 2021 +0100 Bump version to v4.1.0b4. commit 869cab6 Merge: 6a66e71 d4d42bf Author: funilrys <[email protected]> Date: Wed Dec 22 17:13:31 2021 +0100 Merge remote-tracking branch 'origin/chancy-tester' into dev commit 6a66e71 Merge: a8bea19 177a2f0 Author: funilrys <[email protected]> Date: Wed Dec 22 17:13:05 2021 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit d4d42bf Author: funilrys <[email protected]> Date: Wed Dec 22 01:11:45 2021 +0100 Introduction of the chancy tester mode. After discussion with some people and reading some emails of people privately making a comparison between PyFunceble 3 and 4, I decided to write the code for a "chancy" testing mode. Origin: The chancy mode is just a reviving of what I do not agree with - anymore. In PyFunceble 3, the test feedback is almost instant, meaning that the feeling of "speed" is there. But if you read the source code, you will see that it's possible because I chose to ignore a lot of safety to play with my luck, people's luck and machine's luck. Indeed, under PyFunceble 3, it is theoretically possible to have a "line overflow". That means that when: - You are using the multiprocessing mode - You are testing `example.org` and `example.net`, it may have been possible to get the following line as a result: examexampleple.org.net or this: example.orgexample.net That's the fact. If that rare case happens, you may have a corrupted output that you will later parse into your workflow. To solve the problem, I decided to change the workflow design. That's PyFunceble 4. Current State: Under PyFunceble 4, two main things changed: 1. No more single and multiprocessing. 2. No file production from the testing workers. What we used to call the `single-mode` went away and was replaced with a multiprocessing mode that's always there. The idea was that nowadays multiprocessing is the way. On the other hand, I killed the possible "overflow" of PyFunceble 3 by following a clear data workflow that makes output safe. This is an approximate overview of the actual workflow: --- Read Subject ---> Tester Queue \ ---> N Tester Process ---> Producer Queue / 1 Producer Process <--- / Output File | Graphical CLI Output <--- With this workflow, I'm sure that only ONE single process is reporting into any of the generated files. That means that the issue I previously described can't happen. That's the safe workflow. The downside of this workflow is that people think that everything is slow. It is. But for your safety. If you look at some very verbose logs, you will see that the actual testing is as quick as possible (for your hardware) but it's the file generation that is slow. That's because it goes one tested subject at a time - for your safety. Chancy Mode: The chancy mode brings back a PyFunceble 3 "like" way of handling data. That means that if used, no safety is given. An overflow can happen, but the feeling of "speed" is back. This mode exists but won't be actively maintained. To reduce long term maintenance of this mode, it has been built on top of the maintained and developed components of the workflow. Meaning that as long as no new component is built, this mode is "self-maintained". commit 177a2f0 Author: funilrys <[email protected]> Date: Tue Dec 21 16:55:27 2021 +0100 Add @gwarser to the list of contributors. commit a8bea19 Author: funilrys <[email protected]> Date: Tue Dec 21 16:04:03 2021 +0100 Bump version to v4.1.0b3. commit 1806e10 Author: funilrys <[email protected]> Date: Tue Dec 21 15:38:39 2021 +0100 Handle DNS name too long. Indeed before this patch, some DNS name which were longer than 255 octets may produce some fatal error. commit 62d799c Author: funilrys <[email protected]> Date: Sun Dec 19 12:11:15 2021 +0100 fixup! Bump version to v4.1.0b2. commit dad28cc Merge: 1310461 b5ecdc2 Author: funilrys <[email protected]> Date: Sat Dec 18 22:59:43 2021 +0100 Bump version to v4.1.0b2. Also: Fix #272. commit 1310461 Author: funilrys <[email protected]> Date: Sat Dec 18 22:53:09 2021 +0100 Bump version to v4.1.0b1. commit 213a087 Author: funilrys <[email protected]> Date: Sat Dec 18 19:59:12 2021 +0100 Add max internal max rescursion depth. Indeed, before this patch, the CNAME follower when doing HTTP request may habe been able to run without a limit. This patch fixes it by allowing a max depth of 60. Any depth above 60 will just be skipped. commit ebc02c6 Author: funilrys <[email protected]> Date: Sat Dec 18 20:11:18 2021 +0100 Introduction of a the max-http-retries arg. Indeed, before this patch, there was no way to control the maximum number of retries PyFunceble is allowed to perform when doing an HTTP request. This patch fixes that hole by adding the `--max-http-retries` argument. commit fdc4f9c Author: funilrys <[email protected]> Date: Sat Dec 18 22:31:46 2021 +0100 Logical fix. Indeed, before this patch, we didn't actively checked if the inactive and autocontinue dataset lookup was deactivated before doing some backup or update of the dataset. This patch fixes the issue by checking if we are authorized to process such dataset before even trying to do so. commit d31b9b1 Author: funilrys <[email protected]> Date: Sat Dec 18 22:31:16 2021 +0100 Fix issue with counter. commit a85eafb Author: funilrys <[email protected]> Date: Sat Dec 18 22:29:58 2021 +0100 Review requester initialization. Indeed, before this patch, the way the requester was initialized wasn't safe enough for a multiprocessing environment. commit c487a48 Author: funilrys <[email protected]> Date: Sat Dec 18 21:44:22 2021 +0100 Add a way to differs worker startup and creation. Indeed, before this patch, as soon as something is put inside an input queue of a worker, the worker will be automatically created. But, at sometime, for example for worker that should only run at the end, we do not want that behavior. Therefore, this patch introduce a way to differ the creation and startup of a process - when necessary. This patch apply for example for the file sorter that is actually only necessary at the end of a test. commit cb19031 Author: funilrys <[email protected]> Date: Sun Dec 5 19:05:36 2021 +0100 Bump version ot v4.0.2. commit b5ecdc2 Author: funilrys <[email protected]> Date: Sun Dec 5 18:18:30 2021 +0100 Extract the registrar from the WHOIS record. This patch fixes #272. Indeed, before this patch, there was no way to get access to the registrar from the WHOIS record. This patch tries to solve the issue by adding a registrar extractor. This patch also introduce the `--registrar` and `--max-registrar` arguments that can be used to display the top N registrar that were extracted. Using the `--registrar` argument also generate a `logs/registrar.txt` file that can be used to list the all registrar found while testing the current file. Necessary Migration: -- All migration will be automatically triggered. * csv file: The whois.csv file needs to be migrated in order for it to have the `registrar` column. * mariadb: The whois_record tables need to be migrated in order for it to have the `registrar` column. Contributors: * @ryanbr commit d9f93ea Author: funilrys <[email protected]> Date: Sun Dec 5 17:34:48 2021 +0100 Make regex case insensitive. commit f7430a4 Author: Haris Gušić <[email protected]> Date: Fri Oct 29 13:57:02 2021 +0200 Change delay times Signed-off-by: Haris Gušić <[email protected]> commit 42286da Author: Haris Gušić <[email protected]> Date: Fri Oct 29 13:56:47 2021 +0200 Add remaining IP addresses Signed-off-by: Haris Gušić <[email protected]> commit aa75d88 Author: Haris Gušić <[email protected]> Date: Sun Oct 17 18:39:43 2021 +0200 Apply suggestions and create record/upload scripts Signed-off-by: Haris Gušić <[email protected]> commit 8c79508 Author: Haris Gušić <[email protected]> Date: Sun Oct 17 01:54:11 2021 +0200 Add automated CLI demos (proof of concept) Signed-off-by: Haris Gušić <[email protected]>

Add automated CLI demos (proof of concept)

8c79508

Signed-off-by: Haris Gušić <[email protected]>

spirillen reviewed Oct 17, 2021

View reviewed changes

funilrys linked an issue Oct 17, 2021 that may be closed by this pull request

Maintenance idea: A way to automate live CLI demos #260

Closed

funilrys assigned veracioux Oct 17, 2021

funilrys added enhancement features labels Oct 17, 2021

funilrys added this to the 4.1.0 milestone Oct 17, 2021

Apply suggestions and create record/upload scripts

aa75d88

Signed-off-by: Haris Gušić <[email protected]>

veracioux added 2 commits October 29, 2021 13:56

Add remaining IP addresses

42286da

Signed-off-by: Haris Gušić <[email protected]>

Change delay times

f7430a4

Signed-off-by: Haris Gušić <[email protected]>

spirillen reviewed Oct 30, 2021

View reviewed changes

Add @veracioux to the list of contributors.

f6c666e

funilrys mentioned this pull request Dec 25, 2021

FEATURE: automagically push/update demo to pyfunceble.github.io #287

Open

funilrys approved these changes Dec 25, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/dev' into tuterm_demo

f537119

funilrys merged commit b3c4442 into funilrys:dev Dec 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automated CLI demonstrations #262

Add automated CLI demonstrations #262

veracioux commented Oct 17, 2021

spirillen Oct 17, 2021 •

edited

Loading

veracioux Oct 17, 2021

funilrys Oct 17, 2021

spirillen Oct 17, 2021 •

edited

Loading

spirillen Oct 17, 2021

funilrys Oct 17, 2021

veracioux Oct 17, 2021

veracioux Oct 17, 2021

spirillen Oct 17, 2021

spirillen Oct 17, 2021

veracioux Oct 17, 2021 •

edited

Loading

funilrys Oct 17, 2021

spirillen commented Oct 17, 2021

funilrys commented Oct 17, 2021

funilrys commented Oct 17, 2021

veracioux commented Oct 17, 2021 •

edited

Loading

veracioux commented Oct 17, 2021

spirillen commented Oct 17, 2021

funilrys commented Oct 27, 2021

funilrys commented Oct 27, 2021

veracioux commented Oct 27, 2021

spirillen commented Oct 28, 2021

veracioux commented Oct 29, 2021

spirillen Oct 30, 2021 •

edited

Loading

spirillen Oct 30, 2021

veracioux Oct 30, 2021

spirillen Oct 31, 2021

veracioux Oct 31, 2021

spirillen Oct 31, 2021

veracioux Nov 7, 2021

funilrys Nov 7, 2021

spirillen Nov 7, 2021

funilrys left a comment

		c PyFunceble -d 194.187.99.221 # Adult CDN: https://mypdns.org/my-privacy-dns/porn-records/-/issues?scope=all&state=all&label_name[]=IP%3A%3ABlackListing
		c PyFunceble -d 45.136.204.40 # 23 phis: https://github.com/mitchellkrogza/phishing/pull/67

Add automated CLI demonstrations #262

Add automated CLI demonstrations #262

Conversation

veracioux commented Oct 17, 2021

spirillen Oct 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spirillen Oct 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

veracioux Oct 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spirillen commented Oct 17, 2021

funilrys commented Oct 17, 2021

funilrys commented Oct 17, 2021

veracioux commented Oct 17, 2021 • edited Loading

veracioux commented Oct 17, 2021

spirillen commented Oct 17, 2021

funilrys commented Oct 27, 2021

funilrys commented Oct 27, 2021

veracioux commented Oct 27, 2021

spirillen commented Oct 28, 2021

veracioux commented Oct 29, 2021

spirillen Oct 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

funilrys left a comment

Choose a reason for hiding this comment

spirillen Oct 17, 2021 •

edited

Loading

spirillen Oct 17, 2021 •

edited

Loading

veracioux Oct 17, 2021 •

edited

Loading

veracioux commented Oct 17, 2021 •

edited

Loading

spirillen Oct 30, 2021 •

edited

Loading