Mix the dnstest into urltest v4.0.1.x #164
Comments
|
I give this a big +1 |
|
Let's implement DNS lookup first. |
|
DNS lookup first?? You might be more specific here, I'm lost |
|
Well ... |
funilrys
added a commit
that referenced
this issue
Dec 31, 2021
This patch fixes #164. Indeed, before this patch, we were trying to query URL even if the domain didn't have an IP address. This patch fixes the issue by doing a classic (NS, CNAME, A, AAA) query before trying to query any URL. Note: The DNS Lookup is the same as the one we do for a standard domain/ip test. Contributors: * @mitchellkrogza * @spirillen
|
Ahh got it 😄 |
|
And Happy new year 🥳 🎉 |
|
First test seems fine 👍 |
|
YaY ! Happy new year 🎉 🥳 |
funilrys
added a commit
that referenced
this issue
Oct 4, 2022
commit 0582a27 Author: funilrys <[email protected]> Date: Sat Jul 16 17:50:29 2022 +0200 Bump version to v4.1.2.dev commit 1ddc3a0 Author: funilrys <[email protected]> Date: Sat Jul 16 14:38:45 2022 +0200 Fix issue with encoding. This patch fixes #300. Indeed, before this patch, we were using the system-wide encoding instead of the defacto default UTF-8. Contributors: * @avatartw commit 9a5fada Author: funilrys <[email protected]> Date: Mon Mar 21 19:40:47 2022 +0100 Bump version to v4.1.1 commit 07ad55f Author: funilrys <[email protected]> Date: Mon Mar 21 19:35:53 2022 +0100 Squashed commit of the following: commit 8320b72 Author: funilrys <[email protected]> Date: Mon Mar 21 19:29:53 2022 +0100 Upgrade contributor list. This patch adds @bigdargon to the list of contributors. commit a448673 Author: funilrys <[email protected]> Date: Mon Mar 21 19:24:48 2022 +0100 Better handling of subjects without extension. This patch fixes #297. This patch fixes dead-hosts/dev-center#23. This patch touches the proxy matcher. Indeed, before this patch, I didn't assume that it may be possible to have subject without any extension. This error lead some session to crash shortly before the end. This patch fixes the issue by handling the case that the subject without any extension is given. Contributors: * @spirillen * @bigdargon commit a6f8b9c Author: funilrys <[email protected]> Date: Sat Mar 12 12:41:30 2022 +0100 Bump version to v4.1.0 commit 476a942 Author: funilrys <[email protected]> Date: Sat Mar 12 09:13:37 2022 +0100 Bump version to v4.1.0b18. commit 5cae08e Author: funilrys <[email protected]> Date: Sat Mar 12 09:06:38 2022 +0100 Restrict python-box. This patch fixes #294. Contributors: * @cdgriffith commit 3688974 Author: funilrys <[email protected]> Date: Sun Mar 6 16:55:12 2022 +0100 Bump version to v4.1.0b17 commit 7770336 Merge: 47a4e4b 06398ca Author: funilrys <[email protected]> Date: Sun Mar 6 16:51:56 2022 +0100 Merge branch 'maintenance' into dev commit 06398ca Author: funilrys <[email protected]> Date: Sun Mar 6 16:51:19 2022 +0100 Add missing variable. commit 47a4e4b Author: funilrys <[email protected]> Date: Sun Mar 6 14:41:01 2022 +0100 Bump version to v4.1.0b16. commit 513a61f Author: funilrys <[email protected]> Date: Sun Mar 6 14:22:26 2022 +0100 Better handling of proxy usage. This patch touches #132. This patch fixes #293. Indeed, before this patch a special case related to the test on onions subject wasn't handled properly. This patch also replace the socks5 with socks5h protocol within the documentation. User that are testing onions and do not have a DNS resolver/server that can handle them are advised to use the socks5h protocol instead of socks5. That will ensure that the DNS lookup is not handled locally. Contributors: * @spirillen commit 7a52279 Author: funilrys <[email protected]> Date: Sat Mar 5 23:55:19 2022 +0100 Bump version to v4.1.0b15. commit d721eef Author: funilrys <[email protected]> Date: Sat Mar 5 23:53:14 2022 +0100 fixup! Add sock capabilities. commit 2f2eddb Author: funilrys <[email protected]> Date: Sat Mar 5 21:53:33 2022 +0100 Bump version to v4.1.0b14. commit 03f9446 Author: funilrys <[email protected]> Date: Sat Mar 5 21:47:48 2022 +0100 Add sock capabilities. commit fdb9a27 Author: funilrys <[email protected]> Date: Sat Mar 5 19:50:13 2022 +0100 Bump version to v4.1.0b13. commit 5b813d7 Author: funilrys <[email protected]> Date: Sat Mar 5 19:45:29 2022 +0100 Introduction of the *.altervista.org SPECIAL rule. This patch fixes #292. From now on, any subjects matching *.altervista.org that return the 403 status codes are supplied as `INACTIVE`. Contributors: * @spirillen commit d73c097 Author: funilrys <[email protected]> Date: Sat Mar 5 19:41:46 2022 +0100 Fix deployment to AUR. commit 824afd9 Author: funilrys <[email protected]> Date: Sat Mar 5 19:23:51 2022 +0100 Add undocumented SPECIAL rules. commit b393b85 Author: funilrys <[email protected]> Date: Sat Mar 5 19:20:17 2022 +0100 Update Copyright year. commit 0fc29e6 Author: funilrys <[email protected]> Date: Sat Mar 5 18:54:38 2022 +0100 Bump version to v4.1.0b12 commit 8738648 Author: funilrys <[email protected]> Date: Sat Mar 5 18:49:04 2022 +0100 Introduction of the proxy settings. This patch fixes #132. Indeed, before this patch, one could use the HTTP_PROXY and HTTPS_PROXY settings. But, PyFunceble wouldn't actually handle some special rules. This patch introduce the `--http-proxy` and `https-proxy` arguments. This patch introduce the ability to define some extra rules - within the configuration file. Indeed, this is something that was missing. Therefore, from now on, you can define some extra rules around your proxies. For example, if you want to use `example.com` as main proxy but want all `.com` and `.org` subjects to be requested through `example.dev`, this is now possible through the configuration: ```yaml proxy: global: http: http://example.com:8080 https: http://example.com:8080 rules: - http: http://example.dev:8080 https: http://example.dev:8080 tld: - com - org ``` Also: * Fix Codeblock syntax in the documentation. Contributors: * @spirillen commit 152dc4e Author: funilrys <[email protected]> Date: Sun Feb 13 18:53:58 2022 +0100 Bump version to v4.1.0b11. commit 22d39cf Author: funilrys <[email protected]> Date: Sun Feb 13 18:28:25 2022 +0100 Add forgotten explicit encoding. This patch fixes #291. Indeed, before this patch there may have been some encoding collision because I forgot to put the encoding explicitely before opening a CSV file. Contributors: * @avatartw commit ccbf6fe Author: funilrys <[email protected]> Date: Sat Feb 12 13:07:03 2022 +0100 Bump version to v4.1.0b10. commit ae46748 Author: funilrys <[email protected]> Date: Sat Feb 12 13:01:48 2022 +0100 Introduction of the '*-lookup-only' argument. This patch fixes #276. Indeed, before this patch, there way no way to tell - from the CLI - that we only want the method X or Z for the lookup. Contributors: * @spirillen commit 8ebfde6 Author: funilrys <[email protected]> Date: Sat Jan 1 17:36:31 2022 +0100 Bump version to v4.1.0b9. commit ab8f5e7 Author: funilrys <[email protected]> Date: Sat Jan 1 17:30:58 2022 +0100 Improvement of the Support of IPv6 DNS. Indeed, before this patch, the support was not complete. This patch complete the support by expecting the following format for any IPv6 with explicit address: [ip]:port If no braket is found, we will provide the default port. commit 2640455 Author: funilrys <[email protected]> Date: Sat Jan 1 16:11:09 2022 +0100 Bump version to v4.1.0b8. commit 16f3898 Author: funilrys <[email protected]> Date: Sat Jan 1 16:07:07 2022 +0100 fixup! Do DNS Lookup before querying URL. commit b01376b Author: funilrys <[email protected]> Date: Sat Jan 1 14:37:52 2022 +0100 fixup! Fix issue with deployment to the AUR. commit 2393ff3 Author: funilrys <[email protected]> Date: Sat Jan 1 13:47:32 2022 +0100 Fix issue with deployment to the AUR. commit 408e277 Author: funilrys <[email protected]> Date: Fri Dec 31 18:40:50 2021 +0100 Bump version to v4.1.0b7. commit cd14dc8 Author: funilrys <[email protected]> Date: Fri Dec 31 18:21:33 2021 +0100 Do DNS Lookup before querying URL. This patch fixes #164. Indeed, before this patch, we were trying to query URL even if the domain didn't have an IP address. This patch fixes the issue by doing a classic (NS, CNAME, A, AAA) query before trying to query any URL. Note: The DNS Lookup is the same as the one we do for a standard domain/ip test. Contributors: * @mitchellkrogza * @spirillen commit 86e2558 Author: spirillen <[email protected]> Date: Fri Dec 31 01:38:33 2021 +0100 Slight change to the DNS IP Have changed the Quad9 IP address to 9.9.9.10 as we should promoted unfiltered DNS services In general, as that hopefully is the most needed in combination with @PyFunceble commit 8505492 Author: funilrys <[email protected]> Date: Fri Dec 31 16:30:48 2021 +0100 Bump version to v4.1.0b6. commit 088947b Author: funilrys <[email protected]> Date: Fri Dec 31 16:11:24 2021 +0100 Workaround Y2K38. This patch fixes #261. Indeed, before this patch, there was no proper solution to the Y2K38 issue - within PyFunceble. This patch try to workaround the issue by rounding the dates to a few weeks before Y2K38 - when the database vendor didn't implement a solution. Contributors: * @spirillen commit 2e2c517 Author: funilrys <[email protected]> Date: Sat Dec 25 22:32:25 2021 +0100 Bump version to v4.1.0b5. commit 2c4ea7d Merge: b3c4442 8514ca5 Author: funilrys <[email protected]> Date: Sat Dec 25 22:28:20 2021 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit 8514ca5 Author: funilrys <[email protected]> Date: Sat Dec 25 20:29:11 2021 +0100 Review Syntax checker. This patch fixes #282. Our testing module was too close to standards. If we consider blogspot.com a Public Suffix and therefore a effective Top-Level domain (eTLD), hello_world.blogspot.com shouldn't be valid. But the really behind the public suffixes is different. Therefore, this patch relaxes the rules regarding sub-domains behind a known public suffix. commit a346154 Author: funilrys <[email protected]> Date: Sat Dec 25 19:09:14 2021 +0100 Fix issue regarding false-positive invalid. This patch touches #282. Indeed, before this patch our self-healing false-positive lookup through DNS may have been incomplete. This patch fixes the issue by making the ruleset for the continuation to the next test method more transparent. commit b3c4442 Merge: f0292cc f537119 Author: Nissar Chababy <[email protected]> Date: Sat Dec 25 17:28:34 2021 +0100 Merge pull request #262 from veracioux:tuterm_demo Add automated CLI demonstrations Contributors: * @veracioux * @spirillen commit f537119 Merge: f6c666e f0292cc Author: funilrys <[email protected]> Date: Sat Dec 25 17:26:45 2021 +0100 Merge remote-tracking branch 'upstream/dev' into tuterm_demo commit f6c666e Author: funilrys <[email protected]> Date: Sat Dec 25 16:56:52 2021 +0100 Add @veracioux to the list of contributors. commit f0292cc Author: spirillen <[email protected]> Date: Sat Dec 25 16:45:03 2021 +0100 Merge: #284 This patch merges #284. This patch fixes #283. Contributors: * @spirillen commit bb80fa5 Author: funilrys <[email protected]> Date: Wed Dec 22 19:06:15 2021 +0100 Fix sphinx error. commit 7b7ca75 Author: funilrys <[email protected]> Date: Wed Dec 22 18:59:02 2021 +0100 Url with scheme and port shouldn't be INVALID. commit ae9333a Author: funilrys <[email protected]> Date: Wed Dec 22 17:19:04 2021 +0100 Bump version to v4.1.0b4. commit 869cab6 Merge: 6a66e71 d4d42bf Author: funilrys <[email protected]> Date: Wed Dec 22 17:13:31 2021 +0100 Merge remote-tracking branch 'origin/chancy-tester' into dev commit 6a66e71 Merge: a8bea19 177a2f0 Author: funilrys <[email protected]> Date: Wed Dec 22 17:13:05 2021 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit d4d42bf Author: funilrys <[email protected]> Date: Wed Dec 22 01:11:45 2021 +0100 Introduction of the chancy tester mode. After discussion with some people and reading some emails of people privately making a comparison between PyFunceble 3 and 4, I decided to write the code for a "chancy" testing mode. Origin: The chancy mode is just a reviving of what I do not agree with - anymore. In PyFunceble 3, the test feedback is almost instant, meaning that the feeling of "speed" is there. But if you read the source code, you will see that it's possible because I chose to ignore a lot of safety to play with my luck, people's luck and machine's luck. Indeed, under PyFunceble 3, it is theoretically possible to have a "line overflow". That means that when: - You are using the multiprocessing mode - You are testing `example.org` and `example.net`, it may have been possible to get the following line as a result: examexampleple.org.net or this: example.orgexample.net That's the fact. If that rare case happens, you may have a corrupted output that you will later parse into your workflow. To solve the problem, I decided to change the workflow design. That's PyFunceble 4. Current State: Under PyFunceble 4, two main things changed: 1. No more single and multiprocessing. 2. No file production from the testing workers. What we used to call the `single-mode` went away and was replaced with a multiprocessing mode that's always there. The idea was that nowadays multiprocessing is the way. On the other hand, I killed the possible "overflow" of PyFunceble 3 by following a clear data workflow that makes output safe. This is an approximate overview of the actual workflow: --- Read Subject ---> Tester Queue \ ---> N Tester Process ---> Producer Queue / 1 Producer Process <--- / Output File | Graphical CLI Output <--- With this workflow, I'm sure that only ONE single process is reporting into any of the generated files. That means that the issue I previously described can't happen. That's the safe workflow. The downside of this workflow is that people think that everything is slow. It is. But for your safety. If you look at some very verbose logs, you will see that the actual testing is as quick as possible (for your hardware) but it's the file generation that is slow. That's because it goes one tested subject at a time - for your safety. Chancy Mode: The chancy mode brings back a PyFunceble 3 "like" way of handling data. That means that if used, no safety is given. An overflow can happen, but the feeling of "speed" is back. This mode exists but won't be actively maintained. To reduce long term maintenance of this mode, it has been built on top of the maintained and developed components of the workflow. Meaning that as long as no new component is built, this mode is "self-maintained". commit 177a2f0 Author: funilrys <[email protected]> Date: Tue Dec 21 16:55:27 2021 +0100 Add @gwarser to the list of contributors. commit a8bea19 Author: funilrys <[email protected]> Date: Tue Dec 21 16:04:03 2021 +0100 Bump version to v4.1.0b3. commit 1806e10 Author: funilrys <[email protected]> Date: Tue Dec 21 15:38:39 2021 +0100 Handle DNS name too long. Indeed before this patch, some DNS name which were longer than 255 octets may produce some fatal error. commit 62d799c Author: funilrys <[email protected]> Date: Sun Dec 19 12:11:15 2021 +0100 fixup! Bump version to v4.1.0b2. commit dad28cc Merge: 1310461 b5ecdc2 Author: funilrys <[email protected]> Date: Sat Dec 18 22:59:43 2021 +0100 Bump version to v4.1.0b2. Also: Fix #272. commit 1310461 Author: funilrys <[email protected]> Date: Sat Dec 18 22:53:09 2021 +0100 Bump version to v4.1.0b1. commit 213a087 Author: funilrys <[email protected]> Date: Sat Dec 18 19:59:12 2021 +0100 Add max internal max rescursion depth. Indeed, before this patch, the CNAME follower when doing HTTP request may habe been able to run without a limit. This patch fixes it by allowing a max depth of 60. Any depth above 60 will just be skipped. commit ebc02c6 Author: funilrys <[email protected]> Date: Sat Dec 18 20:11:18 2021 +0100 Introduction of a the max-http-retries arg. Indeed, before this patch, there was no way to control the maximum number of retries PyFunceble is allowed to perform when doing an HTTP request. This patch fixes that hole by adding the `--max-http-retries` argument. commit fdc4f9c Author: funilrys <[email protected]> Date: Sat Dec 18 22:31:46 2021 +0100 Logical fix. Indeed, before this patch, we didn't actively checked if the inactive and autocontinue dataset lookup was deactivated before doing some backup or update of the dataset. This patch fixes the issue by checking if we are authorized to process such dataset before even trying to do so. commit d31b9b1 Author: funilrys <[email protected]> Date: Sat Dec 18 22:31:16 2021 +0100 Fix issue with counter. commit a85eafb Author: funilrys <[email protected]> Date: Sat Dec 18 22:29:58 2021 +0100 Review requester initialization. Indeed, before this patch, the way the requester was initialized wasn't safe enough for a multiprocessing environment. commit c487a48 Author: funilrys <[email protected]> Date: Sat Dec 18 21:44:22 2021 +0100 Add a way to differs worker startup and creation. Indeed, before this patch, as soon as something is put inside an input queue of a worker, the worker will be automatically created. But, at sometime, for example for worker that should only run at the end, we do not want that behavior. Therefore, this patch introduce a way to differ the creation and startup of a process - when necessary. This patch apply for example for the file sorter that is actually only necessary at the end of a test. commit cb19031 Author: funilrys <[email protected]> Date: Sun Dec 5 19:05:36 2021 +0100 Bump version ot v4.0.2. commit b5ecdc2 Author: funilrys <[email protected]> Date: Sun Dec 5 18:18:30 2021 +0100 Extract the registrar from the WHOIS record. This patch fixes #272. Indeed, before this patch, there was no way to get access to the registrar from the WHOIS record. This patch tries to solve the issue by adding a registrar extractor. This patch also introduce the `--registrar` and `--max-registrar` arguments that can be used to display the top N registrar that were extracted. Using the `--registrar` argument also generate a `logs/registrar.txt` file that can be used to list the all registrar found while testing the current file. Necessary Migration: -- All migration will be automatically triggered. * csv file: The whois.csv file needs to be migrated in order for it to have the `registrar` column. * mariadb: The whois_record tables need to be migrated in order for it to have the `registrar` column. Contributors: * @ryanbr commit d9f93ea Author: funilrys <[email protected]> Date: Sun Dec 5 17:34:48 2021 +0100 Make regex case insensitive. commit f7430a4 Author: Haris Gušić <[email protected]> Date: Fri Oct 29 13:57:02 2021 +0200 Change delay times Signed-off-by: Haris Gušić <[email protected]> commit 42286da Author: Haris Gušić <[email protected]> Date: Fri Oct 29 13:56:47 2021 +0200 Add remaining IP addresses Signed-off-by: Haris Gušić <[email protected]> commit aa75d88 Author: Haris Gušić <[email protected]> Date: Sun Oct 17 18:39:43 2021 +0200 Apply suggestions and create record/upload scripts Signed-off-by: Haris Gušić <[email protected]> commit 8c79508 Author: Haris Gušić <[email protected]> Date: Sun Oct 17 01:54:11 2021 +0200 Add automated CLI demos (proof of concept) Signed-off-by: Haris Gušić <[email protected]>
funilrys
added a commit
that referenced
this issue
Nov 20, 2022
commit 77d7250 Author: funilrys <[email protected]> Date: Sun Nov 20 12:05:27 2022 +0100 Bump version to v4.1.3a7 commit 9d59070 Author: funilrys <[email protected]> Date: Sun Nov 20 11:54:10 2022 +0100 fixup! Adjust timeout checker. commit cf9688e Author: smed79 <[email protected]> Date: Tue Nov 8 04:13:32 2022 +0100 Fix typo #317 commit b0c5f7b Author: funilrys <[email protected]> Date: Thu Nov 3 17:23:49 2022 +0100 Bump version to v4.1.3a6 commit 618620e Author: funilrys <[email protected]> Date: Thu Nov 3 17:03:34 2022 +0100 Adjust timeout checker. This patch touches #316. Indeed, before this patch, it may have been impossible to set the timeout to 0. commit 1a84b35 Author: funilrys <[email protected]> Date: Sun Oct 30 16:38:42 2022 +0100 fixup! Conditional Check: Timeout. commit 942bf2c Author: funilrys <[email protected]> Date: Sun Oct 30 16:30:27 2022 +0100 Fix linting issues. commit 89b34dd Author: funilrys <[email protected]> Date: Sun Oct 30 16:20:40 2022 +0100 Bump version to v4.1.3a5 commit cd01b10 Merge: fd13855 60a6506 Author: funilrys <[email protected]> Date: Sun Oct 30 16:17:03 2022 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit 60a6506 Author: funilrys <[email protected]> Date: Sun Oct 30 15:56:07 2022 +0100 Conditional Check: Timeout. This patch fixes #316. Indeed, before this patch, it may have been possible to crash pyfunceble by setting nagative timeout. This patch fixes the issue by checking the timeout value before injecting the configuration keys into the mechanism. Contributors: * @spirillen commit b465290 Author: funilrys <[email protected]> Date: Sun Oct 30 15:52:59 2022 +0100 Fix issue when configuration key is missing. Indeed before this patch, it may have been possible to let pyfunceble crash through a missing key. This patch fixes the issue by checking if the read keys exists. commit 4056525 Author: funilrys <[email protected]> Date: Sun Oct 30 15:43:21 2022 +0100 Decrease logging level for urllib3. This patch fixes #315. Indeed, before this patch the urllib3 package was "allowed" of logging some warnings to STDOUT. This patch fixes the issues by only allowind CRITICAL logs. Contributors: * @spirillen commit 8c74caf Author: funilrys <[email protected]> Date: Sun Oct 30 11:39:18 2022 +0100 Fix issue with exception handling. This patch fixes #314. Indeed, our exception handling wasn't broad enough. This patch fixes the issue by handling all possible yaml parsing error. Contributors: * @bigdargon commit fe717b2 Author: funilrys <[email protected]> Date: Sun Oct 30 11:06:35 2022 +0100 Fix issue while loading configuration files. Indeed, if one provides an invalid yaml file, PyFunceble may crash. This patch fixes the issue by checking if the given inputs are really in the expected format. commit 32d0a97 Author: funilrys <[email protected]> Date: Tue Oct 25 13:25:01 2022 +0200 Adjust windows requirements file. This patch touches #310. Indeed, the windows requirements file wasn't matching the standard one. This is the first step in trying to solve the issue described in #310. Contributors: * @DandelionSprout commit fd13855 Author: funilrys <[email protected]> Date: Tue Oct 25 13:16:58 2022 +0200 Update version.yaml. This patch fixes #312. Contributors: * @spirillen commit bfd6120 Author: funilrys <[email protected]> Date: Mon Oct 17 19:06:58 2022 +0200 Bump version to v4.1.3a4. commit e94f2ac Author: funilrys <[email protected]> Date: Mon Oct 17 19:06:37 2022 +0200 fixup! Improvement of exception handling around proxies. commit 974b194 Author: funilrys <[email protected]> Date: Sun Oct 16 21:25:23 2022 +0200 Bump version to v4.1.3a3. commit a2f1594 Author: funilrys <[email protected]> Date: Sat Oct 15 14:02:35 2022 +0200 Improvement of exception handling around proxies. This patch fixes #303. Indeed, before this patch, we were rising exceptions to prevent unknown safe. This patch simply remove those exceptions because the following conditions already handle a broad scopes of possible problems. Contributors: * @spirillen commit 20d4232 Author: funilrys <[email protected]> Date: Fri Oct 7 22:38:58 2022 +0200 Fix typo. commit e5e4552 Author: funilrys <[email protected]> Date: Wed Oct 5 20:16:30 2022 +0200 Introduction of a note about the switch mechanism. This patch touches #265. This patch also introduces some basic examples at in the epilog of the CLI. commit ad9a7f4 Author: funilrys <[email protected]> Date: Wed Oct 5 19:20:05 2022 +0200 Bump version to v4.1.3a2 commit b8a9a61 Author: funilrys <[email protected]> Date: Wed Oct 5 18:51:41 2022 +0200 Better handling of requests errors. This patch (potentially) fixes #303. Indeed, before this patch, we didn't handle all possible requests exceptions. This may have caused some issues similar to #303. This patch solves the issue by handling a broader exception. Contributors: * @spirillen commit 3d209f9 Author: funilrys <[email protected]> Date: Wed Oct 5 18:31:41 2022 +0200 Bump version to 4.1.3a1 commit 62671df Author: funilrys <[email protected]> Date: Sat Sep 24 02:55:43 2022 +0200 Fix FUNDING.yml file. commit 24c9628 Author: funilrys <[email protected]> Date: Sat Sep 24 02:52:43 2022 +0200 Maintenance: GitHub Workflows. * Add Tests with Python 3.10. * Upgrade setup-python action. commit d9a78f4 Author: funilrys <[email protected]> Date: Sat Sep 24 02:41:30 2022 +0200 Remove .pylintrc. From now on the setup has to made inside the setup.cfg file. commit 114d397 Author: funilrys <[email protected]> Date: Sat Sep 24 02:37:27 2022 +0200 Adjust requirements. commit 2ccc39d Merge: de8d27b 0582a27 Author: funilrys <[email protected]> Date: Sun Jul 17 17:01:19 2022 +0200 Merge remote-tracking branch 'origin/dev' into maintenance commit 0582a27 Author: funilrys <[email protected]> Date: Sat Jul 16 17:50:29 2022 +0200 Bump version to v4.1.2.dev commit 1ddc3a0 Author: funilrys <[email protected]> Date: Sat Jul 16 14:38:45 2022 +0200 Fix issue with encoding. This patch fixes #300. Indeed, before this patch, we were using the system-wide encoding instead of the defacto default UTF-8. Contributors: * @avatartw commit de8d27b Author: funilrys <[email protected]> Date: Mon May 30 22:35:49 2022 +0200 Disable too-many-function-args. commit 62f989e Author: funilrys <[email protected]> Date: Sun Apr 17 14:14:57 2022 +0200 Fix linting issues. commit da4f27f Author: funilrys <[email protected]> Date: Sun Apr 17 13:53:05 2022 +0200 Fix possible issue with latest version of git. commit 9a5fada Author: funilrys <[email protected]> Date: Mon Mar 21 19:40:47 2022 +0100 Bump version to v4.1.1 commit 07ad55f Author: funilrys <[email protected]> Date: Mon Mar 21 19:35:53 2022 +0100 Squashed commit of the following: commit 8320b72 Author: funilrys <[email protected]> Date: Mon Mar 21 19:29:53 2022 +0100 Upgrade contributor list. This patch adds @bigdargon to the list of contributors. commit a448673 Author: funilrys <[email protected]> Date: Mon Mar 21 19:24:48 2022 +0100 Better handling of subjects without extension. This patch fixes #297. This patch fixes dead-hosts/dev-center#23. This patch touches the proxy matcher. Indeed, before this patch, I didn't assume that it may be possible to have subject without any extension. This error lead some session to crash shortly before the end. This patch fixes the issue by handling the case that the subject without any extension is given. Contributors: * @spirillen * @bigdargon commit 2f759d9 Author: funilrys <[email protected]> Date: Mon Mar 21 19:14:45 2022 +0100 fixup! Apply suggestions and create record/upload scripts commit d969fef Author: funilrys <[email protected]> Date: Mon Mar 21 19:12:33 2022 +0100 Maintenance: .github commit a6f8b9c Author: funilrys <[email protected]> Date: Sat Mar 12 12:41:30 2022 +0100 Bump version to v4.1.0 commit 476a942 Author: funilrys <[email protected]> Date: Sat Mar 12 09:13:37 2022 +0100 Bump version to v4.1.0b18. commit 5cae08e Author: funilrys <[email protected]> Date: Sat Mar 12 09:06:38 2022 +0100 Restrict python-box. This patch fixes #294. Contributors: * @cdgriffith commit 3688974 Author: funilrys <[email protected]> Date: Sun Mar 6 16:55:12 2022 +0100 Bump version to v4.1.0b17 commit 7770336 Merge: 47a4e4b 06398ca Author: funilrys <[email protected]> Date: Sun Mar 6 16:51:56 2022 +0100 Merge branch 'maintenance' into dev commit 06398ca Author: funilrys <[email protected]> Date: Sun Mar 6 16:51:19 2022 +0100 Add missing variable. commit 47a4e4b Author: funilrys <[email protected]> Date: Sun Mar 6 14:41:01 2022 +0100 Bump version to v4.1.0b16. commit 513a61f Author: funilrys <[email protected]> Date: Sun Mar 6 14:22:26 2022 +0100 Better handling of proxy usage. This patch touches #132. This patch fixes #293. Indeed, before this patch a special case related to the test on onions subject wasn't handled properly. This patch also replace the socks5 with socks5h protocol within the documentation. User that are testing onions and do not have a DNS resolver/server that can handle them are advised to use the socks5h protocol instead of socks5. That will ensure that the DNS lookup is not handled locally. Contributors: * @spirillen commit 7a52279 Author: funilrys <[email protected]> Date: Sat Mar 5 23:55:19 2022 +0100 Bump version to v4.1.0b15. commit d721eef Author: funilrys <[email protected]> Date: Sat Mar 5 23:53:14 2022 +0100 fixup! Add sock capabilities. commit 2f2eddb Author: funilrys <[email protected]> Date: Sat Mar 5 21:53:33 2022 +0100 Bump version to v4.1.0b14. commit 03f9446 Author: funilrys <[email protected]> Date: Sat Mar 5 21:47:48 2022 +0100 Add sock capabilities. commit fdb9a27 Author: funilrys <[email protected]> Date: Sat Mar 5 19:50:13 2022 +0100 Bump version to v4.1.0b13. commit 5b813d7 Author: funilrys <[email protected]> Date: Sat Mar 5 19:45:29 2022 +0100 Introduction of the *.altervista.org SPECIAL rule. This patch fixes #292. From now on, any subjects matching *.altervista.org that return the 403 status codes are supplied as `INACTIVE`. Contributors: * @spirillen commit d73c097 Author: funilrys <[email protected]> Date: Sat Mar 5 19:41:46 2022 +0100 Fix deployment to AUR. commit 824afd9 Author: funilrys <[email protected]> Date: Sat Mar 5 19:23:51 2022 +0100 Add undocumented SPECIAL rules. commit b393b85 Author: funilrys <[email protected]> Date: Sat Mar 5 19:20:17 2022 +0100 Update Copyright year. commit 0fc29e6 Author: funilrys <[email protected]> Date: Sat Mar 5 18:54:38 2022 +0100 Bump version to v4.1.0b12 commit 8738648 Author: funilrys <[email protected]> Date: Sat Mar 5 18:49:04 2022 +0100 Introduction of the proxy settings. This patch fixes #132. Indeed, before this patch, one could use the HTTP_PROXY and HTTPS_PROXY settings. But, PyFunceble wouldn't actually handle some special rules. This patch introduce the `--http-proxy` and `https-proxy` arguments. This patch introduce the ability to define some extra rules - within the configuration file. Indeed, this is something that was missing. Therefore, from now on, you can define some extra rules around your proxies. For example, if you want to use `example.com` as main proxy but want all `.com` and `.org` subjects to be requested through `example.dev`, this is now possible through the configuration: ```yaml proxy: global: http: http://example.com:8080 https: http://example.com:8080 rules: - http: http://example.dev:8080 https: http://example.dev:8080 tld: - com - org ``` Also: * Fix Codeblock syntax in the documentation. Contributors: * @spirillen commit 152dc4e Author: funilrys <[email protected]> Date: Sun Feb 13 18:53:58 2022 +0100 Bump version to v4.1.0b11. commit 22d39cf Author: funilrys <[email protected]> Date: Sun Feb 13 18:28:25 2022 +0100 Add forgotten explicit encoding. This patch fixes #291. Indeed, before this patch there may have been some encoding collision because I forgot to put the encoding explicitely before opening a CSV file. Contributors: * @avatartw commit ccbf6fe Author: funilrys <[email protected]> Date: Sat Feb 12 13:07:03 2022 +0100 Bump version to v4.1.0b10. commit ae46748 Author: funilrys <[email protected]> Date: Sat Feb 12 13:01:48 2022 +0100 Introduction of the '*-lookup-only' argument. This patch fixes #276. Indeed, before this patch, there way no way to tell - from the CLI - that we only want the method X or Z for the lookup. Contributors: * @spirillen commit 8ebfde6 Author: funilrys <[email protected]> Date: Sat Jan 1 17:36:31 2022 +0100 Bump version to v4.1.0b9. commit ab8f5e7 Author: funilrys <[email protected]> Date: Sat Jan 1 17:30:58 2022 +0100 Improvement of the Support of IPv6 DNS. Indeed, before this patch, the support was not complete. This patch complete the support by expecting the following format for any IPv6 with explicit address: [ip]:port If no braket is found, we will provide the default port. commit 2640455 Author: funilrys <[email protected]> Date: Sat Jan 1 16:11:09 2022 +0100 Bump version to v4.1.0b8. commit 16f3898 Author: funilrys <[email protected]> Date: Sat Jan 1 16:07:07 2022 +0100 fixup! Do DNS Lookup before querying URL. commit b01376b Author: funilrys <[email protected]> Date: Sat Jan 1 14:37:52 2022 +0100 fixup! Fix issue with deployment to the AUR. commit 2393ff3 Author: funilrys <[email protected]> Date: Sat Jan 1 13:47:32 2022 +0100 Fix issue with deployment to the AUR. commit 408e277 Author: funilrys <[email protected]> Date: Fri Dec 31 18:40:50 2021 +0100 Bump version to v4.1.0b7. commit cd14dc8 Author: funilrys <[email protected]> Date: Fri Dec 31 18:21:33 2021 +0100 Do DNS Lookup before querying URL. This patch fixes #164. Indeed, before this patch, we were trying to query URL even if the domain didn't have an IP address. This patch fixes the issue by doing a classic (NS, CNAME, A, AAA) query before trying to query any URL. Note: The DNS Lookup is the same as the one we do for a standard domain/ip test. Contributors: * @mitchellkrogza * @spirillen commit 86e2558 Author: spirillen <[email protected]> Date: Fri Dec 31 01:38:33 2021 +0100 Slight change to the DNS IP Have changed the Quad9 IP address to 9.9.9.10 as we should promoted unfiltered DNS services In general, as that hopefully is the most needed in combination with @PyFunceble commit 8505492 Author: funilrys <[email protected]> Date: Fri Dec 31 16:30:48 2021 +0100 Bump version to v4.1.0b6. commit 088947b Author: funilrys <[email protected]> Date: Fri Dec 31 16:11:24 2021 +0100 Workaround Y2K38. This patch fixes #261. Indeed, before this patch, there was no proper solution to the Y2K38 issue - within PyFunceble. This patch try to workaround the issue by rounding the dates to a few weeks before Y2K38 - when the database vendor didn't implement a solution. Contributors: * @spirillen commit 2e2c517 Author: funilrys <[email protected]> Date: Sat Dec 25 22:32:25 2021 +0100 Bump version to v4.1.0b5. commit 2c4ea7d Merge: b3c4442 8514ca5 Author: funilrys <[email protected]> Date: Sat Dec 25 22:28:20 2021 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit 8514ca5 Author: funilrys <[email protected]> Date: Sat Dec 25 20:29:11 2021 +0100 Review Syntax checker. This patch fixes #282. Our testing module was too close to standards. If we consider blogspot.com a Public Suffix and therefore a effective Top-Level domain (eTLD), hello_world.blogspot.com shouldn't be valid. But the really behind the public suffixes is different. Therefore, this patch relaxes the rules regarding sub-domains behind a known public suffix. commit a346154 Author: funilrys <[email protected]> Date: Sat Dec 25 19:09:14 2021 +0100 Fix issue regarding false-positive invalid. This patch touches #282. Indeed, before this patch our self-healing false-positive lookup through DNS may have been incomplete. This patch fixes the issue by making the ruleset for the continuation to the next test method more transparent. commit b3c4442 Merge: f0292cc f537119 Author: Nissar Chababy <[email protected]> Date: Sat Dec 25 17:28:34 2021 +0100 Merge pull request #262 from veracioux:tuterm_demo Add automated CLI demonstrations Contributors: * @veracioux * @spirillen commit f537119 Merge: f6c666e f0292cc Author: funilrys <[email protected]> Date: Sat Dec 25 17:26:45 2021 +0100 Merge remote-tracking branch 'upstream/dev' into tuterm_demo commit f6c666e Author: funilrys <[email protected]> Date: Sat Dec 25 16:56:52 2021 +0100 Add @veracioux to the list of contributors. commit f0292cc Author: spirillen <[email protected]> Date: Sat Dec 25 16:45:03 2021 +0100 Merge: #284 This patch merges #284. This patch fixes #283. Contributors: * @spirillen commit bb80fa5 Author: funilrys <[email protected]> Date: Wed Dec 22 19:06:15 2021 +0100 Fix sphinx error. commit 7b7ca75 Author: funilrys <[email protected]> Date: Wed Dec 22 18:59:02 2021 +0100 Url with scheme and port shouldn't be INVALID. commit ae9333a Author: funilrys <[email protected]> Date: Wed Dec 22 17:19:04 2021 +0100 Bump version to v4.1.0b4. commit 869cab6 Merge: 6a66e71 d4d42bf Author: funilrys <[email protected]> Date: Wed Dec 22 17:13:31 2021 +0100 Merge remote-tracking branch 'origin/chancy-tester' into dev commit 6a66e71 Merge: a8bea19 177a2f0 Author: funilrys <[email protected]> Date: Wed Dec 22 17:13:05 2021 +0100 Merge remote-tracking branch 'origin/maintenance' into dev commit d4d42bf Author: funilrys <[email protected]> Date: Wed Dec 22 01:11:45 2021 +0100 Introduction of the chancy tester mode. After discussion with some people and reading some emails of people privately making a comparison between PyFunceble 3 and 4, I decided to write the code for a "chancy" testing mode. Origin: The chancy mode is just a reviving of what I do not agree with - anymore. In PyFunceble 3, the test feedback is almost instant, meaning that the feeling of "speed" is there. But if you read the source code, you will see that it's possible because I chose to ignore a lot of safety to play with my luck, people's luck and machine's luck. Indeed, under PyFunceble 3, it is theoretically possible to have a "line overflow". That means that when: - You are using the multiprocessing mode - You are testing `example.org` and `example.net`, it may have been possible to get the following line as a result: examexampleple.org.net or this: example.orgexample.net That's the fact. If that rare case happens, you may have a corrupted output that you will later parse into your workflow. To solve the problem, I decided to change the workflow design. That's PyFunceble 4. Current State: Under PyFunceble 4, two main things changed: 1. No more single and multiprocessing. 2. No file production from the testing workers. What we used to call the `single-mode` went away and was replaced with a multiprocessing mode that's always there. The idea was that nowadays multiprocessing is the way. On the other hand, I killed the possible "overflow" of PyFunceble 3 by following a clear data workflow that makes output safe. This is an approximate overview of the actual workflow: --- Read Subject ---> Tester Queue \ ---> N Tester Process ---> Producer Queue / 1 Producer Process <--- / Output File | Graphical CLI Output <--- With this workflow, I'm sure that only ONE single process is reporting into any of the generated files. That means that the issue I previously described can't happen. That's the safe workflow. The downside of this workflow is that people think that everything is slow. It is. But for your safety. If you look at some very verbose logs, you will see that the actual testing is as quick as possible (for your hardware) but it's the file generation that is slow. That's because it goes one tested subject at a time - for your safety. Chancy Mode: The chancy mode brings back a PyFunceble 3 "like" way of handling data. That means that if used, no safety is given. An overflow can happen, but the feeling of "speed" is back. This mode exists but won't be actively maintained. To reduce long term maintenance of this mode, it has been built on top of the maintained and developed components of the workflow. Meaning that as long as no new component is built, this mode is "self-maintained". commit 177a2f0 Author: funilrys <[email protected]> Date: Tue Dec 21 16:55:27 2021 +0100 Add @gwarser to the list of contributors. commit a8bea19 Author: funilrys <[email protected]> Date: Tue Dec 21 16:04:03 2021 +0100 Bump version to v4.1.0b3. commit 1806e10 Author: funilrys <[email protected]> Date: Tue Dec 21 15:38:39 2021 +0100 Handle DNS name too long. Indeed before this patch, some DNS name which were longer than 255 octets may produce some fatal error. commit 62d799c Author: funilrys <[email protected]> Date: Sun Dec 19 12:11:15 2021 +0100 fixup! Bump version to v4.1.0b2. commit dad28cc Merge: 1310461 b5ecdc2 Author: funilrys <[email protected]> Date: Sat Dec 18 22:59:43 2021 +0100 Bump version to v4.1.0b2. Also: Fix #272. commit 1310461 Author: funilrys <[email protected]> Date: Sat Dec 18 22:53:09 2021 +0100 Bump version to v4.1.0b1. commit 213a087 Author: funilrys <[email protected]> Date: Sat Dec 18 19:59:12 2021 +0100 Add max internal max rescursion depth. Indeed, before this patch, the CNAME follower when doing HTTP request may habe been able to run without a limit. This patch fixes it by allowing a max depth of 60. Any depth above 60 will just be skipped. commit ebc02c6 Author: funilrys <[email protected]> Date: Sat Dec 18 20:11:18 2021 +0100 Introduction of a the max-http-retries arg. Indeed, before this patch, there was no way to control the maximum number of retries PyFunceble is allowed to perform when doing an HTTP request. This patch fixes that hole by adding the `--max-http-retries` argument. commit fdc4f9c Author: funilrys <[email protected]> Date: Sat Dec 18 22:31:46 2021 +0100 Logical fix. Indeed, before this patch, we didn't actively checked if the inactive and autocontinue dataset lookup was deactivated before doing some backup or update of the dataset. This patch fixes the issue by checking if we are authorized to process such dataset before even trying to do so. commit d31b9b1 Author: funilrys <[email protected]> Date: Sat Dec 18 22:31:16 2021 +0100 Fix issue with counter. commit a85eafb Author: funilrys <[email protected]> Date: Sat Dec 18 22:29:58 2021 +0100 Review requester initialization. Indeed, before this patch, the way the requester was initialized wasn't safe enough for a multiprocessing environment. commit c487a48 Author: funilrys <[email protected]> Date: Sat Dec 18 21:44:22 2021 +0100 Add a way to differs worker startup and creation. Indeed, before this patch, as soon as something is put inside an input queue of a worker, the worker will be automatically created. But, at sometime, for example for worker that should only run at the end, we do not want that behavior. Therefore, this patch introduce a way to differ the creation and startup of a process - when necessary. This patch apply for example for the file sorter that is actually only necessary at the end of a test. commit cb19031 Author: funilrys <[email protected]> Date: Sun Dec 5 19:05:36 2021 +0100 Bump version ot v4.0.2. commit b5ecdc2 Author: funilrys <[email protected]> Date: Sun Dec 5 18:18:30 2021 +0100 Extract the registrar from the WHOIS record. This patch fixes #272. Indeed, before this patch, there was no way to get access to the registrar from the WHOIS record. This patch tries to solve the issue by adding a registrar extractor. This patch also introduce the `--registrar` and `--max-registrar` arguments that can be used to display the top N registrar that were extracted. Using the `--registrar` argument also generate a `logs/registrar.txt` file that can be used to list the all registrar found while testing the current file. Necessary Migration: -- All migration will be automatically triggered. * csv file: The whois.csv file needs to be migrated in order for it to have the `registrar` column. * mariadb: The whois_record tables need to be migrated in order for it to have the `registrar` column. Contributors: * @ryanbr commit d9f93ea Author: funilrys <[email protected]> Date: Sun Dec 5 17:34:48 2021 +0100 Make regex case insensitive. commit f7430a4 Author: Haris Gušić <[email protected]> Date: Fri Oct 29 13:57:02 2021 +0200 Change delay times Signed-off-by: Haris Gušić <[email protected]> commit 42286da Author: Haris Gušić <[email protected]> Date: Fri Oct 29 13:56:47 2021 +0200 Add remaining IP addresses Signed-off-by: Haris Gušić <[email protected]> commit aa75d88 Author: Haris Gušić <[email protected]> Date: Sun Oct 17 18:39:43 2021 +0200 Apply suggestions and create record/upload scripts Signed-off-by: Haris Gušić <[email protected]> commit 8c79508 Author: Haris Gušić <[email protected]> Date: Sun Oct 17 01:54:11 2021 +0200 Add automated CLI demos (proof of concept) Signed-off-by: Haris Gušić <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
When testing the phishing url it caught my eye that we keep testing URI on a dead domain
Describe the solution you'd like
When testing URI, we should first validate / test the domain
Describe alternatives you've considered
Currently none
Additional context
By appending this feature we can save 1000's of lookups on not ACTIVE domains
We also do a DNS test to the DB at the same time, it becomes a two in one solution, as you don't need to make a DNS test secondly.
The text was updated successfully, but these errors were encountered: