Remove Malware Domain List

[Yuki2718]

Prerequisites

• I verified that this is not a filter issue
  • Filter issues MUST be reported at filter issue tracker
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uBlockOrigin
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
• I tried to reproduce the issue when...
  • uBlock Origin is the only extension
  • uBlock Origin with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uBlock Origin
• I checked the documentation to understand that the issue I report is not a normal behavior

Description

gorhill promised the removal on a condition:
https://www.wilderssecurity.com/threads/ublock-a-lean-and-fast-blocker.365273/page-163#post-2852075

Then just before the one year passes, MDL was updated at Jan. 22, 2020. However, this was just removing one domain from the list (I used WBM https://web.archive.org/web/20191006191919/https://www.malwaredomainlist.com/hostslist/hosts.txt):
https://www.diffchecker.com/1tI1KbuZ
and the list has not been updated for more than 2 months now.

[uBlock-user]

Are you suggesting it should be removed now ?

[Yuki2718]

Are you suggesting it should be removed now ?

Yes!

[gorhill]

The list changed last January: uBlockOrigin/uAssets@5ad7faa#diff-53fe5504d16f8ca7c58a72186493bdef

[Yuki2718]

The list changed last January: uBlockOrigin/uAssets@5ad7faa#diff-53fe5504d16f8ca7c58a72186493bdef

Yeah, I noted that. A question is whether removing one domain should be regarded as active maintenance, particularly when there has been no update for 2 months after that.

[gwarser]

Few people suggested to add https://gitlab.com/curben/urlhaus-filter to "Malware domains" section. For example https://www.reddit.com/r/uBlockOrigin/comments/e0z3zr/expanding_a_malware_domain_list/

[jspenguin2017]

It's been well over a year since anything was added to the filter, although there are removals from time to time: https://github.com/NanoMeow/MDLMirror/blame/master/filter.txt

In fact, I don't believe anything was ever added to the filter since June 2018, when NanoMeow started mirroring the filter.

[jspenguin2017]

@DandelionSprout suggested these filters on Slack:

https://www.dshield.org/feeds/suspiciousdomains_Medium.txt
https://raw.githubusercontent.com/stamparm/maltrail/master/trails/static/suspicious/pua.txt
https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-online.txt

Slack direct link (for those who have access): https://listauthorschat.slack.com/archives/C010N14G4TF/p1586368680366200

[Yuki2718]

There are many lists for malware or phishing, say,
https://hexxiumcreations.github.io/threat-list/hexxiumthreatlist.txt
https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
https://phishing.army/download/phishing_army_blocklist.txt
but my personal opinion is they are not catching up with the latest threat landscape where most of threats live in only few days.
e.g. https://www.akamai.com/us/en/multimedia/documents/state-of-the-internet/soti-security-phishing-baiting-the-hook-report-2019.pdf

In fact, over a 60-day period, Akamai observed more than 2,064,053,300 unique domains commonly associated with malicious activity. Of those, 89% had a lifespan of less than 24 hours, and 94% had a lifespan of less than three days.

Given uBO's update frequency they are at most better-than-nothing.

[DandelionSprout]

I recall the "12 months without an update" criteria mentioned in the Wilders Security thread, was established back when Schacks Adblock Plus Liste was removed last year. At that point the idea was to weed out lists where the maintainers had disappeared from the face of the planet (which happens surprisingly often), and I didn't expect the mere concept of listmakers gaming it and resting too far back on their laurels.

Considering there's a whopping 468 anti-malware lists (562 with anti-phishing lists included) on Filterlists.com that are supported by uBO to whichever degree, surely at least a couple of them must meet the wishes of being up-to-date and thorough? Especially so since I've been advocating behind closed doors on Slack for the removal of DNS-BH Malware Domains for having too many false positives (Nordic construction company sites in particular).

[DandelionSprout]

If we are to believe https://www.malwaredomainlist.com/update.php at face value, then no new domains have been added to Malware Domain List since December 2017, which is normally a pretty bad sign.

[gorhill]

Given uBO's update frequency they are at most better-than-nothing.

Should there be any malware lists selected by default? It has been argued in the past that the browsers have probably better up to date malware list for their "block dangerous and deceptive content" feature.

[DandelionSprout]

I personally think it's pretty important for uBO to show off itself as "We can block malware too, not just ads or trackers". So yes, there should be at least 1 anti-malware/-phishing list enabled by default, if you ask me.

[gorhill]

So at this point it comes down to pick a list with the following qualities:

• Not too large or too small -- between 20,000 and 50,000 entries, so as to not change too much uBO's expected base memory footprint and launch time
• Long track record of being well-maintained
• Little to no issue of false positives

[Yuki2718]

Should there be any malware lists selected by default? It has been argued in the past that the browsers have probably better up to date malware list for their "block dangerous and deceptive content" feature.

I say no. It has been years after I became tired of malware testing, but from my experience and also some others' reports e.g.
https://malwaretips.com/threads/updated-29-12-2018-browser-extension-comparison-malwares-and-phishings.80915/
surely Google SafeBrowsing tends to be generally more effective than any downloadable lists - also note about 90% of the current list of Malware Domains are sourced from GSB. Yet two things to keep in mind are some people disable GSB for privacy concern, and even GSB has been beaten by bad actors - @krystian3w and I have occasionally seen such cases on AdGuardFilters repo. More and more reports claim that the number of bad domains are now tremendous and they are becoming more short-lived. I don't say malware lists are useless, as there are still some relatively long-lived domains slipped through GSB - we just need to throw an illusion away that they'll fairly protect user.

Many of these lists are maintained by small number of people and often have regional/language biases. So I think they shouldn't be enabled by default as they rarely or never come into play, more so if you keep GSB enabled or you happened to live in a certain region, but keeping some lists under malware section as options will be nice particularly for those who turned GSB off.

[Yuki2718]

I personally think it's pretty important for uBO to show off itself as "We can block malware too, not just ads or trackers". So yes, there should be at least 1 anti-malware/-phishing list enabled by default, if you ask me.

I have a different opinion. Surely uBO can block many attacks IF default-deny mode which only geeks use is chosen, and even without that still can block malvertising which is diminishing (https://www.bleepingcomputer.com/news/security/almost-60-percent-of-malicious-ads-come-from-three-ad-providers/), but I think uBO should not be advertised as a security tool - that leads to e.g. people comparing uBO with security addons and talking about good and bad, noobs having false sense of security that they are safe because using uBO with malware filters, and possibly criticize by a security researcher with real-world data of malware and phishing like what I posted earlier. I believe 99% of user use uBO for ads or other annoyances so this won't affect its popularity.

[gorhill]

I think uBO should not be advertised as a security tool - that leads to e.g. people comparing uBO with security addons and talking about good and bad, noobs having false sense of security that they are safe because using uBO with malware filters

This is a good argument, and given this I am leaning toward not having any malware list selected by default. I would still like to provide good stock lists so picking a good list to replace MDL is still something I would like to do.

[Yuki2718]

Quicly tested the lists mentioned. My suggestion is to add only URLhaus filter (once a problem is solved) or at most URLhaus + Phishing Army, unless someone knows better lists.

Details: every lists included some dead domains (yes, despite "online" version) and apparent false positives. But /trails/static/suspicious/pua.txt includes too many dead domains, and given PUA is not as serious as malware, I don't recommend it - we already have Badware risks so better to expand it if needed. Of note, some domains in the PUA list are included in AdGuard Base. I think DShield's list shouldn't be used because the author don't recommend: https://www.dshield.org/xml.html

Our data does include false positives, and we will not remove them. It would make it harder to observe long term trends. If a report is a false positive or not depends to a large extend on the question being asked.

We offer one blocklist, and one blocklist only (http://www.dshield.org /block.txt). Unlike for our other lists, we will remove IPs from this blocklist if asked to.

They're talking about IP lists here, but I guess the comment applies to domain lists too - anyway the list is for suspicious domains and not for confirmed bad ones. Hexxium's list was last updated on Feb. 26 so no update for more than a month, not a good sign for malware list. Also their list includes some domains with 0 detection on VirusToal, not sure if they're FP or FN by VT.

URLhaus online filter doesn't overlap with current Malware domains, tho it won't be needed for those who keep GSB (data are shared and actually most are blocked by it). An advantage of this list is page-level blocking in contrast to domain-level, which is important as bad guys have been abusing trusted domains (https://www.helpnetsecurity.com/2019/03/01/malicious-urls-good-domains/). The problem is these rules are written in forms like
cdn.truelife.vn/webtube/201310/2139273/pianito.exe
which don't work with uBO. If we are to adopt the list, we have to ask @MDLeom to convert all the rules to ABP syntax. Phishing Army includes some obvious FPs and again most are covered by GSB. One of its source is OpenPhish https://openphish.com/feed.txt which may be less FP-prone and also allows page-level blocking but I think their format is not performance-friendly - no prefix "|" and very long URLs mean they have to be checked against every requests, right?

[gorhill]

cdn.truelife.vn/webtube/201310/2139273/pianito.exe which don't work with uBO

That does work with uBO; the absence of the || prefix just mean the pattern can match at any point in the URL, i.e. https://example.org/?u=cdn.truelife.vn/webtube/201310/2139273/pianito.exe would also match.

[Yuki2718]

cdn.truelife.vn/webtube/201310/2139273/pianito.exe which don't work with uBO

That does work with uBO; the absence of the || prefix just mean the pattern can match at any point in the URL, i.e. https://example.org/?u=cdn.truelife.vn/webtube/201310/2139273/pianito.exe would also match.

In this case, the problem is about strict blocking - you can download the exe if the rule was
cdn.truelife.vn/webtube/201310/2139273/pianito.exe
or
||cdn.truelife.vn/webtube/201310/2139273/pianito.exe,
so it must be
||cdn.truelife.vn/webtube/201310/2139273/pianito.exe$document
or
||cdn.truelife.vn/webtube/201310/2139273/pianito.exe$all

[kulfoon]

As per requirement nr 1 from #984 (comment) : "between 20,000 and 50,000 entries," as so far none of lists proposed in the current thread meets this criteria:

~10000 https://phishing.army/download/phishing_army_blocklist.txt
 ~2700 https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-online.txt
 ~2700 https://hexxiumcreations.github.io/threat-list/hexxiumthreatlist.txt
 ~2600 http://www.dshield.org/feeds/suspiciousdomains_Medium.txt
 ~1500 https://raw.githubusercontent.com/stamparm/maltrail/master/trails/static/suspicious/pua.txt
  ~100 https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt

Nor the one I've just found (just a quick search, not a deep investigation nor analysis yet):
https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites, but it might be worth checking coz it's the biggest one as far from lists proposed in the current thread:

~14000 https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hosts
 ~9000 https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list

I know that big (even giant ones) malware hosts lists do exist as well but most of them don't qualify because they are kind of "A big merged/ultimate collection of hosts from reputable sources." and they contain mixed some non-malware related websites like pornsites, social, gambling etc. and hell knows what else and they contain many false positives, examples:
https://github.com/StevenBlack/hosts
https://github.com/mitchellkrogza/Ultimate.Hosts.Blacklist
https://github.com/AdroitAdorKhan/EnergizedProtection

But as I said, I haven't done a deep search so far, perhaps such lists with 20000 - 50000 malware related entries do exist somewhere (like the one which is currently in uBO: "Malware domains" with ~27000 domains).

[gorhill]

between 20,000 and 50,000 entries," as so far none of lists proposed in the current thread meets this criteria

Looking again, I realize MDL is around 1,100 entries -- I had in mind we were talking about the other malware list, which is ~26K entries. So mainly what I am saying is to replace MDL by a list with roughly similar size.

[KonoromiHimaries]

@gorhill i'am suggest replacing with multi list, that just include MDL. like, create new repo and add these list into script https://github.com/KonoromiHimaries/PolishSubFilters/tree/master/scripts

[Yuki2718]

@KonoromiHimaries Why include MDL, it should rather be excluded. We need to find at least one high-quality list. As noted, all the lists (except osint, too short; didn't test) mentioned here included some false positives and thus combining them will increase the rate of FPs. And 99% of entries in these lists will never be hit for each user even without Google Safe Browsing which covers most of those lists.

[KonoromiHimaries]

here included some false positives and thus combining them will increase the rate of FPs

only stable lists, but any unstable filters can be added manual. like FiltersHeroes/KAD#1297

[jawz101]

malware domain lists are useless, IMO.

• your browser likely already has mechanisms to block them
• most entries are not necessarily related to browser-based malware (e.g. communication that an exe in an email attachment might use will never involve your browser)
• malware domains don't last long. Bad guys don't buy domain names for 7 year stretches just to host an old, crusty piece of malware.