Remove Malware domains #1116

nicolaasjan · 2020-06-19T11:25:43Z

Prerequisites

I verified that this is not a filter issue
- Filter issues MUST be reported at filter issue tracker
This is not a support issue or a question
- Support issues and questions are handled at /r/uBlockOrigin
I performed a cursory search of the issue tracker to avoid opening a duplicate issue
- Your issue may already be reported.
I tried to reproduce the issue when...
- uBlock Origin is the only extension
- uBlock Origin with default lists/settings
- using a new, unmodified browser profile
I am running the latest version of uBlock Origin
I checked the documentation to understand that the issue I report is not a normal behavior

Description

Similar to #984.

I've been watching this list for quite some time now and virtually nothing changes...

The last entry is:
usibw.top phishing private 20200518

usibw.top is a dead domain by now...

I really think it's worth considering the removal of this once great list.

Example of a better alternative at this point would be:
https://gitlab.com/curben/urlhaus-filter (?)

A specific URL where the issue occurs

n/a

Steps to Reproduce:

n/a

Expected behavior:

n/a

Actual behavior:

n/a

Your environment

uBlock Origin version: v1.27.11rc2
Browser Name and version: Firefox 77.0.1
Operating System and version: Linux Mint 19.3

The text was updated successfully, but these errors were encountered:

gorhill · 2020-06-19T12:28:46Z

Replacing both malware list with Online Malicious URL Blocklist?

The other list, Malicious URL Blocklist, is too large to have it enabled by default, and though uBO can efficiently store it in memory, the large size is an issue because:

Default lists have to be part of the package, so this has a negative impact on the size of the package
In Chromium, the storage used figure climbs to 37 MB (from ~20 MB)

nicolaasjan · 2020-06-19T13:21:48Z

Replacing both malware list with Online Malicious URL Blocklist?

The other list, Malicious URL Blocklist, is too large to have it enabled by default, and though uBO can efficiently store it in memory, the large size is an issue because:
* Default lists have to be part of the package, so this has a negative impact on the size of the package

* In Chromium, the storage used figure climbs to 37 MB (from ~20 MB)

https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-online.txt
would be fine , I guess.

I have it in my Custom Filters and it is updated very frequently.

However, I read somewhere that the format is not entirely UBO compatible.
Is that true and if so, is it up to Urlhaus to address that?

gorhill · 2020-06-19T13:35:22Z

I looked at the list and it's fine. Ideally, entries for which there is a path should be prefixed with ||, but given how unique those paths are, it's highly unlikely to lead to false positive. For example:

ak.imgfarm.com/images/nocache/vicinio/100000417/19562-111117113753/[email protected]

Would normally be written as:

||ak.imgfarm.com/images/nocache/vicinio/100000417/19562-111117113753/[email protected]

But the filter is so unique that I really don't see the need for the ||, and actually it's being stored slightly more efficiently without the || (because one less condition to evaluate). So I am fine to have the list as is.

Yuki2718 · 2020-06-19T13:39:46Z

However, I read somewhere that the format is not entirely UBO compatible.
Is that true and if so, is it up to Urlhaus to address that?

Add the list and pick a URL from it for downloading .exe, and then access the site and it will be downloaded without an issue. $document or $all modifier is needed to prevent it.

gorhill · 2020-06-19T13:44:22Z

Yes, I overlooked this. Maybe the best way to approach this is for a list to be able to hint at its primary purpose so that uBO parses the filters to ensure they apply to requests for the main document as well.

Yuki2718 · 2020-06-19T13:48:09Z

Yes, I overlooked this. Maybe the best way to approach this is for a list to be able to hint at its primary purpose so that uBO parses the filters to ensure they apply to requests for the main document as well.

Is that worth troubling? I wonder whether @MDLeom can simply add $all, even for all the rules if choosing specific rules is trouble for him.

nicolaasjan · 2020-06-19T13:51:22Z

However, I read somewhere that the format is not entirely UBO compatible.
Is that true and if so, is it up to Urlhaus to address that?

Add the list and pick a URL from it for downloading .exe, and then access the site and it will be downloaded without an issue. $document or $all modifier is needed to prevent it.

Yes, when I enter the URL (.xpi file) Gorhill mentioned above I get this:

VernonStow · 2020-07-06T00:01:49Z

I noticed today that the Urlhaus author has added $all to the entries targeting specific bad file downloads for the filterlists recommended for uBO. Much appreciated!

https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-online.txt

gwarser · 2020-07-06T08:47:01Z

...ajax-made/1^$all

Hmmm, ^ should be removed or replaced by *.

https://gitlab.com/curben/urlhaus-filter/-/commit/859dfd03c6e9e5af6f1fac29bc9b22d1e0046111

gorhill · 2020-07-06T11:04:42Z

Great, this means I can go ahead with this. So both current malware domain lists will be removed from stock lists and replaced with Online Malicious URL Blocklist.

Related issue: - uBlockOrigin/uBlock-issues#1116

Related issues: - uBlockOrigin/uBlock-issues#1116 - uBlockOrigin/uBlock-issues#984 It has been found that the two malware lists uBO uses by default appear to be essentially no longer maintained. urlhaus list[1] has been identified by the community as being well maintained and as being actually useful as a default malware-related list in uBO, since it's being updated every day from a database of top domains and specific URLs identified as serving malicious content. Additionally, the maintainer of urlhaus list has taken steps to increase compatibility with uBO[2]. The decision has been to replace the current two malware- related lists with urlhaus list, which will be enabled by default in uBO -- and this means that list will be part of uBO's package from now on. For those who have the two removed malware lists enabled, these will be moved to the custom lists section -- they will still be enabled. It is suggested users remove them from their selection of lists as their usefulness at this point is questionable. [1] https://gitlab.com/curben/urlhaus-filter [2] https://gitlab.com/curben/urlhaus-filter/-/commit/859dfd03c6e9e5af6f1fac29bc9b22d1e0046111

Yuki2718 · 2020-07-06T13:36:13Z

Hmmm, ^ should be removed or replaced by *.

Can I ask about what is the exact problem? I mean if ajax-made/1_ or such doesn't proceed.

gwarser · 2020-07-06T14:47:32Z

Can I ask about what is the exact problem?

?

https://gitlab.com/curben/urlhaus-filter/-/commit/859dfd03c6e9e5af6f1fac29bc9b22d1e0046111#0329d79171f0f92caecdb36bd26a669696d1042b_103_106

gorhill · 2020-07-06T14:59:55Z

The ^ is not really a big issue, it's just that it's not really needed. The filters will be dealt with properly by uBO with the ^, but semantically the filters are better without it. I also consider that semantically the * is also not needed, it appears all those URL-based filters are complete, I don't think any of them is truncated mid-token.

Yuki2718 · 2020-07-06T15:19:43Z

@gwarser @gorhill Thanks!

Related issues: - uBlockOrigin/uBlock-issues#1116 - uBlockOrigin/uBlock-issues#984 It has been found that the two malware lists uBO uses by default appear to be essentially no longer maintained. urlhaus list[1] has been identified by the community as being well maintained and as being actually useful as a default malware-related list in uBO, since it's being updated every day from a database of top domains and specific URLs identified as serving malicious content. Additionally, the maintainer of urlhaus list has taken steps to increase compatibility with uBO[2]. The decision has been to replace the current two malware- related lists with urlhaus list, which will be enabled by default in uBO -- and this means that list will be part of uBO's package from now on. For those who have the two removed malware lists enabled, these will be moved to the custom lists section -- they will still be enabled. It is suggested users remove them from their selection of lists as their usefulness at this point is questionable. [1] https://gitlab.com/curben/urlhaus-filter [2] https://gitlab.com/curben/urlhaus-filter/-/commit/859dfd03c6e9e5af6f1fac29bc9b22d1e0046111

curbengh · 2020-07-07T01:15:01Z

Maintainer of urlhaus-filter here. Currently urlhaus-filter-online.txt uses:

||example-bad.com/bad-page.html^$all

Should I use the following syntax instead (i.e. remove the ^)?

||example-bad.com/bad-page.html$all

gorhill · 2020-07-07T01:19:52Z

Yes, second form is better, it's semantically correct -- the ^ is equivalent to regex /[^%.0-9a-z_-]|$/ -- I don't think it's something we expect after the pattern in your case.

Thanks for maintaining that list, I consider this helps make uBO better.

- uBlockOrigin/uBlock-issues#1116 (comment)

curbengh · 2020-07-07T01:34:19Z

Updated the script. The filter will be updated in ~10 hours.

Thanks for maintaining that list, I consider this helps make uBO better.

No problem. I'm a longtime uBO user, always wanted to contribute.

lipici · 2020-07-07T04:27:47Z

i updated to uBlock Origin development build v1.28.1b5 but Malware Domain List 1,104 used out of 1,104 and Malware domains 26,838 used out of 26,853 didnt moved to custom list and https://gitlab.com/curben/urlhaus-filter hasnt be added

L.E: appears only after you purge cache and update filters

- uBlockOrigin/uBlock-issues#1116 (comment)

sith-on-mars · 2020-07-11T09:13:30Z

Does that mean now the new Online Malicious URL Blocklist can completely replace both Malware Domain List and Malware Domain?

If yes, could you please update the recommended lists in different blocking modes, like this page?

gwarser · 2020-07-11T10:04:16Z

https://github.com/uBlockOrigin/uBlock-issues/wiki/_compare/c65978da28d98a402840957bbb11c25b36ad699c...f68f791486395a7b15860ccd4ce5a12fbd51061d

Related issues: - uBlockOrigin/uBlock-issues#1116 - uBlockOrigin/uBlock-issues#984 It has been found that the two malware lists uBO uses by default appear to be essentially no longer maintained. urlhaus list[1] has been identified by the community as being well maintained and as being actually useful as a default malware-related list in uBO, since it's being updated every day from a database of top domains and specific URLs identified as serving malicious content. Additionally, the maintainer of urlhaus list has taken steps to increase compatibility with uBO[2]. The decision has been to replace the current two malware- related lists with urlhaus list, which will be enabled by default in uBO -- and this means that list will be part of uBO's package from now on. For those who have the two removed malware lists enabled, these will be moved to the custom lists section -- they will still be enabled. It is suggested users remove them from their selection of lists as their usefulness at this point is questionable. [1] https://gitlab.com/curben/urlhaus-filter [2] https://gitlab.com/curben/urlhaus-filter/-/commit/859dfd03c6e9e5af6f1fac29bc9b22d1e0046111

uBlock-user added the something to address something to address label Jun 19, 2020

gorhill added a commit to uBlockOrigin/uAssets that referenced this issue Jul 6, 2020

Add urlhaus-filter as packaged third party resource

ddc4249

Related issue: - uBlockOrigin/uBlock-issues#1116

curbengh pushed a commit to curbengh/urlhaus-filter that referenced this issue Jul 7, 2020

fix: more accurate static filter

6643bd0

- uBlockOrigin/uBlock-issues#1116 (comment)

curbengh pushed a commit to curbengh/phishing-filter that referenced this issue Jul 7, 2020

fix: more accurate static filter

14cf36c

- uBlockOrigin/uBlock-issues#1116 (comment)

jspenguin2017 mentioned this issue Jul 8, 2020

Stop updating MDLMirror NanoAdblocker/NanoCore#351

Closed

gwarser added the fixed issue has been addressed label Jul 8, 2020

gwarser closed this as completed Jul 8, 2020

gwarser added the wiki related to wiki label Jul 11, 2020

uBlock-user removed the something to address something to address label Jul 13, 2020

zmni mentioned this issue Jul 14, 2020

UrlHaus by Abuse.ch notracking/hosts-blocklists#398

Closed

superlex mentioned this issue Jul 17, 2020

Consider replacing filter_208_MalwareDomains with Online Malicious URL Blocklist AdguardTeam/FiltersRegistry#364

Closed

crssi mentioned this issue Aug 5, 2020

Obsolete/non-maintained lists StevenBlack/hosts#1367

Closed

Yuki2718 mentioned this issue Nov 30, 2021

Add domains uBlockOrigin/uAssets#10647

Closed

0xRustlang mentioned this issue Dec 29, 2021

Remove "malwaredomains.com" list and replace it with Urlhaus Filterlists serverless-dns/blocklists#32

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Malware domains #1116

Remove Malware domains #1116

nicolaasjan commented Jun 19, 2020

gorhill commented Jun 19, 2020

nicolaasjan commented Jun 19, 2020

gorhill commented Jun 19, 2020 •

edited

Loading

Yuki2718 commented Jun 19, 2020

gorhill commented Jun 19, 2020 •

edited

Loading

Yuki2718 commented Jun 19, 2020

nicolaasjan commented Jun 19, 2020

VernonStow commented Jul 6, 2020 •

edited

Loading

gwarser commented Jul 6, 2020 •

edited

Loading

gorhill commented Jul 6, 2020

Yuki2718 commented Jul 6, 2020

gwarser commented Jul 6, 2020

gorhill commented Jul 6, 2020

Yuki2718 commented Jul 6, 2020

curbengh commented Jul 7, 2020 •

edited

Loading

gorhill commented Jul 7, 2020 •

edited

Loading

curbengh commented Jul 7, 2020 •

edited

Loading

lipici commented Jul 7, 2020 •

edited

Loading

sith-on-mars commented Jul 11, 2020

gwarser commented Jul 11, 2020

Remove Malware domains #1116

Remove Malware domains #1116

Comments

nicolaasjan commented Jun 19, 2020

Prerequisites

Description

A specific URL where the issue occurs

Steps to Reproduce:

Expected behavior:

Actual behavior:

Your environment

gorhill commented Jun 19, 2020

nicolaasjan commented Jun 19, 2020

gorhill commented Jun 19, 2020 • edited Loading

Yuki2718 commented Jun 19, 2020

gorhill commented Jun 19, 2020 • edited Loading

Yuki2718 commented Jun 19, 2020

nicolaasjan commented Jun 19, 2020

VernonStow commented Jul 6, 2020 • edited Loading

gwarser commented Jul 6, 2020 • edited Loading

gorhill commented Jul 6, 2020

Yuki2718 commented Jul 6, 2020

gwarser commented Jul 6, 2020

gorhill commented Jul 6, 2020

Yuki2718 commented Jul 6, 2020

curbengh commented Jul 7, 2020 • edited Loading

gorhill commented Jul 7, 2020 • edited Loading

curbengh commented Jul 7, 2020 • edited Loading

lipici commented Jul 7, 2020 • edited Loading

sith-on-mars commented Jul 11, 2020

gwarser commented Jul 11, 2020

gorhill commented Jun 19, 2020 •

edited

Loading

gorhill commented Jun 19, 2020 •

edited

Loading

VernonStow commented Jul 6, 2020 •

edited

Loading

gwarser commented Jul 6, 2020 •

edited

Loading

curbengh commented Jul 7, 2020 •

edited

Loading

gorhill commented Jul 7, 2020 •

edited

Loading

curbengh commented Jul 7, 2020 •

edited

Loading

lipici commented Jul 7, 2020 •

edited

Loading