Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support blocking rules for a defined list of domains #330

Closed
8 tasks done
medavox opened this issue Dec 5, 2018 · 22 comments
Closed
8 tasks done

Support blocking rules for a defined list of domains #330

medavox opened this issue Dec 5, 2018 · 22 comments
Labels
declined declined enhancement New feature or request

Comments

@medavox
Copy link

medavox commented Dec 5, 2018

Prerequisites

  • I verified that this is not a filter issue
  • This is not a support issue or a question
  • I performed a cursory search of the issue tracker to avoid opening a duplicate issue
    • Your issue may already be reported.
  • I tried to reproduce the issue when...
    • uBlock Origin is the only extension
    • uBlock Origin with default lists/settings
    • using a new, unmodified browser profile
  • I am running the latest version of uBlock Origin
  • I checked the documentation to understand that the issue I report is not a normal behavior

Description

Certain companies use a large, growable list of domains -- namely Stack Overflow, Medium.

The same static blocking filter rules apply to all sites by those companies.

But at the moment, every filter rule needs to specify all the domains in the list.

It would be helpful if instead I could (eg) !#define a list of domains that a static block rule applies to.

Steps to Reproduce

  1. Open the uBlock Origin Dashboard
  2. Navigate to the My Filters tab
  3. Write a static blocking filter

Current behavior:

An example -- the situation now:

!Medium
medium.com,hackernoon.com,medium.freecodecamp.org##.js-stickyFooter
medium.com,hackernoon.com,medium.freecodecamp.org##.u-tintSpectrum
medium.com,hackernoon.com,medium.freecodecamp.org##.js-metabar
medium.com,hackernoon.com,medium.freecodecamp.org##.u-clearfix.metabar
medium.com,hackernoon.com,medium.freecodecamp.org##.butterBar-message
medium.com,hackernoon.com,medium.freecodecamp.org##.is-other.markup--p-quote.markup--quote
medium.com,hackernoon.com,medium.freecodecamp.org##.js-postActionsBar
medium.com,hackernoon.com,medium.freecodecamp.org##.postActionsBar
medium.com,hackernoon.com,medium.freecodecamp.org##.overlay--lighter.overlay
medium.com,hackernoon.com,medium.freecodecamp.org##.js-metabarMiddle
medium.com,hackernoon.com,medium.freecodecamp.org##.js-postActionsBarContent
medium.com,hackernoon.com,medium.freecodecamp.org##.metabar-block

superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###hero-content
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###herobox
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###newsletter-ad
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###herobox-mini
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###js-gdpr-consent-banner
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###noscript-warning
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com##.js-dismissable-hero
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com##.s-hero--container
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com##.hero-container.s-hero
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###clc-tlb
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###announcement-banner
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###dfp-tlb
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com##.community-bulletin.module
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###left-sidebar
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###overlay-header
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com##header.top-bar:style(position:absolute !important ; margin-top:0px)
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com##.unread-feature-notice.-feature-notice

Desired behavior:

The desired situation (the used syntax is only a suggestion for illustrative purposes):

!#define medium_sites medium.com,hackernoon.com,medium.freecodecamp.org

medium_sites##.js-stickyFooter
medium_sites##.u-tintSpectrum
medium_sites##.js-metabar
medium_sites##.u-clearfix.metabar
medium_sites##.butterBar-message
medium_sites##.is-other.markup--p-quote.markup--quote
medium_sites##.js-postActionsBar
medium_sites##.postActionsBar
medium_sites##.overlay--lighter.overlay
medium_sites##.js-metabarMiddle
medium_sites##.js-postActionsBarContent
medium_sites##.metabar-block

!#define so_sites superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com

so_sites###hero-content
so_sites###herobox
so_sites###newsletter-ad
so_sites###herobox-mini
so_sites###js-gdpr-consent-banner
so_sites###noscript-warning
so_sites##.js-dismissable-hero
so_sites##.s-hero--container
so_sites##.hero-container.s-hero
so_sites###clc-tlb
so_sites###announcement-banner
so_sites###dfp-tlb
so_sites##.community-bulletin.module
so_sites###left-sidebar
so_sites###overlay-header
so_sites##header.top-bar:style(position:absolute !important ; margin-top:0px)
so_sites##.unread-feature-notice.-feature-notice

Your environment

  • uBlock Origin version: 1.17.4
  • Browser Name and version: Firefox 60.0.1 (64-bit)
  • Operating System and version: OS X El Capitan 10.11.6 (15G31)
@uBlock-user uBlock-user added the enhancement New feature or request label Dec 5, 2018
@krystian3w
Copy link

krystian3w commented Dec 5, 2018

/medium\.(com|freecodecamp\.org)/,hackernoon.com##.js-stickyFooter, .u-tintSpectrum, .js-metabar, .u-clearfix.metabar, .butterBar-message, .is-other.markup--p-quote.markup--quote, .js-postActionsBar, .postActionsBar, .overlay--lighter.overlay, .js-metabarMiddle, .js-postActionsBarContent, .metabar-block
superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com###hero-content, #herobox, #newsletter-ad, #herobox-mini, #js-gdpr-consent-banner, #noscript-warning, .js-dismissable-hero, .s-hero--container, .hero-container.s-hero, #clc-tlb, #announcement-banner, #dfp-tlb, .community-bulletin.module, #left-sidebar, #overlay-header, .unread-feature-notice.-feature-notice

superuser.com,serverfault.com,stackoverflow.com,askubuntu.com,mathoverflow.net,stackexchange.com##header.top-bar:style(position:absolute !important ; margin-top:0px)

➖ exceptions disable all,
➖ foundnd bad cosmetic filter is hard / difficulty development,
➖ no possible combine filters when use procedural filtering with with different values.

@medavox
Copy link
Author

medavox commented Dec 6, 2018

Thanks for the reply.

Could you please elaborate on the first two points?

EDIT

I think I understand. These are comma-separated selectors, as defined by CSS, yes?

@krystian3w
Copy link

krystian3w commented Dec 6, 2018

if combine all simple cosmetic filters you must use all in exception filter.

for example hackernoon.com now is blank page and guilty is:

hackernoon.com##.js-stickyFooter, .u-tintSpectrum, .js-metabar, .u-clearfix.metabar, .butterBar-message, .is-other.markup--p-quote.markup--quote, .js-postActionsBar, .postActionsBar, .overlay--lighter.overlay, .js-metabarMiddle, .js-postActionsBarContent, .metabar-block

exception:

hackernoon.com#@#.js-stickyFooter, .u-tintSpectrum, .js-metabar, .u-clearfix.metabar, .butterBar-message, .is-other.markup--p-quote.markup--quote, .js-postActionsBar, .postActionsBar, .overlay--lighter.overlay, .js-metabarMiddle, .js-postActionsBarContent, .metabar-block

only this exception will turn it off and then you have to manually check what has become obsolete or compare with the html code from web development tools (with example <html> / <body> tags has anyone got this class).


I think I understand. These are comma-separated selectors, as defined by CSS, yes?

Probably it was modeled on it, as it was added to Adblock Plus and then others started to support it.

@medavox
Copy link
Author

medavox commented Dec 6, 2018

So the purpose of the exception is only to check which selectors have become obsolete?

@medavox
Copy link
Author

medavox commented Dec 6, 2018

Sorry for opening a feature request for this.

I didn't realise uBlock could already do this.

It didn't occur to me to read the CSS selectors documentation to achieve what I want (I did read the uBlock Wiki and the AdBlock Plus Filters page).

@krystian3w
Copy link

krystian3w commented Dec 6, 2018

I didn't realise uBlock could already do this.

In total, there is no such functionality, it is only similar and more difficult to use.

So the purpose of the exception is only to check which selectors have become obsolete?

example: class butterBar-message moved to body and combined filter break page.

and then be wise here with these ~12 classes of css in the filter is guilty without preview in the developer's tools (uBO logger tree for this rather average, IMO).


!#define so_sites bla,bla,bla

rather, it could be programmed to work almost as if it were handwritten in domain filters to exceptions #@# / @@ or use $badfilter / ,badfilter for "network filter".

@medavox
Copy link
Author

medavox commented Dec 6, 2018

Ah, interesting. Thanks for the workaround!

@gwarser
Copy link

gwarser commented Dec 6, 2018

Not likely to be implemented, similar thing has been asked for dynamic filtering and was rejected (cannot find issue now) and also this introduces more incompatibility to ABP filter syntax. Why not use local server or github/gist and pre-parse list in bash (or any other tool)?

@krystian3w
Copy link

Hmmm... it may be difficult to get a step by step guide or some old one.

I would not cry for the bot, e.g. @travis-ci.

@medavox
Copy link
Author

medavox commented Dec 6, 2018

@gwarser good suggestion for a workaround, but I maintain my own personal filter list which I use on all my devices.

Several of these devices are portable (they don't share the same local network), so this server would have to be on the public internet to be accessible by them all, and I don't want the hassle of administering & maintaining a public server.


Edit if you meant I could pre-process them locally, that is technically an option. But I'd have to update each device manually (running the pre-processor myself, then updating the filter list in the uBlock dashboard), or find a way for uBlock to trigger the local preprocessor to run before updating that filter-list.

@gwarser
Copy link

gwarser commented Dec 6, 2018

I meant, have two versions of your list, current one and rules-dev.txt. After each change to rules-dev.txt you will run some pre-process.sh and push changes to github (you can even use git-hooks to do this automatically on every push). All other devices will update automatically from rules.txt.

@krystian3w
Copy link

or modular build list - maybe...

@gorhill
Copy link
Member

gorhill commented Dec 6, 2018

Sorry for opening a feature request for this.

Actually this is an idea I entertained in the past, having variables to be used to replace long instances of strings which repeats often. I didn't pursue though as it's not clear how this could be implemented. The !#define itself is easy enough, but then the placeholders to receive the replacement is something that must be thought through.

Not likely to be implemented, similar thing has been asked for dynamic filtering and was rejected

This is a very different case than dynamic filtering. uBO does modify/add/delete the output of dynamic filtering rules, so this complicates quite a lot such feature. On the other hand, the output of a filter lists is never modified by uBO (hence "static") -- except for user filters, but even then a feature like the one here would not interfere with uBO merely adding filters in the list (the only programmatic modification allowed).

@medavox
Copy link
Author

medavox commented Dec 6, 2018

My original goal was to define a 'single source of truth' for the sites that many cosmetic filter rules applied to.

I consider this to be largely solved by @krystian3w 's suggestion of comma-separated selectors on the same line -- although as they pointed out, it only works if the action for all the selectors is the same.

With regards to implementation, I would naively suggest using a map/dictionary
(please excuse my terrible Javascript-ish pseudocode, I work mostly with Java & Kotlin):

//example line:
//#!define arbitrary_key_nospaces literally anything
var dictionary;//string:string
for (line in filter_list_file){
    if (line.startsWith('#!define ')){
        name = line.split(' ')[1];//=arbitrary_key_nospaces
        value = line.substring(indexOf(name)+name.length, line.length);//='literally anything'
        dictionary.put(name, value);
    }
    else {
        for((name, value) in dictionary) {
            if(line.contains(name)) {
                line.replace(name, value);
            }
        }
    }
}

Obviously this probably has poor performance.

@gorhill
Copy link
Member

gorhill commented Dec 6, 2018

Ugh.. When I said "the placeholders to receive the replacement is something that must be thought through" I didn't mean I had no clue how to write code. I meant that the placeholder syntax must not interfere with all the current and future parsing code.

Also the consequences of such features must be taken into account: it could be introduced with allowing only very specific places to be used validly (like say to replace only lists of domains for cosmetic filters), but it would be just a matter of time before issues are opened with people complaining that their placeholders are not filled in by uBO when used e;sewhere.

Overall, I lean on declining this, but I will keep the issue opened and see if ever in the future the pressure from filter list maintainers to implement such feature increases.

@medavox
Copy link
Author

medavox commented Dec 6, 2018

I didn't mean I had no clue how to write code

Apologies, I didn't mean to imply that.

I'm aware that many people often turn up to prominent projects like this, and ask/demand that a feature be implemented, with no intention of helping implement that feature. I just didn't want to be another one of those people.

I am also aware (if probably only dimly, due to lack of familiarity with the language & codebase) that there would be rather a lot of issues with implementing this.

Thanks for your feedback.

@jspenguin2017
Copy link

jspenguin2017 commented Dec 7, 2018

I would design it like this:

The identifier of the defined domains set must start with % (or any other special character that does not cause conflicts), and followed by a comma separated list of domains, for example:

!#define %examples www.example.com,www2.example.com

When a domain name is expected (prefix of cosmetic filter and $domain= option), if the character starts with % , then that entry is replaced by the set of domains defined before, for example:

www3.example.com,%examples##.example
||example.com^$script,domain=%examples|www3.example.com

Are the same as:

www3.example.com,www.example.com,www2.example.com##.example
||example.com^$script,domain=www.example.com|www2.example.com|www3.example.com

In terms of code, I would carry a global state storage for the compiler and update the state in:

storage.js > µBlock.compileFilters

The compiler is synchronous, and will probably always be synchronous, so a global state storage should be fine. Or you can pass the state storage as parameter, but that's way more places to change.

@Alhadis
Copy link

Alhadis commented Mar 30, 2021

What about revising the domain component to support shell-style brace expansion? E.g., this:

{foo,bar}.*##.advert

is expanded into

foo.*##.advert
bar.*##.advert

possibly with support for line-continuation:

{askubuntu,mathoverflow,serverfault,\
stack{apps,exchange,overflow},\
superuser}.{com,net\
}###sidebar > *:not(.sidebar-related):not(#hot-network-questions)

Since { and } aren't[1] legal hostname characters, this syntax has no risk of matching a legitimate domain name.

I know it won't solve the broader issue of being unable to assign an identifier to a reusable domain list. However, it's at least a step in the right direction, and certainly better than having to write

askubuntu.*###sidebar > *:not(.sidebar-related):not(#hot-network-questions)
mathoverflow.*###sidebar > *:not(.sidebar-related):not(#hot-network-questions)
serverfault.*###sidebar > *:not(.sidebar-related):not(#hot-network-questions)
stackapps.*###sidebar > *:not(.sidebar-related):not(#hot-network-questions)
stackexchange.*###sidebar > *:not(.sidebar-related):not(#hot-network-questions)
stackoverflow.*###sidebar > *:not(.sidebar-related):not(#hot-network-questions)
superuser.*###sidebar > *:not(.sidebar-related):not(#hot-network-questions)

@liamengland1
Copy link

@Alhadis
askubuntu.*, mathoverflow.*, serverfault.*, stackapps.*, stackexchange.*, stackoverflow.*, superuser.*###sidebar > *:not(.sidebar-related):not(#hot-network-questions)

?

Alhadis added a commit to Alhadis/.files that referenced this issue Mar 30, 2021
uBlock Origin doesn't seem to support multiple hostnames in its extended
filter syntax, which makes this ugly-looking copy+pasta necessary.

References: uBlockOrigin/uBlock-issues#330
@Alhadis
Copy link

Alhadis commented Mar 30, 2021

@llacb47 Brace expansion Is more flexible and compact, not to mention it's immediately familiar to shell users (especially since the * wildcard resembles glob expansion). Compare:

$ printf '%s\n' {askubuntu,mathoverflow,serverfault,stack{apps,exchange,overflow},superuser}.{com,net}
askubuntu.com
askubuntu.net
mathoverflow.com
mathoverflow.net
serverfault.com
serverfault.net
stackapps.com
stackapps.net
stackexchange.com
stackexchange.net
stackoverflow.com
stackoverflow.net
superuser.com
superuser.net

@gorhill
Copy link
Member

gorhill commented Jun 24, 2021

Declined.

If ever other top blockers come up with a solution for this, I will consider adopting it.

@BlazeFTL
Copy link

BlazeFTL commented May 14, 2024

Can This Be Considered Now To Be Used For Special Cases Only ?
Or Just For Using In My Filter's
Site Owner Buying Domains In BULk For Evading AdBlock And It Would Also Reduce Filter Maintainers Burden
Ex - http://gplinks.co/rJgS
#3212 (comment)
Can Provide More Examples Where Site Owners Using 1-2 Domains Everyday

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
declined declined enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

9 participants