"IRN: Adblock-Iran" Regional filterlist breakage and unmaintained

[MasterKia]

Prerequisites

• This is not a support issue or a question
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
• I am running the latest version of uBlock Origin

I tried to reproduce the issue when...

• uBO is the only extension
• uBO with default lists/settings
• using a new, unmodified browser profile

URL(s) where the issue occurs

http://www.ashoora.ir
https://bayanbox.ir
https://asriran.com (http://darunameh.blogfa.com/post/16)

Describe the issue

IRN: Adblock-Iran is one of the regional filterlists in uBO.

• ashoora.ir:
  The .ir/*/banners/ filter from Adblock-Iran blocks ashoora.ir/images/banners/Rahyan_410.gif and ashoora.ir/images/banners/Behesht_410.gif at the bottom of the page.
  https://user-images.githubusercontent.com/17685483/168642056-69a0aaf3-113d-44ec-9f5b-bbf8f2f2925b.png

• bayanbox.ir (Upload Site):
  It's a regular upload site, where you can upload stuff for free.
  The ||bayanbox.ir/view/*/*.gif filter from Adblock-Iran blocks normal gifs like https://bayanbox.ir/view/6514224053252251384/Quby-Sticker-2-12.gif and https://bayanbox.ir/view/4639030023267517922/Kak-Eunsang.gif (I forgot which page those gifs were on, but overall this filter is unnecessary).
  https://user-images.githubusercontent.com/17685483/168645566-ae6e3617-cf06-4bb8-ae01-c24413ae1caa.png

• asriran.com:
  It's a news site.
  The ||cdn.asriran.com^$third-party filter from Adblock-Iran wrongly blocks https://cdn.asriran.com/files/fa/news/1398/6/29/1017722_194.jpg.
  https://user-images.githubusercontent.com/17685483/168646580-6d859e0b-3690-4cee-994c-f3068c0a1b1f.png

Screenshot(s)

ashoora.ir:
https://user-images.githubusercontent.com/17685483/168642139-c79f6336-365e-4390-8d46-d0b6c8f8181b.png

Notes

These filters shouldn't exist:

.ir/*/banners/
||bayanbox.ir/view/*/*.gif
||cdn.asriran.com^$third-party

Configuration

uBlock Origin: 1.42.4
Chromium: 101
filterset (summary): 
  network: 83580
  cosmetic: 71753
  scriptlet: 18184
  html: 0
listset (total-discarded, last updated): 
  added: 
    fanboy-cookiemonster: 31879-83, 19m
  default: 
    user-filters: 0-0, never
    easylist: 66428-51, 2d.21h.53m
    easyprivacy: 26943-121, 3d.3h.5m
    plowe-0: 3676-805, 9d.5h.47m
    ublock-abuse: 75-0, 3d.3h.4m
    ublock-badware: 4149-92, 3d.3h.4m
    ublock-filters: 30970-98, 3d.3h.4m
    ublock-privacy: 215-2, 3d.3h.4m
    ublock-quick-fixes: 220-0, 19m
    ublock-unbreak: 1779-41, 2d.20h.18m
    urlhaus-1: 8632-0, 19m
filterset (user): [empty]
modifiedUserSettings: [none]
modifiedHiddenSettings: [none]
supportStats: 
  allReadyAfter: 235 ms (selfie)
  maxAssetCacheWait: 219 ms

[MasterKia]

Side note:
The current regional Persian filterlist (adblock-iran) isn't maintained (last commit: 5 months ago) unlike other regional lists (For instance: Albania, Indonesian, Indian, Korean, Lithuania and Dutch all have been updated in the last week).
Also the maintainer of adblock-iran has been nonresponsive to issues in the last 5 months (12 open issues, 7 of them submitted by me).

I propose adding the uBOPa (uBO Parsi/Persian) filterlist.
uBOPa's maintainer has been responsive to all 46 issues I've submitted and he updates the list frequently.

Otherwise since adblock-iran is abandoned, I'd have to submit all persian regional issues here.

[JobcenterTycoon]

We know this list already but the list contains many problematic filters like *##*[class*=" adv "]

[MasterKia]

@nimasaj

[MasterKia]

We know this list already but the list contains many problematic filters like *##*[class*=" adv "]

If he removes problematic (mostly cosmetic) filters, is there a chance of adding this list?

[Yuki2718]

No, not only cosmetic. We really can't accept such way too generic rules like *-ads-^$script ||ad0* *_adv-^$image. Note ||ad0* is not even tokenizable.

[MasterKia]

I assume he'd be willing to get rid of those filters you mentioned and only include network filters and non-generic cosmetic rules.

The Adblock-Iran has some redundant or problematic filters too
Some redundant filters on top of my head:
https://github.com/farrokhi/adblock-iran/blob/master/filter.txt#L684-L703
https://github.com/farrokhi/adblock-iran/blob/master/filter.txt#L1263
https://github.com/farrokhi/adblock-iran/blob/master/filter.txt#L280

As for problematic filters, for instance it has 183 ".gif" filters for no apparent reason.
https://user-images.githubusercontent.com/17685483/168655887-bfaea245-f818-4a59-b70c-efe318e0d8be.png

Can't a website upload anything other than ads as a gif?

[JobcenterTycoon]

.gif can be animated so it get most used for ads, other picture formats like .png have a higher quality so it get used for the content. thats why.

[MasterKia]

Fair enough.
If Adblock-Iran is gonna stay unmaintained, do you suggest merging future Persian filters into uBlock filters or EasyList Ads/Privacy?

Although if uBOPa has a chance of getting added to uBO I'm sure uBOPa's maintainer will remove those problematic filters.

[krystian3w]

Maybe possible fork list and enable actively maintaing - if older repo can't be move to new hands (the best create organization to fallback legacy links) or enable new admin user with high privileges.

PS. AdGuard now need ~10 stars on repo and ~10 commits at month to add or change exist Iran list to new.
https://github.com/AdguardTeam/FiltersRegistry/blob/master/filters/ThirdParty/filter_235_adblock-iran/metadata.json

[MasterKia]

Maybe possible fork list and enable actively maintain

Good idea, me and @nimasaj can maintain it.

There are 3 persian filterlist that I know of:
https://github.com/farrokhi/adblock-iran
https://github.com/SlashArash/adblockfa
https://github.com/nimasaj/uBOPa

We could combine these into one new list and maintain it from there.

uBOPa has 15 stars and ~500 commits.

[MasterKia]

I'm combining those 3 lists into one list and removing redundant filters. Currently on hold

[nimasaj]

I find some of the comments in this topic irrelevant or unclear. The filters are either specific to a domain or generic, acting as regular expressions, when a pattern used by a number of websites is found.
The generic filters, besides their benefits, might introduce some wrong triggers, and that's one of the reasons of developing/maintaining the list by adjusting the filters, to either narrow down and/or remove a filter or add exceptions for wrongly affected domains, as it could be seen in other filter lists too. uBOPa is not an exception here and if there are reports of a wrongly blocked element/address for a generic filter, a similar approach will be taken, by either adding an exception, or narrowing down or removing the filter.

The filters in uBOPa are developed based on the keywords used for ad elements/addresses in Iran. And even though some of the filters might not look good/acceptable to you, but the actual case is different and they're performing as they should do.

A regional filter lists is mostly used by the people of that region and if there are issues, it could be reported to the list maintainer. Again, I don't see why there is resistance from people who are not intended to use that list.
I don't want to go in a discussion of proving how bad/wrong some of the filters are in some of the lists that are already used or provided by default by uBO as there can be many examples.

If the goal of having the regional list is to serve the people with most updated and maintained list, my recommendation is to have a trial phase to have the feedback from the users.

[Yuki2718]

@nimasaj One question: are you willing to cooperate with us? Frankly, a number of rules in your list break uBO's effort for it to be efficient blocker. You probably just didn't know and it's not my intention to attack you or your list, but in any case for the list to be considered at least following must be addressed or improved:

1. Remove all unneeded wildcards.

ABP syntax automatically assumes wildcard around each rule, so baner& matches /etc/baner&param= without explicit wildcard. Here uBO-specific exception is that on uBO it matches ?baner& but does not abaner& by itself because here token from the request is abaner while token from the rule is baner. This is the key mechanism of uBO and adding wildcard *baner& breaks this, now it matches abaner& but the number of such rules directly affects performance while the number of rules like baner& not.

*##*[alt="ads"] too (* after # is simply redundant and not needed), * before # makes the rule specific generic which should be used only if necessary and there's no other way. Also note ###id or ##.class is far prefered to ##[class*="something"], we're even discussing whether we should disable rules like the latter by default. I also noticed quite many rules end with ^ if not *, but ^ is a separator matching anything that isn't alphanumerical, _-.% or the end-of-lines, I guess most of those ^ are not needed either.

2. Remove or replace unduly broad generic rules

I read your argument but any regional list can well be used by user in other region, and even in the intended region people visit international sites. The difference with other stock lists are the number and extent of such rules. User may report issues not only to your repo but to uAssets too, here's gorhill's opinion about generic rule which I fully agree with: uBlockOrigin/uBlock-issues#738 (comment) Which rules you regard as potentially too broad is certainly your choice, IIRC some websites use class name adv for legitimate contents to represent "adventure", but there will be many such cases if inspect further. Plus they can be a fascinating bait for anti-adb. You can probably have discussion with @MasterKia about this.

[krystian3w]

End of path will be | in network filtering

[MasterKia]

@Yuki2718 I don't disagree with your points, because I've come across some sites where *##*[class$="..."] broke something and I've repeatedly pointed out the problem of those generic cosmetic rules in several issues.

As for the current state of the Persian filterlist, I hope @nimasaj comes to a conclusion and the uBOPa list would have a chance to get considered.

Otherwise, I have an idea to combine the three well known Persian filterlist into one list, removing redundant/dead filters and the generic network/cosmetic rules and only including site-specific filters (example.ir/ads-link - example.ir##.adsBlock - example.ir##+js(...)).
Filters that are already covered by uBO stock lists and EasyList Ads/Privacy wouldn't be included.

Would such list have a possibility of being considered if I continue expanding it?

[Yuki2718]

That's not what I can answer without internal discussion. I proposed some changes and waiting answer from the author as you see, if he can accept this will be best - we don't know if your new combined list will be well-maintained. But one point in your comment I didn't pointed out is

Filters that are already covered by uBO stock lists and EasyList Ads/Privacy wouldn't be included.

Sure good compatibility with defualt lists is preferable but TBH some regional lists have quite redundancy with EL.

[MasterKia]

More breakage
https://article.tebyan.net/306262/%D8%AD%D8%A7%D9%84%D8%A7-%D8%A7%DA%AF%D8%B1-%D9%85%DB%8C-%D8%AE%D9%88%D8%A7%D9%87%DB%8C%D8%AF-%D9%BE%D9%81%DA%A9-%D8%A8%D8%AE%D9%88%D8%B1%DB%8C%D8%AF

tebyan.net##.theiaStickySidebar

https://user-images.githubusercontent.com/17685483/169347205-f07680a6-fd82-44b6-a598-5cf66bfa4333.png

[mapx-]

You should prepare a PR for all these breakages

[MasterKia]

#13308

[MasterKia]

That's not what I can answer without internal discussion. I proposed some changes and waiting answer from the author as you see, if he can accept this will be best - we don't know if your new combined list will be well-maintained.

Well since it's been a week so I assume the author isn't interested.

I'm still volunteering to maintain a new combined Persian filterlist (Adblock-Iran + uBOPa + Adblock Farsi).
@Yuki2718 May I know whether you discussed that internally?

[Yuki2718]

Discussing

[Yuki2718]

@MasterKia Got only positive response. Once your new list have got matured and is ready to be included, tell us again in some way.

[MasterKia]

Sure thing, thank you for your time.

[MasterKia]

PersianBlocker:
https://raw.githubusercontent.com/MasterKia/PersianBlocker/main/PersianBlocker.txt

I've rewritten filters of adblock-iran from scratch (per site) and I've removed dead sites/filters and added some missing ads/tracker filters for each site.
So as of now, this list should hopefully be a drop-in improved replacement for adblock-iran.

In the next days I'll bring new sites/filters from uBOPa and Adblock Farsi to increase the list's coverage and usefulness.

If the list is acceptable, should I submit a PR for https://github.com/gorhill/uBlock/blob/master/assets/assets.json?

Also there were some breakages coming from EasyList/uAssest which I've marked with "TODO" and I'm going to submit issues for them.

[okiehsch]

If the list is acceptable, should I submit a PR for https://github.com/gorhill/uBlock/blob/master/assets/assets.json?

The only one with write access in that repo is gorhill and right now he is not available, so you can open a pull request but it could take some time until it is either accepted or declined.

[MasterKia]

Alright, I understand.

you can open a pull request until it is either accepted or declined.

I was asking for some kind of approval on the list from [one of the] maintaines before I go and open a pull request since @Yuki2718 told me to inform you when the list is ready and matured.

[okiehsch]

I am one of those maintainers and I am in favour of replacing the regional list with yours and I said as much in the internal discussion.
The final decision is gorhills to make and he has not chimed in as of now.

[MasterKia]

Thank you, that's what I wanted to know, I'll open a pull request and wait to see what happens.

[MasterKia]

"An owner of this repository has limited the ability to open a pull request to users that have contributed to this repository in the past."

I'd be grateful if one of the maintainers could open a pull request on my behalf, otherwise I guess we should wait until @gorhill adds the list.

MasterKia/uBlock@fa838ff

[Yuki2718]

Just wait for his decision with commit in your branch kept.

[Yuki2718]

BTW gorhill seems to be silent in all activities for a few weeks. Probably you have to be patient.

[MasterKia]

May I know what is your opinion on this?
https://github.com/easylist/easylist/issues/12147
EasyList Coverage states Languages for which there are no supplements may have issues resolved in EasyList and currently there is no supplement for Persian but the maintainer arbitrarily rejected fixing those breakages.

I'm opening a pull request for uBlock Unbreak.

[ghost]

Does AdGuard know about the PersianBlocker filter list, @Yuki2718 ?

[Yuki2718]

I think breakage caused by EL should be fixed in EL as far as possible and actually opened many such PRs most of which were merged. If I fix in regional list (Japanese filter in my case), it's either to save my time or because there are too many cases. Maybe you put too many in one PR but I don't know, it's their decision. PR in uAssets will be merged after check until IRN list get replaced by yours.

[MasterKia]

Maybe you put too many in one PR

I agree, I thought of creating several (maybe 18 issues for each site) at first but then I assumed that it would be more irritating than one creating one issue.

[MasterKia]

Does AdGuard know about the PersianBlocker filter list?

@Branzeturi
https://github.com/AdguardTeam/FiltersRegistry/issues/617

[ghost]

Also, perhaps easylist/easylist#12147 may be useful with decision making regarding the PersianBlocker filter list -- or adding filters to the uBO unbreak filter list

[MasterKia]

@gorhill #13259 (comment)

[MasterKia]

And two of my other issues (eghtesadonline.com ads/ashoora.ir breakage) have been closed tonight without any answer.

[MasterKia]

whitelist ads some well known sites (i.e. soft98, asriran, etc).

Soft98:
https://github.com/MasterKia/PersianBlocker/blob/main/PersianBlocker.txt#L1504-L1516

Asriran:
https://github.com/MasterKia/PersianBlocker/blob/main/PersianBlocker.txt#L408-L413

To be specific, I created this issue for soft98 and pushed for over a month to achieve better/effective filters despite soft98's monstrous anti-adblock.
The soft98 cat and mouse game even got to the point where I proposed adding a clever userscript written by @DRSDavidSoft to uBO's stock scriptlets just to counter soft98's pathetic attempts, which gorhill reasonably declined (because the userscript was huge).

Considering all that, how could I be the one who "clearly made a move to whitelist ads of well known sites"?

[MasterKia]

This is one [attempt to whitelist ads from soft98]: farrokhi/adblock-iran#71

Quotes from #10883:
@MasterKia:

Some of these filters from the uBlock filters aren't working anymore (similar filters can be found in the IRN: Adblock-Iran list from @farrokhi)

@DRSDavidSoft:

I tried pushing some filters to the known iranian filters (https://github.com/farrokhi/adblock-iran/blob/master/filter.txt) and that didn't quite work (remnants are still present), so I went ahead and wrote my userscript.

So:

this is clearly a move to whitelist ads some well known sites (i.e. soft98)

It seems you were the one reluctant to address the issues of soft98, why?

[farrokhi]

False positive and misrecognition happens, that's okay. What led to the change was 6 months inactivity without any sign of being alive (e.g. response to issues). We've recently got upset as List-FR had been inactive for a month, even discussed for the worst case.

This is a list that I update when I run into new ads, as well as based on user input. I regularly receive claims of sites being broken without additional details or proof. The breakage may have various reasons and without enough technical details, I cannot verify the claim. Contributors want to push their changes without proper testing, etc, and I need to moderate everything properly. As a maintainer you are not supposed to let every change land into the repo. In this case, this user created a new repo and started spamming on the comments to promote the new list. Contributors should be helpful, not pushy. All these effort on spamming and self promotion could have gone into creating proper bug reports.
There are also numerous examples of proper reports and contributions being merged into the repo (as you can see in git history).

[MasterKia]

creating proper bug reports

I'll let the pictures speak for themselves:

https://user-images.githubusercontent.com/17685483/172455152-0ed624d6-7d02-4b5b-9b9a-c66cd0ba51a5.png

https://user-images.githubusercontent.com/17685483/172455192-1174dad1-13ac-4c9e-8c39-cb8723634709.png

These issues haven't been answered in 6 months, despite being "proper bug reports".