Skip to content
This repository has been archived by the owner on Nov 6, 2023. It is now read-only.

Batch removal of duplicated test urls #14924

Merged
merged 24 commits into from
Mar 20, 2018
Merged

Batch removal of duplicated test urls #14924

merged 24 commits into from
Mar 20, 2018

Conversation

cschanaj
Copy link
Collaborator

@cschanaj cschanaj commented Mar 17, 2018

@Hainish A dirty command is used to remove duplicated test urls found; log.txt is the Travis output of the failing test in #14660

Block #14660

TRAVIS_REPLACE='s/^ERROR \([^:]*\): Duplicated test URL found \(.*\)$/\1 \2/g'

grep ^ERROR ~/log.txt  | tr -d '\r' | sed "$TRAVIS_REPLACE" | sort -u | while read FILE HOST; 
do 
    REGEX_ESCAPED_HOST=`python3.6 -c "import re; print(re.escape('$HOST'))"`;  
    sed -i "0,/<test.*\"$REGEX_ESCAPED_HOST\"/{//d;}" $FILE; 
done

See https://stackoverflow.com/questions/23696871/how-to-remove-only-the-first-occurrence-of-a-line-in-a-file-using-sed

TODO

16 rulesets failed fetch-test

  • Destructoid.xml (3672bc6)
  • Hdfcbank.com.xml (59c6493)
  • Komoona.com.xml (115b6d7)
  • PropellerAds.xml (8888416)
  • WP-Engine.xml (fddbf28)
  • Wemfbox.ch.xml (f0ad991)
  • cloud66.com.xml (f8e49bf)
  • greenhouseci.com.xml (fc55268)
  • JD.com-Problematic.xml
  • espivblogs.net.xml
  • Twenga.com.xml
  • Fenland.gov.uk.xml
  • Wandoujia.xml
  • luosimao.com.xml
  • tanx.com.xml

6 rulesets failed validations-test

2 rulesets having more than 1 duplicates

<test url="http://panel.cloudatcost.com:12345/" />
<test url="http://panel.cloudatcost.com:59848/console.html" />

<rule from="^http:" to="https:" />

<test url="http://panel.cloudatcost.com/" />
<test url="http://members.cloudatcost.com/" />
<test url="http://panel.cloudatcost.com/" />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are even more duplicates.

@J0WI
Copy link
Contributor

J0WI commented Mar 17, 2018

Rulesets that don't pass our tests anyway should be excluded in this PR. Maybe some of them are already whitelisted and the hash needs to be updated. I think just updating the hash would be fine here.

@cschanaj
Copy link
Collaborator Author

@J0WI thank for pointing out that some rulesets having more than 1 duplicates. I have updated the top comment and I will go through them later.

Pasu Chan Chak Shing added 16 commits March 18, 2018 08:57
Remove store.destructoid.com, gone from dns
Remove img.propellerads.com, gone from dns
Remove 20minro-ssl.wemfbox.ch, timeout

Remove qs-ssl.wemfbox.ch, cert-chain

Remove complicate rule for qs.wemfbox.ch
Remove app.greenhouseci.com, mismatch
Remove birdseye.cloud66.com, expired
Remove complicated rules for press.wpengine.com, destination gone from dns
Remove imaging.hdfcbank.com, cert-chain

Remove punjabadds.hdfcbank.com, cert-chain
@cschanaj cschanaj closed this Mar 18, 2018
@cschanaj cschanaj reopened this Mar 18, 2018
@cschanaj cschanaj closed this Mar 18, 2018
@cschanaj cschanaj reopened this Mar 18, 2018
Pasu Chan Chak Shing added 4 commits March 18, 2018 15:49
Remove complicated rules for c-nfa.jd.com, obsolete
Remove eleftheriako-giro-giro.espivblogs.net, timeout
Fenland.gov.uk.xml, Wandoujia.xml, luosimao.com.xml, tanx.com.xml required modifications too complicated to be performed in this PR.
@cschanaj cschanaj closed this Mar 18, 2018
@cschanaj cschanaj reopened this Mar 18, 2018
@cschanaj cschanaj closed this Mar 18, 2018
@cschanaj cschanaj reopened this Mar 18, 2018
@cschanaj cschanaj closed this Mar 18, 2018
@cschanaj cschanaj reopened this Mar 18, 2018
@cschanaj
Copy link
Collaborator Author

@Hainish @J0WI Travis is finally passing now.

@J0WI
Copy link
Contributor

J0WI commented Mar 18, 2018

squat.gr and mobilityrpmcc.hdfcbank.com WFM, all other changes LGTM.

@cschanaj cschanaj closed this Mar 20, 2018
@cschanaj cschanaj reopened this Mar 20, 2018
@cschanaj cschanaj closed this Mar 20, 2018
@cschanaj cschanaj reopened this Mar 20, 2018
@J0WI J0WI closed this Mar 20, 2018
@J0WI J0WI reopened this Mar 20, 2018
@J0WI J0WI closed this Mar 20, 2018
@J0WI J0WI reopened this Mar 20, 2018
@J0WI J0WI closed this Mar 20, 2018
@J0WI J0WI reopened this Mar 20, 2018
@J0WI J0WI closed this Mar 20, 2018
@J0WI J0WI reopened this Mar 20, 2018
@J0WI
Copy link
Contributor

J0WI commented Mar 20, 2018

Finally all tests passed. Thanks for fixing all those issues!

@J0WI J0WI merged commit 6905298 into EFForg:master Mar 20, 2018
@cschanaj cschanaj deleted the auto-remove-duplicated-urls branch March 20, 2018 23:47
This was referenced Mar 31, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants