Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to bans #574

Merged
merged 23 commits into from
Nov 28, 2020
Merged

Improvements to bans #574

merged 23 commits into from
Nov 28, 2020

Conversation

stwalkerster
Copy link
Member

@stwalkerster stwalkerster commented Jun 27, 2020

  • Bans can now target a combination of attributes, such as IP address and email address
  • Useragent bans now supported
  • Username, email, and UA bans are regex-aware
  • IP bans are now CIDR aware
  • Ban targets are now restricted visibility - non-admins can't see email addresses, and non-checkusers can't see user agents
  • Reason visibility added on a per-ban basis - restrict visibility of a ban's reason to admins or CUs.
  • Bans now cannot be unset unless you can see the entirety of the ban
  • Ban targets no longer appear in log entries. Log entries are linked to view the specific ban instead
  • Bans can now auto-defer a request to a different queue, automatically drop the request, or do nothing, in addition to blocking the request from being submitted
  • Adds a checkuser-level request comment visibility check.

On the back-end:

  • The ban table has grown substantially wider; each ban how has a column for each type, as well as new columns for the new functionality
  • A new netmask table was added, containing CIDR prefix to subnet mask conversions, for use in address-in-range queries. This table will never be changed, and is intended to be entirely read-only.
  • Bump MariaDB version requirement to 10.3.22 due to use of CHECK constraints. This also allows us window functions and common table expressions. I expect versions as low as 10.2.1 will work for now, but if we're upgrading the prod DB, we should be forcing usage of that version in the dev environments too.

* Split ban table into multiple columns for different ban types
* Add a new netmask table for doing CIDR calculations more easily in the
  database
* Add a bunch of additional fields to ban table for future use
* Change how bans are processed on request
* Bump MariaDB version requirement to 10.2.1 due to use of CHECK
  constraints. This also allows us window functions and common table
  expressions.
* Allow setting multiple parameters at once on a ban
* New template to describe ban targets
* Fix duplicate ban checking
* Fix IP range checking
* Move "can unban" checks to BanHelper; ensures that a user can fully
  see a ban before unbanning it. Note that you still need the unban user
  right to remove bans.
* Don't include ban target in log entries
* Add links to view specific ban from log
* New field on new ban form
* New column on ban list
* Actions:
  - "Defer" sends the request to a specific queue automatically
  - "Drop" drops the request automatically and silently to the user
  - "Block" stops the request from being submitted
  - "Do nothing" does, well, nothing.
@codecov-commenter
Copy link

codecov-commenter commented Jun 27, 2020

Codecov Report

Merging #574 into master will decrease coverage by 0.64%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##             master    #574      +/-   ##
===========================================
- Coverage      5.90%   5.25%   -0.65%     
- Complexity     2247    2318      +71     
===========================================
  Files           173     173              
  Lines         11134   11471     +337     
===========================================
- Hits            657     603      -54     
- Misses        10477   10868     +391     
Impacted Files Coverage Δ Complexity Δ
includes/DataObjects/Ban.php 0.00% <0.00%> (ø) 49.00 <32.00> (+23.00)
includes/DataObjects/Comment.php 0.00% <0.00%> (ø) 3.00 <3.00> (-14.00)
includes/Helpers/BanHelper.php 0.00% <0.00%> (ø) 33.00 <33.00> (+29.00)
includes/Helpers/IrcNotificationHelper.php 0.00% <0.00%> (ø) 35.00 <1.00> (ø)
includes/Helpers/LogHelper.php 0.00% <0.00%> (ø) 46.00 <0.00> (ø)
includes/Pages/PageBan.php 0.00% <0.00%> (ø) 67.00 <34.00> (+25.00)
includes/Pages/PageEditComment.php 0.00% <0.00%> (ø) 12.00 <0.00> (+1.00)
includes/Pages/Request/PageRequestAccount.php 0.00% <0.00%> (ø) 18.00 <2.00> (+2.00)
includes/Pages/RequestAction/PageComment.php 0.00% <0.00%> (ø) 9.00 <0.00> (+2.00)
includes/Router/RequestRouter.php 98.00% <ø> (ø) 15.00 <0.00> (ø)
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5138436...9f07e0c. Read the comment docs.

@stwalkerster stwalkerster marked this pull request as ready for review June 27, 2020 23:50
@stwalkerster stwalkerster requested a review from dqwiki June 27, 2020 23:50
@stwalkerster stwalkerster added this to the H2 2020 milestone Jun 27, 2020
Base automatically changed from bugsquish to master July 16, 2020 22:02
@stwalkerster stwalkerster mentioned this pull request Aug 6, 2020
18 tasks
@stwalkerster
Copy link
Member Author

I've updated the database instances on wmflabs to MariaDB 10.3 in preparation for this.

Conflicts: includes/Pages/PageEditComment.php
@stwalkerster stwalkerster added the schema changes This task requires database schema changes label Aug 15, 2020
@stwalkerster
Copy link
Member Author

I've also just cloned the RC environment to https://accounts-dev.wmflabs.org/bans/internal.php if you want to test this without running your own test environment


return $errorList;
$bans = $this->banHelper->getBans($request);
Copy link
Member

@dqwiki dqwiki Nov 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wishing to verify that this goes through only the active bans without dumpster diving the code. Resolve if it does. Actually i'm not sure what this code is doing. Could I get a clarify here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BanHelper gets a list of bans which apply to supplied Request object.

The getBans(Request) call is a caching wrapper around getBansForRequestFromDatabase(Request), which in turn runs a database query to find any matching bans. For most attributes, such as names, this is a simple coalesce(:name rlike name, true) - in other words, match the requested name to a regex stored in the ban table, and if the result is null, return true. IP ranges are a little more complex, as it involves a join to the new netmask table to check ranges, but it's essentially doing the same thing. We also filter where active = 1 and duration isn't in the past. See L157 of BanHelper for the full SQL query.

We then take that list of applicable bans, and loop through them, deciding what to do based on the type of ban. If it's a "do nothing ban", we do nothing. If it's a "defer" ban, we defer it to the relevant queue. If it's a drop ban, we do the relevant drop logic with relevant log entries so it is traceable that a request was created and silently dropped according to the user. Note that "block"-type bans are handled as a pre-save validation, not as a post-save validation, so we don't need to worry about them at this stage.

Copy link
Member

@dqwiki dqwiki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Is templates/bans/main.tpl supposed to mirror almost showbans.tpl?
  • Functionality of comment saving on requests is counter intuitive giving the potential need to edit comments. You must select the special save option after typing your comment.
  • Bans that drop or defer should be limited to /20s and /48s. I know I originally said /16s, but big ranges should not be automatically banned as we are a tool of last resort for making an account. I don't see where a drop or ban would be beneficial beyond this size.

TODO: Add 2 issues:

  • One requesting Ban method to be modifiable. (Priority: Low)
  • A modification interface for the rest (Priority: Wishlist)

@codecov-io
Copy link

codecov-io commented Nov 7, 2020

Codecov Report

Merging #574 (0a25546) into master (9a78b5c) will decrease coverage by 0.60%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##             master    #574      +/-   ##
===========================================
- Coverage      5.83%   5.22%   -0.61%     
- Complexity     2253    2346      +93     
===========================================
  Files           173     173              
  Lines         11257   11534     +277     
===========================================
- Hits            657     603      -54     
- Misses        10600   10931     +331     
Impacted Files Coverage Δ Complexity Δ
includes/DataObjects/Ban.php 0.00% <0.00%> (ø) 49.00 <32.00> (+23.00)
includes/DataObjects/Comment.php 0.00% <0.00%> (ø) 3.00 <3.00> (-14.00)
includes/Helpers/BanHelper.php 0.00% <0.00%> (ø) 33.00 <33.00> (+29.00)
includes/Helpers/IrcNotificationHelper.php 0.00% <0.00%> (ø) 35.00 <1.00> (ø)
includes/Helpers/LogHelper.php 0.00% <0.00%> (ø) 46.00 <0.00> (ø)
includes/Pages/PageBan.php 0.00% <0.00%> (ø) 79.00 <46.00> (+37.00)
includes/Pages/PageEditComment.php 0.00% <0.00%> (ø) 20.00 <0.00> (+9.00)
includes/Pages/Request/PageRequestAccount.php 0.00% <0.00%> (ø) 18.00 <2.00> (+2.00)
includes/Pages/RequestAction/PageComment.php 0.00% <0.00%> (ø) 9.00 <0.00> (+2.00)
includes/Router/RequestRouter.php 98.00% <ø> (ø) 15.00 <0.00> (ø)
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a78b5c...2e796f9. Read the comment docs.

@stwalkerster stwalkerster self-assigned this Nov 8, 2020
@stwalkerster
Copy link
Member Author

stwalkerster commented Nov 9, 2020

  • Is templates/bans/main.tpl supposed to mirror almost showbans.tpl?

Hrm. I guess it was split out because it is different, but the differences can be handled with a boolean variable, so I've merged them.

  • Functionality of comment saving on requests is counter intuitive giving the potential need to edit comments. You must select the special save option after typing your comment.

Yeah, I'm not entirely sure on the UI to use here. My theory was that restricted-access comments are so rarely used that it's probably not too disruptive to have a slightly harder-to-use system than I'd ideally like. For now, I'm going to leave this as an open point and have a think about it.

  • Bans that drop or defer should be limited to /20s and /48s. I know I originally said /16s, but big ranges should not be automatically banned as we are a tool of last resort for making an account. I don't see where a drop or ban would be beneficial beyond this size, especially since IPs for bans aren't stored in a particularly human-friendly format in the database.

Drop or defer? I presume you mean drop or block? I'm quite tempted to still permit huge ranges, but hidden behind a tool-root-only flag on the off-chance that it is needed. It'll be easier to find a tool root to apply a relevant ban than it would be to make a config change to allow larger ranges temporarily. I'm also happy to remove this flag from the tool root group at some point in the future, but I'm not entirely comfortable with removing it from the code entirely again.

I'm also applying a limit for defer/nothing actions; both of these are configurable in case we want to change the limits around at some point.

TODO: Add 2 issues:

  • One requesting Ban method to be modifiable. (Priority: Low)
  • A modification interface for the rest (Priority: Wishlist)

I presume these are TODOs for you, or are you wanting me to raise the relevant issues for later consideration?

I've also found/fixed a bug with the "Ban Name" button on the view request page... 🤦

* Add limits to IP range bans
* Add tool root user right to bypass IP range limits
* Simplify duplicated request lists
* Remove unused template
* Fix bug with name bans when based on a request
Redesign visibility into a dropdown of radios
@stwalkerster
Copy link
Member Author

OK, I've redesigned the comment save thing into something which is hopefully more intuitive.

By default, this is what you see:

image

Clicking the padlock gives you this menu, from which you can choose a relevant protection type:

image

Choosing an option other than the default "locks" the padlock, and changes the button to a different colour to indicate an option has been chosen. The chosen colour matches the hue used as a highlight for the protected rows.

image
image

Does this feel like a better design to you?

@stwalkerster
Copy link
Member Author

/bans/ environment updated with latest commits

@stwalkerster stwalkerster requested a review from dqwiki November 9, 2020 01:33
@stwalkerster stwalkerster removed their assignment Nov 9, 2020
@stwalkerster stwalkerster added the actually quite difficult This task has technical intricacies which mean it needs analysis by someone familiar with the system label Nov 19, 2020
@stwalkerster
Copy link
Member Author

/bans/ environment updated with latest commits

and again.

@dqwiki
Copy link
Member

dqwiki commented Nov 27, 2020

#641 and #642 have been created from my to do.

Drop or defer? I presume you mean drop or block? I'm quite tempted to still permit huge ranges, but hidden behind a tool-root-only flag on the off-chance that it is needed. It'll be easier to find a tool root to apply a relevant ban than it would be to make a config change to allow larger ranges temporarily. I'm also happy to remove this flag from the tool root group at some point in the future, but I'm not entirely comfortable with removing it from the code entirely again.

I'm also applying a limit for defer/nothing actions; both of these are configurable in case we want to change the limits around at some point.

If I remember our conversation last time, I do mean drop and block. I'm fine with them being hid behind the root flag.

Copy link
Member

@dqwiki dqwiki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the opening of #643

@stwalkerster stwalkerster merged commit 2e796f9 into master Nov 28, 2020
@stwalkerster stwalkerster deleted the bantools branch November 28, 2020 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
actually quite difficult This task has technical intricacies which mean it needs analysis by someone familiar with the system Priority: High schema changes This task requires database schema changes
Projects
Archived in project
Status: Done
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants