Skip to content

Commit

Permalink
mysql - limit statements to 10s
Browse files Browse the repository at this point in the history
Some of these queries can take 2min+, during which time the connection
is consumed.

Fail quickly to allow more overall throughput, at the cost of a loss of
correctness in the edge cases (edits will be skipped).

Note: The queries are usually slow on accounts with lots of edits e.g.
bots, which likely are not going to be reverted anyway.

Sample timings (from botng, which uses the same queries):
name					P95(duration_ms)	AVG(duration_ms)
loader.LoadDistinctPagesCount		18,179.03402		3,005.03574
loader.LoadUserWarnsCount		572.02909		127.03932
loader.LoadPageRecentRevertCount	253.82347		66.91965
loader.LoadPageRevision			163.07911		106.22102
loader.LoadPageMetadata			104.4249		179.18445
loader.LoadUserEditCount		84.84196		11.34932
loader.LoadPageRecentEditCount		9.95227			6.23804

Example edits (also from botng, but shows the challenge)
UUID					duration_ms	Related username
6f9b2f64-bced-4a2f-af1e-5f02bc3b4218	174,505.79937	InternetArchiveBot
94141464-459e-48cf-9901-9a9161fae17c	168,008.14787	AnomieBOT
05e3e528-3f36-493f-91d6-fb7e9aaaaa93	47,951.95634	EmausBot
89f2ef9c-1ec3-425e-b667-a92476a358cf	13,345.4171	Plantdrew
68733457-aafe-426f-b91c-cd85e4c733a6	5,618.97978	Dimadick
  • Loading branch information
DamianZaremba committed Dec 12, 2024
1 parent 7c76d0a commit b02ec47
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions mysql_functions.php
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
);
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
'SELECT `rev_timestamp`, `actor_name` FROM `page`' .
' JOIN `revision` ON `rev_page` = `page_id`' .
' JOIN `actor` ON `actor_id` = `rev_actor`' .
Expand All @@ -107,6 +108,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
}
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
'SELECT COUNT(*) as count FROM `page`' .
' JOIN `revision` ON `rev_page` = `page_id`' .
' WHERE `page_namespace` = "' .
Expand All @@ -125,6 +127,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
}
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
'SELECT COUNT(*) as count FROM `page`' .
' JOIN `revision` ON `rev_page` = `page_id`' .
' JOIN `comment` ON `rev_comment_id` = `comment_id`' .
Expand All @@ -150,6 +153,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
$data['user_reg_time'] = time();
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
'SELECT COUNT(*) AS `user_editcount` FROM `revision_userindex` ' .
' JOIN `actor` ON `actor_id` = `rev_actor`' .
' WHERE `actor_name` = "' .
Expand All @@ -165,6 +169,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
} else {
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
'SELECT `user_registration` FROM `user` WHERE `user_name` = "' .
mysqli_real_escape_string($mw_mysql, $user) . '"'
);
Expand All @@ -178,6 +183,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
if (!$data['user_reg_time']) {
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
'SELECT `rev_timestamp` FROM `revision_userindex` ' .
' JOIN `actor` ON `actor_id` = `rev_actor`' .
' WHERE `actor_name` = "' .
Expand All @@ -193,6 +199,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
}
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
'SELECT `user_editcount` FROM `user` WHERE `user_name` = "' .
mysqli_real_escape_string($mw_mysql, $user) . '"'
);
Expand All @@ -206,6 +213,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
}
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
'SELECT COUNT(*) as count FROM `page`' .
' JOIN `revision` ON `rev_page` = `page_id`' .
' JOIN `comment` ON `rev_comment_id` = `comment_id`' .
Expand All @@ -223,6 +231,7 @@ function getCbData($user = '', $nsid = '', $title = '', $timestamp = '')
}
$res = mysqli_query(
$mw_mysql,
'SET STATEMENT max_statement_time=10 FOR ' .
"SELECT count(distinct rev_page) AS count FROM' .
' `revision_userindex` JOIN `actor` ON `actor_id` = `rev_actor`' .
' WHERE `actor_name` = '" . mysqli_real_escape_string($mw_mysql, $userPage) . "'"
Expand Down

0 comments on commit b02ec47

Please sign in to comment.