Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CRM-21811: Optimize advanced search by relationship with target g… #11732

Conversation

twomice
Copy link
Contributor

@twomice twomice commented Feb 28, 2018

…roup for reciprocal relationship types

Overview

The issue https://issues.civicrm.org/jira/browse/CRM-21811 describes how a site with large numbers of "spouse of" relationships (or any other reciprocal relationship types) can hit crippling performance problems in Advanced Search with criteria in both Relationships > Relationship Type and Relationships > Target Contact(s) in Group.

There's already code in place that creates a temporary table with UNION to flatten relationships for reciprocal types, but it specifically avoids doing so when Relationships > Target Contact(s) in Group criteria is used.

This PR adds support for that optimization when Relationships > Target Contact(s) in Group is used.

Before

In certain situations (see ticket), with certain criteria, Advanced Search takes a very long time to run, either ending in a WSOD, or taking dozens of minutes to complete.

After

In the same situation, with the same criteria, the Advanced Search completes in a few seconds.

Technical Details

PR includes the addition of a 5th parameter (with appropriate default value) to CRM_Contact_BAO_Query::addGroupContactCache(). Seems reasonable to me (of course).

Comments

None.

@twomice twomice changed the title Fix CRM-21811: Optimize advanced search by relationship with target g… CRM-21811: Optimize advanced search by relationship with target g… Feb 28, 2018
@eileenmcnaughton
Copy link
Contributor

I think the technical approach seems OK - I've skimmed it. You know what I think it needs of course....

@twomice
Copy link
Contributor Author

twomice commented Mar 5, 2018

Better than ever -- now with unit tests!

@twomice
Copy link
Contributor Author

twomice commented Mar 6, 2018

@eileenmcnaughton Is there anything I can do to make this easier to merge?

@twomice
Copy link
Contributor Author

twomice commented Mar 13, 2018

Hi. Any thoughts on merging this and/or something I should do to improve it?

@twomice twomice changed the title CRM-21811: Optimize advanced search by relationship with target g… WIP: CRM-21811: Optimize advanced search by relationship with target g… Mar 14, 2018
@twomice
Copy link
Contributor Author

twomice commented Mar 14, 2018

Changed to WIP. Production usage of this code indicates a problem related to calculation of smart groups based on saved searches using criteria like those described here. Will debug, add tests, and update the PR.

@twomice
Copy link
Contributor Author

twomice commented Mar 17, 2018

FYI, repro steps on the regression:

  1. Perform an Advanced Search with these criteria:
  • Relationships > Relationship Type: any non-reciprocally named relationship type, such as "child of"
  • Relationships > Target Contact(s) in Group: [any one or more groups]
  1. Observe the fatal error:
Array
(
    [callback] => Array
        (
            [0] => CRM_Core_Error
            [1] => handle
        )

    [code] => -19
    [message] => DB Error: no such field
    [mode] => 16
    [debug_info] => SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name  FROM civicrm_contact contact_a use index(index_sort_name) LEFT JOIN civicrm_relationship ON (civicrm_relationship.contact_id_b = contact_a.id ) LEFT JOIN civicrm_contact contact_b ON (civicrm_relationship.contact_id_a = contact_b.id )  LEFT JOIN civicrm_group_contact civicrm_relationship_group_contact ON civicrm_relationship_group_contact.contact_id = civicrm_relationship.contact_id_alt AND civicrm_relationship_group_contact.status = 'Added'   LEFT JOIN civicrm_group_contact_cache civicrm_relationship_group_contact_cache ON civicrm_relationship.contact_id_alt = civicrm_relationship_group_contact_cache.contact_id   WHERE  ( ( ( civicrm_relationship_group_contact.group_id IN  (582) )  OR ( civicrm_relationship_group_contact_cache.group_id IN ("582") ) ) AND (
civicrm_relationship.is_active = 1 AND
( civicrm_relationship.end_date IS NULL OR civicrm_relationship.end_date >= 20180318 ) AND
( civicrm_relationship.start_date IS NULL OR civicrm_relationship.start_date <= 20180318 )
) AND (contact_b.is_deleted = 0) AND civicrm_relationship.relationship_type_id = 1 )  AND (contact_a.is_deleted = 0)    ORDER BY UPPER(LEFT(contact_a.sort_name, 1)) asc  [nativecode=1054 ** Unknown column 'civicrm_relationship.contact_id_alt' in 'on clause']
    [type] => DB_Error
    [user_info] => SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name  FROM civicrm_contact contact_a use index(index_sort_name) LEFT JOIN civicrm_relationship ON (civicrm_relationship.contact_id_b = contact_a.id ) LEFT JOIN civicrm_contact contact_b ON (civicrm_relationship.contact_id_a = contact_b.id )  LEFT JOIN civicrm_group_contact civicrm_relationship_group_contact ON civicrm_relationship_group_contact.contact_id = civicrm_relationship.contact_id_alt AND civicrm_relationship_group_contact.status = 'Added'   LEFT JOIN civicrm_group_contact_cache civicrm_relationship_group_contact_cache ON civicrm_relationship.contact_id_alt = civicrm_relationship_group_contact_cache.contact_id   WHERE  ( ( ( civicrm_relationship_group_contact.group_id IN  (582) )  OR ( civicrm_relationship_group_contact_cache.group_id IN ("582") ) ) AND (
civicrm_relationship.is_active = 1 AND
( civicrm_relationship.end_date IS NULL OR civicrm_relationship.end_date >= 20180318 ) AND
( civicrm_relationship.start_date IS NULL OR civicrm_relationship.start_date <= 20180318 )
) AND (contact_b.is_deleted = 0) AND civicrm_relationship.relationship_type_id = 1 )  AND (contact_a.is_deleted = 0)    ORDER BY UPPER(LEFT(contact_a.sort_name, 1)) asc  [nativecode=1054 ** Unknown column 'civicrm_relationship.contact_id_alt' in 'on clause']
    [to_string] => [db_error: message="DB Error: no such field" code=-19 mode=callback callback=CRM_Core_Error::handle prefix="" info="SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name  FROM civicrm_contact contact_a use index(index_sort_name) LEFT JOIN civicrm_relationship ON (civicrm_relationship.contact_id_b = contact_a.id ) LEFT JOIN civicrm_contact contact_b ON (civicrm_relationship.contact_id_a = contact_b.id )  LEFT JOIN civicrm_group_contact civicrm_relationship_group_contact ON civicrm_relationship_group_contact.contact_id = civicrm_relationship.contact_id_alt AND civicrm_relationship_group_contact.status = 'Added'   LEFT JOIN civicrm_group_contact_cache civicrm_relationship_group_contact_cache ON civicrm_relationship.contact_id_alt = civicrm_relationship_group_contact_cache.contact_id   WHERE  ( ( ( civicrm_relationship_group_contact.group_id IN  (582) )  OR ( civicrm_relationship_group_contact_cache.group_id IN ("582") ) ) AND (
civicrm_relationship.is_active = 1 AND
( civicrm_relationship.end_date IS NULL OR civicrm_relationship.end_date >= 20180318 ) AND
( civicrm_relationship.start_date IS NULL OR civicrm_relationship.start_date <= 20180318 )
) AND (contact_b.is_deleted = 0) AND civicrm_relationship.relationship_type_id = 1 )  AND (contact_a.is_deleted = 0)    ORDER BY UPPER(LEFT(contact_a.sort_name, 1)) asc  [nativecode=1054 ** Unknown column 'civicrm_relationship.contact_id_alt' in 'on clause']"]
)

So in short: the PR currently fixes performance problems for Advanced Search on a group for reciprocally named relationship types, but breaks Advanced Search on a group for non-reciprocally named relationship types.

Working on a solution...

@twomice twomice force-pushed the CRM-21811_optimize_search_reciprocal_relationship_group branch from f460398 to 14a01e7 Compare March 17, 2018 16:11
@twomice
Copy link
Contributor Author

twomice commented Mar 19, 2018

Jenkins, retest this please.

@twomice
Copy link
Contributor Author

twomice commented Mar 19, 2018

The only failing test is "CRM_Utils_versionCheckTest::testGetSiteStats" which (per https://chat.civicrm.org/civicrm/pl/qkoexom5obd77rezopzhmgje1y) seems to be failing for all PRs.

So, I'm removing "WIP" and looking for feedback and/or movement toward merge.

@twomice twomice changed the title WIP: CRM-21811: Optimize advanced search by relationship with target g… CRM-21811: Optimize advanced search by relationship with target g… Mar 19, 2018
@twomice twomice force-pushed the CRM-21811_optimize_search_reciprocal_relationship_group branch from 14a01e7 to dfede84 Compare March 19, 2018 21:19
@twomice
Copy link
Contributor Author

twomice commented Mar 19, 2018

Forgot to mention: This PR no longer creates the regression mentioned in my comment above (#11732 (comment)). It also contains a new test for that regression.

@seamuslee001 seamuslee001 self-assigned this Mar 19, 2018
@seamuslee001
Copy link
Contributor

Assigning this to myself, i'll try and get this looked over hopefully today.

@twomice
Copy link
Contributor Author

twomice commented Mar 19, 2018

Thanks, Seamus!

@seamuslee001
Copy link
Contributor

@twomice Allen the Test failure here looks relevant

@twomice
Copy link
Contributor Author

twomice commented Mar 19, 2018

Yep, thanks. Will fix and ping you.

@twomice twomice force-pushed the CRM-21811_optimize_search_reciprocal_relationship_group branch from dfede84 to 3f1bbf4 Compare March 20, 2018 00:56
@twomice twomice force-pushed the CRM-21811_optimize_search_reciprocal_relationship_group branch from 3f1bbf4 to 72a2eea Compare March 20, 2018 01:38
@twomice
Copy link
Contributor Author

twomice commented Mar 20, 2018

@seamuslee001 "All checks have passed." Thanks for reviewing when you can.

@seamuslee001
Copy link
Contributor

(CiviCRM Review Template WORD-1.1)

  • (r-explain) Pass
  • (r-test) Pass
  • (r-code) Pass
  • (r-doc) Pass
  • (r-maint) Pass
  • (r-run) Pass was able to perform relationship searches and targets on groups on both reciprocal and non reciprocal relationships. Also contains a test
  • (r-user) Pass
  • (r-tech) Pass

@seamuslee001
Copy link
Contributor

Merging

@seamuslee001 seamuslee001 merged commit 0e4e454 into civicrm:master May 22, 2018
@twomice
Copy link
Contributor Author

twomice commented May 22, 2018

Thanks @seamuslee001 !

@eileenmcnaughton
Copy link
Contributor

@seamuslee001 @twomice I updated the fix version in the JIRA ticket & closed it.

As a matter of process it is the job of EVERYONE to close the JIRA or gitlab ticket - because it is quick to do & often forgotten it's best if the merger, the reviewer & the submitter all feel like it is on them

@twomice
Copy link
Contributor Author

twomice commented May 22, 2018

Good point, @eileenmcnaughton. I'll try to give that my attention in the future. Thanks!

@sandrajvillagenetwork
Copy link

Hi there - This issue has resurfaced since we upgraded our client sites to 5.3.0. Advanced Search - relationship type 'spouse of' - target contacts in group ' is resulting in a lot of hang time and then the error. This is the error code I got when I turned on the backtrace/debugging:

backTrace

#0 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/Error.php(190): CRM_Core_Error::backtrace()
#1 internal function: CRM_Core_Error::handle(Object(DB_Error))
#2 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/PEAR.php(921): call_user_func((Array:2), Object(DB_Error))
#3 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/DB.php(985): PEAR_Error->__construct("DB Error: unknown error", -1, 16, (Array:2), "SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...")
#4 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/PEAR.php(575): DB_Error->__construct(-1, 16, (Array:2), "SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...")
#5 internal function: PEAR->_raiseError(Object(DB_mysqli), NULL, -1, NULL, NULL, "SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...", "DB_Error", TRUE)
#6 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/PEAR.php(224): call_user_func_array((Array:2), (Array:8))
#7 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/DB/common.php(1905): PEAR->__call("raiseError", (Array:7))
#8 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/DB/common.php(1905): PEAR->raiseError(NULL, -1, NULL, NULL, "SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...", "DB_Error", TRUE)
#9 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/DB/mysqli.php(933): DB_common->raiseError(-1, NULL, NULL, NULL, "1969 ** Query execution was interrupted (max_statement_time exceeded)")
#10 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/DB/mysqli.php(403): DB_mysqli->mysqliRaiseError()
#11 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/DB/common.php(1216): DB_mysqli->simpleQuery("SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...")
#12 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/DB/DataObject.php(2415): DB_common->query("SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...")
#13 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/DB/DataObject.php(1607): DB_DataObject->_query("SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...")
#14 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/DAO.php(415): DB_DataObject->query("SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...")
#15 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/DAO.php(1371): CRM_Core_DAO->query("SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...", TRUE)
#16 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Contact/BAO/Query.php(4952): CRM_Core_DAO::executeQuery("SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name FROM civicr...")
#17 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Contact/Selector.php(1213): CRM_Contact_BAO_Query->searchQuery(NULL, NULL, NULL, FALSE, FALSE, TRUE)
#18 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Utils/PagerAToZ.php(108): CRM_Contact_Selector->alphabetQuery()
#19 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Utils/PagerAToZ.php(136): CRM_Utils_PagerAToZ::getDynamicCharacters(Object(CRM_Contact_Selector), FALSE)
#20 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Utils/PagerAToZ.php(52): CRM_Utils_PagerAToZ::createLinks(Object(CRM_Contact_Selector), NULL, FALSE)
#21 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Contact/Form/Search.php(869): CRM_Utils_PagerAToZ::getAToZBar(Object(CRM_Contact_Selector), NULL)
#22 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Contact/Form/Search/Advanced.php(319): CRM_Contact_Form_Search->postProcess()
#23 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/Form.php(466): CRM_Contact_Form_Search_Advanced->postProcess()
#24 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/QuickForm/Action/Refresh.php(75): CRM_Core_Form->mainProcess()
#25 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/HTML/QuickForm/Controller.php(203): CRM_Core_QuickForm_Action_Refresh->perform(Object(CRM_Contact_Form_Search_Advanced), "refresh")
#26 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/packages/HTML/QuickForm/Page.php(103): HTML_QuickForm_Controller->handle(Object(CRM_Contact_Form_Search_Advanced), "refresh")
#27 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/Controller.php(351): HTML_QuickForm_Page->handle("refresh")
#28 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/Invoke.php(309): CRM_Core_Controller->run((Array:4), (Array:0))
#29 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/Invoke.php(84): CRM_Core_Invoke::runItem((Array:13))
#30 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/CRM/Core/Invoke.php(52): CRM_Core_Invoke::_invoke((Array:4))
#31 /var/aegir/platforms/civicrm-4.7/sites/all/modules/civicrm/drupal/civicrm.module(445): CRM_Core_Invoke::invoke((Array:4))
#32 internal function: civicrm_invoke("contact", "search", "advanced")
#33 /var/aegir/platforms/civicrm-4.7/includes/menu.inc(527): call_user_func_array("civicrm_invoke", (Array:3))
#34 /var/aegir/platforms/civicrm-4.7/index.php(21): menu_execute_active_handler()
#35 {main}
Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
DB Error: unknown error
Error Details
Database Error Code: Query execution was interrupted (max_statement_time exceeded), 1969
Additional Details:
Array
(
[callback] => Array
(
[0] => CRM_Core_Error
[1] => handle
)

[code] => -1
[message] => DB Error: unknown error
[mode] => 16
[debug_info] => SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name  FROM civicrm_contact contact_a use index(index_sort_name) INNER JOIN civicrm_rel_temp_f6b31edfb4d7736043171d11112f29df civicrm_relationship ON civicrm_relationship.contact_id = contact_a.id  LEFT JOIN civicrm_group_contact civicrm_relationship_group_contact ON civicrm_relationship_group_contact.contact_id = civicrm_relationship.contact_id_alt AND civicrm_relationship_group_contact.status = 'Added'   LEFT JOIN civicrm_group_contact_cache civicrm_relationship_group_contact_cache ON civicrm_relationship.contact_id_alt = civicrm_relationship_group_contact_cache.contact_id   WHERE  ( ( ( civicrm_relationship_group_contact.group_id IN  (1418) )  OR ( civicrm_relationship_group_contact_cache.group_id IN ("1418") ) ) )  AND (contact_a.is_deleted = 0)    ORDER BY UPPER(LEFT(contact_a.sort_name, 1)) asc  [nativecode=1969 ** Query execution was interrupted (max_statement_time exceeded)]
[type] => DB_Error
[user_info] => SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name  FROM civicrm_contact contact_a use index(index_sort_name) INNER JOIN civicrm_rel_temp_f6b31edfb4d7736043171d11112f29df civicrm_relationship ON civicrm_relationship.contact_id = contact_a.id  LEFT JOIN civicrm_group_contact civicrm_relationship_group_contact ON civicrm_relationship_group_contact.contact_id = civicrm_relationship.contact_id_alt AND civicrm_relationship_group_contact.status = 'Added'   LEFT JOIN civicrm_group_contact_cache civicrm_relationship_group_contact_cache ON civicrm_relationship.contact_id_alt = civicrm_relationship_group_contact_cache.contact_id   WHERE  ( ( ( civicrm_relationship_group_contact.group_id IN  (1418) )  OR ( civicrm_relationship_group_contact_cache.group_id IN ("1418") ) ) )  AND (contact_a.is_deleted = 0)    ORDER BY UPPER(LEFT(contact_a.sort_name, 1)) asc  [nativecode=1969 ** Query execution was interrupted (max_statement_time exceeded)]
[to_string] => [db_error: message="DB Error: unknown error" code=-1 mode=callback callback=CRM_Core_Error::handle prefix="" info="SELECT DISTINCT UPPER(LEFT(contact_a.sort_name, 1)) as sort_name  FROM civicrm_contact contact_a use index(index_sort_name) INNER JOIN civicrm_rel_temp_f6b31edfb4d7736043171d11112f29df civicrm_relationship ON civicrm_relationship.contact_id = contact_a.id  LEFT JOIN civicrm_group_contact civicrm_relationship_group_contact ON civicrm_relationship_group_contact.contact_id = civicrm_relationship.contact_id_alt AND civicrm_relationship_group_contact.status = 'Added'   LEFT JOIN civicrm_group_contact_cache civicrm_relationship_group_contact_cache ON civicrm_relationship.contact_id_alt = civicrm_relationship_group_contact_cache.contact_id   WHERE  ( ( ( civicrm_relationship_group_contact.group_id IN  (1418) )  OR ( civicrm_relationship_group_contact_cache.group_id IN ("1418") ) ) )  AND (contact_a.is_deleted = 0)    ORDER BY UPPER(LEFT(contact_a.sort_name, 1)) asc  [nativecode=1969 ** Query execution was interrupted (max_statement_time exceeded)]"]

)
Return to home page.

@twomice
Copy link
Contributor Author

twomice commented Aug 28, 2018

The bad behavior mentioned by @sandrajvillagenetwork does look similar to this issue, but internally it's caused by something else, described here: https://lab.civicrm.org/dev/core/issues/367

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants