Add install and runtime status warnings if MySQL utf8mb4 is not supported

[mfb]

Overview

At a future date, it would be a good idea for CiviCRM to require utf8mb4 support and migrate the schema to utf8mb4.

To work towards this goal, we should add install and runtime status warnings if MySQL utf8mb4 is not supported.

Before

Civi instances could have a setup that doesn't support utf8mb4 (i.e. versions of MySQL that require additional configuration).

After

A warning is displayed on the system status page, as well as at install time, if utf8mb4 is not supported by the stack.

Technical Details

This is incredibly redundant, as there seem to be multiple places where requirement checks are made. If it makes sense to abstract it and put the code in one place, I would gladly do so.

Comments

See also discussion @ https://lab.civicrm.org/dev/mail/issues/37 https://lab.civicrm.org/dev/core/issues/339 https://lab.civicrm.org/dev/core/issues/392

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[JoeMurray]

I'm in favour of the outcome sought by this PR.

Would it be worthwhile to see testing on a diverse set of instances before merge? I don't know the options on all of the versions of MySQL out there http://stats.civicrm.org/?tab=technology (there are a fair number of Percona installs too).

[mfb]

Would it be worthwhile to see testing on a diverse set of instances before merge?

The test query is part of Drupal 7 so it's already pretty well tested at this point. The install requirements there are MySQL >= 5.0.15, MariaDB >= 5.1.44 or Percona Server >= 5.1.70

[MegaphoneJon]

Eileen requested additional community input. I'm a +1 on this. I also imagine it's fairly rare that this will be triggered - most distros that ship a version of MySQL/etc. that Civi supports will support utf8mb4 out of the box - but that's perhaps why it IS important to catch this for configs which aren't supported.

[pfigel]

LGTM. Flagging future compatibility issues as early as possible is a great approach. 👍

[colemanw]

Merging based on community review.

[mattwire]

This check seems to cause a fatal error on a client site during login. Backtrace below:

$backTrace = #0 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Core/Error.php(232): CRM_Core_Error::backtrace("backTrace", TRUE) #1 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/PEAR.php(921): CRM_Core_Error::handle(Object(DB_Error)) #2 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/DB.php(985): PEAR_Error->__construct("DB Error: unknown error", -1, 16, (Array:2), "CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...") #3 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/PEAR.php(575): DB_Error->__construct(-1, 16, (Array:2), "CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...") #4 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/PEAR.php(223): PEAR->_raiseError(Object(DB_mysqli), NULL, -1, 16, (Array:2), "CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...", "DB_Error", TRUE) #5 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/DB/common.php(1907): PEAR->__call("raiseError", (Array:7)) #6 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/DB/mysqli.php(933): DB_common->raiseError(-1, NULL, NULL, "CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...", "1071 ** Specified key was too long; max key length is 767 bytes") #7 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/DB/mysqli.php(403): DB_mysqli->mysqliRaiseError() #8 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/DB/common.php(1216): DB_mysqli->simpleQuery("CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...") #9 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/DB/DataObject.php(2415): DB_common->query("CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...") #10 /home/exampleorg/drupal/sites/all/modules/civicrm/packages/DB/DataObject.php(1607): DB_DataObject->_query("CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...") #11 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Core/DAO.php(438): DB_DataObject->query("CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...") #12 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Core/DAO.php(1413): CRM_Core_DAO->query("CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...", TRUE) #13 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Utils/Check/Component/Env.php(944): CRM_Core_DAO::executeQuery("CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(...") #14 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Utils/Check/Component.php(54): CRM_Utils_Check_Component_Env->checkMysqlUtf8mb4() #15 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Utils/Check.php(209): CRM_Utils_Check_Component->checkAll() #16 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Utils/Check.php(93): CRM_Utils_Check::checkAll() #17 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Core/Page.php(218): CRM_Utils_Check->showPeriodicAlerts() #18 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Contact/Page/DashBoard.php(69): CRM_Core_Page->run() #19 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php(309): CRM_Contact_Page_DashBoard->run((Array:1), NULL) #20 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php(84): CRM_Core_Invoke::runItem((Array:13)) #21 /home/exampleorg/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php(52): CRM_Core_Invoke::_invoke((Array:1)) #22 /home/exampleorg/drupal/sites/all/modules/civicrm/drupal/civicrm.module(445): CRM_Core_Invoke::invoke((Array:1)) #23 /home/exampleorg/drupal/includes/menu.inc(527): civicrm_invoke() #24 /home/exampleorg/drupal/index.php(21): menu_execute_active_handler() #25 {main}

$Fatal Error Details = Array ( [callback] => Array ( [0] => CRM_Core_Error [1] => handle ) [code] => -1 [message] => DB Error: unknown error [mode] => 16 [debug_info] => CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(255))) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC ENGINE=INNODB [nativecode=1071 ** Specified key was too long; max key length is 767 bytes] [type] => DB_Error [user_info] => CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(255))) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC ENGINE=INNODB [nativecode=1071 ** Specified key was too long; max key length is 767 bytes] [to_string] => [db_error: message="DB Error: unknown error" code=-1 mode=callback callback=CRM_Core_Error::handle prefix="" info="CREATE TEMPORARY TABLE civicrm_utf8mb4_test (id VARCHAR(255), PRIMARY KEY(id(255))) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ROW_FORMAT=DYNAMIC ENGINE=INNODB [nativecode=1071 ** Specified key was too long; max key length is 767 bytes]"] )

[mlutfy]

@mattwire Can you open a gitlab issue and cross-ref here? (what version of mysql?)

[mattwire]

https://lab.civicrm.org/dev/core/issues/749

[davejenx]

This falls over on key length for a VARCHAR(255) field rather than unavailability of utf8mb4 per se. I imagine that will happen on many environments. The server where it's reported here is CentOS 6 with cPanel. I have seen a similar error in a different context when setting up a site locally on Debian 9 (Stretch) with MariaDB 10.1.37 . There I resolved it by adding this to MariaDB config:

innodb_file_per_table   = ON
innodb_file_format      = Barracuda
innodb_large_prefix     = ON

That's not default on either of the above environments so I don't share the optimism that it's fairly rare that this will be triggered.

I understand that the code is intended to catch the exception and a fix has been submitted for this at #13682, so we shouldn't see a fatal error going forward. Just wanted to offer the above as evidence about the likelihood of environments not passing the check.

[mfb]

This should issue a warning on an out-of-the-box configuration of MySQL < 5.7 and MariaDB < 10.2