[4.0] Removing artificial sharding from com_finder #20185

Hackwar · 2018-04-17T09:51:25Z

Finder has an artificial sharding, which splits up the links between terms and content items onto 16 seperate tables. This PR removes that sharding. It makes the code extremely complex and bloats it unnecessarily and it also slows down the whole finder system. Indexing the original sample data of Joomla 4.0 with the current finder indexer takes about 2:40 minutes on my system. With these changes, we are already down to 1:30 minute.

This PR also fixes a few other bugs that prevented com_finder from working in Joomla 4.0 right now and I standardised all the language fields to be 7 characters long. While I plan to provide more PRs with further improvements to make com_finder more usable, this PR is in itself complete and can be merged like this.

How to test

Install 4.0-dev and go to the backend of Smart Search. Enable the Smart Search Plugin and start the indexing process. Notice that it takes a long time (preferably measure the time with a stopwatch). Go to the frontend and open the Smart Search menu item. See that it throws errors and doesn't work right now.
Apply this PR and clear the index in the backend. Start the re-indexing and notice that it finishes quicker. Go to the frontend and see that that menu item no longer throws errors and that all features work like expected.

When typing in a phrase and then pressing enter, sending the request to the server, a message "error" is displayed before the page is reloaded both before and after applying the patch. I'm still looking for the source of that issue.

I want to thank cloudaccess.net for supporting these changes. Without their generous help, this wouldn't be possible.

brianteeman · 2018-04-17T09:55:06Z

As there is administrator/components/com_admin/sql/updates/mysql/4.0.0-2018-04-14.sql
I assume that there should also be an equivalent postgres file?

Hackwar · 2018-04-17T10:04:10Z

I would need help with the postgres stuff. I don't have postgres on my system and would need someone to provide such a file and test that part.

Hackwar · 2018-04-17T18:32:41Z

@alikon Could you help me with adding the right migration script for postgres?

alikon · 2018-04-18T05:51:12Z

yes i'll do it

Hackwar · 2018-04-18T06:28:44Z

Thank you!

infograf768 · 2018-04-18T10:05:27Z

Will test. Please let me know, apart from the indexing speed (and obvious code corrections), what I can expect, as a multilingual user, from this PR?

Hackwar · 2018-04-18T10:26:45Z

At this moment, there is no change in terms of multilingual users. In order to keep these changes testable, I'm restricting my changes to one thing at a time. So this only removes the sharding and gets it all working again in 4.0. The language changes are coming in the next few steps.

alikon · 2018-04-21T11:13:44Z

made a pr Hackwar#22
unfortunately without this one #19707 merged
unable to test/work

the update sql for postgresql

Hackwar · 2018-04-21T11:41:06Z

Merged

brianteeman · 2018-05-09T10:07:21Z

I was unable to replicate the error that you mentioned before applying this PR

After applying the PR the indexer did not work at all

brianteeman · 2018-05-09T10:07:33Z

I have tested this item 🔴 unsuccessfully on 7716cd6

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/20185.}

Hackwar · 2018-05-11T21:16:35Z

The current 4.0-dev branch fails in displaying the frontend when opening the smart search component and having debug mode enabled with the following error message:
1055 'j4search2.t.parent_id' isn't in GROUP BY
This has been fixed in this PR.

I have updated the branch with the latest changes from 4.0-dev and checked it again. It does work. Could it be that you did not apply the database changes, that this introduces? I would be happy if you could test this again.

Hackwar · 2018-05-20T19:58:35Z

@brianteeman Did you see my last message? Could you check your test again, especially with the database changes?

brianteeman · 2018-05-20T20:04:06Z

Sorry with JAB and JDFR it slipped my radar - thanks for the reminder I will put it on the todo for tomorrow

Hackwar · 2018-05-21T08:19:24Z

Just to list all current PRs for Finder/Smart Search:

[4.0] Removing artificial sharding from com_finder #20185 [4.0] Removing artificial sharding from com_finder
[4.0] Make the tuple length of com_finder configurable #20384 [4.0] Removing terms tuples from com_finder
[4.0] Using language specific tokeniser and stemmer for com_finder #20391 [4.0] Using language specific tokeniser and stemmer for com_finder

brianteeman · 2018-05-23T09:37:01Z

@Hackwar no change from my previous report

Hackwar · 2018-05-23T10:19:03Z

Can you see what the ajax call from the indexer returns as error?

laoneo · 2018-06-04T04:17:05Z

Yes please one pr per task.

laoneo · 2018-06-04T04:21:18Z

Thanks Chris for the explanation.

…4finder_sharding # Conflicts: # components/com_finder/Model/SearchModel.php

Hackwar · 2018-06-05T13:34:43Z

I've reverted the changes of the router and cleaned up the commits.

laoneo · 2018-06-05T13:59:20Z

components/com_finder/helpers/html/query.php

@@ -9,6 +9,8 @@

 defined('_JEXEC') or die;

+JLoader::register('FinderHelperLanguage', JPATH_ADMINISTRATOR . '/components/com_finder/helpers/language.php');


Is this still needed?

Yes, otherwise you get a fatal error when using the advanced filters. And please don't ask me to put all these single line fixes to make com_finder run in 4.0 into separate PRs....

laoneo · 2018-06-05T14:00:54Z

Would be good if we can get another round of testing and then I think it is good to merge.

Hackwar · 2018-06-05T14:13:00Z

We have 2 successfull tests. Why do we need another round of tests now?

carlitorweb · 2018-06-05T14:20:32Z

I have tested this item ✅ successfully on c80b975

Tested again. Only the route change was removed from this PR, so I think will be same. Anyway, I tested again, and work as before.

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/20185.}

Quy · 2018-06-05T16:50:26Z

administrator/components/com_admin/sql/updates/mysql/4.0.0-2018-04-14.sql

+DROP TABLE #__finder_links_termsd;
+DROP TABLE #__finder_links_termse;
+DROP TABLE #__finder_links_termsf;
+


Changed CHANGE COLUMN to CHANGE, dropped AFTER ... and be on same line to make it work.

ALTER TABLE `#__finder_terms` CHANGE `language` `language` CHAR(7) NOT NULL DEFAULT ''; ALTER TABLE `#__finder_terms_common` CHANGE `language` `language` CHAR(7) NOT NULL DEFAULT ''; ALTER TABLE `#__finder_tokens` CHANGE `language` `language` CHAR(7) NOT NULL DEFAULT ''; ALTER TABLE `#__finder_tokens_aggregate` DROP COLUMN `map_suffix`; ALTER TABLE `#__finder_tokens_aggregate` CHANGE `language` `language` CHAR(7) NOT NULL DEFAULT ''; ALTER TABLE `#__finder_links` CHANGE `language` `language` CHAR(7) NOT NULL DEFAULT '';

Quy · 2018-06-05T16:51:50Z

administrator/components/com_admin/sql/updates/mysql/4.0.0-2018-04-14.sql

+COLLATE='utf8mb4_general_ci'
+ENGINE=InnoDB
+;
+DROP TABLE #__finder_links_terms0;


Tables are not dropped and I don't know why.

I did this operation manually, and work ok

Yes, but not via Extensions > Manage > Database > Fix

Iirc you can't do a drop that way

laoneo · 2018-06-07T09:59:41Z

@Hackwar can you please add the suggested changes to make the DB fixing work and then we are good to go here.

brianteeman · 2018-06-07T10:33:34Z

there is no reason to make database>fix work
That is a misuse of the functionality

laoneo · 2018-06-07T10:37:51Z

Ok, misinterpreted that comment. What about the suggestion from @Quy?

Hackwar · 2018-06-07T10:55:23Z

I'm hesitant to change the SQL again, mainly because I don't see how what is supposed to work how and more importantly because there are quite a few additional changes on the way for the finder tables. At one point we will have to make one PR that combines all these changes into one SQL file. Can we tackle that issue then? (There are 12 open PRs for finder from me, 4 of which have DB changes)

laoneo · 2018-06-07T11:17:58Z

Does the upgrade script work without the recommendations from @Quy?

Quy · 2018-06-07T15:52:57Z

With fresh install, it works.
With upgrade/patch tester, it stalls at Starting Indexer.

mbabker · 2018-06-07T15:58:49Z

This has database schema changes. Patch tester can't deal with database schema changes (you can apply the patch and run Database > Fix but you can't reverse the schema changes if you do that). The component, as useful as it can be, is not a definitive way of identifying issues with patches.

Hackwar · 2018-06-07T21:04:38Z

Just to clarify @chrisdavenport s comment here: With the sharding removed, he had a performance improvement of 44%, not a performance reduction.

Hackwar · 2018-06-12T06:58:45Z

Thank you!

wilsonge · 2018-06-12T08:43:40Z

Thankyou for being so patient with this! Sorry it took so long to deal with!

* Decoupling highlighter plugin from com_finder * Codestyle * Codestyle * Version comment * Fixing bug due to #20185 * Using url for display and route for route-URL * Cleaning up finder plugins and adding cleanURL attribute in model * Making OutputFilter::stringJSSafe() utf8 aware

Removing artificial sharding from com_finder

d437f07

joomla-cms-bot added the PR-4.0-dev label Apr 17, 2018

Fixing codestyle

8d12204

Merge branch '4.0-dev' into j4finder_sharding

ad345a2

Hackwar mentioned this pull request Apr 17, 2018

[4.0] Refactoring com_search & com_finder #12664

Closed

6 tasks

the update sql for postgresql

3763187

alikon mentioned this pull request Apr 21, 2018

the update sql for postgresql Hackwar/joomla-cms#22

Merged

Merge pull request #22 from alikon/patch-104

7dd2bea

the update sql for postgresql

Hackwar requested a review from mbabker as a code owner May 11, 2018 20:57

Hackwar mentioned this pull request May 13, 2018

[4.0] Make the tuple length of com_finder configurable #20384

Merged

This was referenced May 21, 2018

[4.0] Using language specific tokeniser and stemmer for com_finder #20391

Merged

Add "whole word" search option to search's #17751

Closed

Hackwar mentioned this pull request May 23, 2018

[4.0] Fixing Suggestions in Finder/Smart Search #20523

Merged

Hackwar added 2 commits June 5, 2018 15:28

Reverting changes to com_finder router

455fc19

Merge branch '4.0-dev' of https://github.com/joomla/joomla-cms into j…

c80b975

…4finder_sharding # Conflicts: # components/com_finder/Model/SearchModel.php

Hackwar force-pushed the j4finder_sharding branch from d9cb32f to c80b975 Compare June 5, 2018 13:30

laoneo reviewed Jun 5, 2018

View reviewed changes

Quy reviewed Jun 5, 2018

View reviewed changes

Hackwar mentioned this pull request Jun 7, 2018

Add new Terms view #13377

Closed

carlitorweb mentioned this pull request Jun 8, 2018

[4.0] Refactor component router of com_finder #20668

Merged

wilsonge merged commit 5613476 into joomla:4.0-dev Jun 11, 2018

joomla-cms-bot removed the RTC This Pull Request is Ready To Commit label Jun 11, 2018

wilsonge added this to the Joomla 4.0 milestone Jun 11, 2018

Hackwar deleted the j4finder_sharding branch June 12, 2018 06:58

Hackwar added a commit to Hackwar/joomla-cms that referenced this pull request Jun 18, 2018

Fixing bug due to joomla#20185

a52bd58

		@@ -9,6 +9,8 @@

		defined('_JEXEC') or die;

		JLoader::register('FinderHelperLanguage', JPATH_ADMINISTRATOR . '/components/com_finder/helpers/language.php');

[4.0] Removing artificial sharding from com_finder #20185

[4.0] Removing artificial sharding from com_finder #20185

Conversation

Hackwar commented Apr 17, 2018 • edited by joomla-cms-bot Loading

How to test

brianteeman commented Apr 17, 2018

Hackwar commented Apr 17, 2018

Hackwar commented Apr 17, 2018

alikon commented Apr 18, 2018

Hackwar commented Apr 18, 2018

infograf768 commented Apr 18, 2018

Hackwar commented Apr 18, 2018

alikon commented Apr 21, 2018

Hackwar commented Apr 21, 2018

brianteeman commented May 9, 2018

brianteeman commented May 9, 2018

Hackwar commented May 11, 2018

Hackwar commented May 20, 2018

brianteeman commented May 20, 2018

Hackwar commented May 21, 2018

brianteeman commented May 23, 2018

Hackwar commented May 23, 2018

laoneo commented Jun 4, 2018

laoneo commented Jun 4, 2018

Hackwar commented Jun 5, 2018

laoneo Jun 5, 2018

Choose a reason for hiding this comment

Hackwar Jun 5, 2018

Choose a reason for hiding this comment

laoneo commented Jun 5, 2018

Hackwar commented Jun 5, 2018

carlitorweb commented Jun 5, 2018

Quy Jun 5, 2018

Choose a reason for hiding this comment

Quy Jun 5, 2018

Choose a reason for hiding this comment

carlitorweb Jun 5, 2018

Choose a reason for hiding this comment

Quy Jun 5, 2018

Choose a reason for hiding this comment

brianteeman Jun 5, 2018

Choose a reason for hiding this comment

laoneo commented Jun 7, 2018

brianteeman commented Jun 7, 2018

laoneo commented Jun 7, 2018

Hackwar commented Jun 7, 2018

laoneo commented Jun 7, 2018

Quy commented Jun 7, 2018

mbabker commented Jun 7, 2018

Hackwar commented Jun 7, 2018

Hackwar commented Jun 12, 2018

wilsonge commented Jun 12, 2018

Hackwar commented Apr 17, 2018 •

edited by joomla-cms-bot

Loading