Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[4.2] Smart Search: Add optimisations & garbage collection #36751

Merged
merged 20 commits into from
May 27, 2022

Conversation

Hackwar
Copy link
Member

@Hackwar Hackwar commented Jan 20, 2022

Summary of Changes

There are situations where the index of Smart Search can get poluted with old data (unused terms and taxonomies, already deleted content items) and where the only solution so far has been to clear the reindex and start indexing all over again. This is not something that you really want to do in a live scenario with a large site, so this attempts some different solutions.
First of all, it runs a garbage collection, which by default deletes all entries, which aren't present in the base table of the base component. We then delete all orphaned term and taxonomy mappings as well as all then orphaned taxonomies and terms. In the end, we then execute the already previously present optimisation queries. To trigger all of this, a new dropdown button was added in the backend named "Maintenance", where you can choose between purging the index and doing the optimisation.
Note: The garbage collection has been added as a default to the Adapter base class and there could be situations where this results in a fail. I'm open for ideas here.

Testing Instructions

  1. Apply the PR
  2. Click on "Optimise" in the backend
  3. Afterwards see, that searching still works properly.

Notes

  1. Since 4.1 is in RC, this will have to go into 4.2. Since there is no 4.2-dev branch yet, I'm pointing this towards 4.1-dev for now.

@joomla-cms-bot joomla-cms-bot added Language Change This is for Translators NPM Resource Changed This Pull Request can't be tested by Patchtester PR-4.1-dev labels Jan 20, 2022
@sakiss
Copy link
Contributor

sakiss commented Jan 21, 2022

Would it be possible to add that task to the new com_scheduler ?
Removing that responsibility from the admin would be great.

@Hackwar
Copy link
Member Author

Hackwar commented Jan 21, 2022

This task is not something that you should execute regularly.

@Quy Quy added PR-4.2-dev and removed PR-4.1-dev labels Jan 24, 2022
@Hackwar Hackwar changed the base branch from 4.1-dev to 4.2-dev January 24, 2022 20:20
@nadjak77
Copy link
Contributor

I have tested this item 🔴 unsuccessfully on 7a4097b

After applieing the patch I get an error in seachrindex - indes:
There is no "table.columns" asset of a "script" type in the registry.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36751.

@Hackwar
Copy link
Member Author

Hackwar commented May 20, 2022

Can you please check if you need to do a composer install and/or npm i? This PR actually does not change anything in that regard, but maybe a previous change introduced a problem here?

@webnet-assmann
Copy link

I have tested this item ✅ successfully on 7a4097b

I've tested it with the prebuilt package and sample data. Searching was working before and after click on "Optimise".


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36751.

Copy link
Contributor

@joomdonation joomdonation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and it worked OK. Just minor things with the code and I will mark my test result.

@joomdonation
Copy link
Contributor

I have tested this item ✅ successfully on 49576c6


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36751.

@joomdonation
Copy link
Contributor

The last change was a minor change, so I restored test result of @webnet-assmann. This PR is now RTC.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36751.

@joomla-cms-bot joomla-cms-bot added the RTC This Pull Request is Ready To Commit label May 22, 2022
@Hackwar
Copy link
Member Author

Hackwar commented May 25, 2022

I'm investigating why the api tests are failing. Please don't merge this right now. I want to check that first.

@Hackwar
Copy link
Member Author

Hackwar commented May 25, 2022

I found the issue and fixed it. This could be merged now.

@fancyFranci fancyFranci added this to the Joomla 4.2.0 milestone May 27, 2022
@fancyFranci fancyFranci merged commit 5d1b7ff into joomla:4.2-dev May 27, 2022
@joomla-cms-bot joomla-cms-bot removed the RTC This Pull Request is Ready To Commit label May 27, 2022
@fancyFranci
Copy link
Contributor

Thank you Hannes!

heelc29 added a commit to heelc29/joomla that referenced this pull request Jun 5, 2022
@Hackwar Hackwar deleted the 4.2-finder-optimise branch June 14, 2022 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Language Change This is for Translators NPM Resource Changed This Pull Request can't be tested by Patchtester
Projects
None yet
Development

Successfully merging this pull request may close these issues.