Category Pages served from ElasticSearch indexes

[badev-ajay]

Hi ElasticSearch Team,

Can we be served fully category page from ElasticSearch currently product list is coming from the database?

[rbayet]

Hi @badev-ajay,

Can you clarify your question ? I'm not 100% sure I am going to answer it correctly.

At the moment, both on catalog search and on a category page - besides layered navigation filters data - Elasticsuite only pulls from ElasticSearch the product_ids of the products to render in the product list.
That list of product_ids is then applied in a SQL where condition on the collection (see \Smile\ElasticsuiteCatalog\Model\ResourceModel\Product\Fulltext\Collection::_renderFiltersBefore`).
Hence the products data is actually loaded from the DB.

Regards,

[Swahjak]

I think that is the question; can we retrieve the entire product list from Elasticsearch instead of from the database. Skipping the database could have some interesting benefits.

[romainruaud]

I agree, it could probably be a nice performance gain on frontend.

Probably quite easy to do since we have the source of any document in the search response actually.

It would require replacing/skipping the _loadAttributes of the collection to use data coming from Elasticsearch (and fallback to the legacy SQL query based on many many JOINs if an attribute is not indexed/retrieven).

BUT :

• data coming from ES can be outdated (probably not that much, but it's not real time indexing)
• there is probably something special to deal with stock status and price

Anyway, that's something we already thought about. I'd be happy to review any PR on this one.

[badev-ajay]

Hi Team,
Thanks for understanding my concerns. I think we can manage it via partial indexing.

[badev-ajay]

Hi Team,

When We can aspect this feature or please give some clue about that so i can change it myself

[BrocksiNet]

If you want to have the full catalog from ES you should maybe check this open-source stuff here:
https://github.com/lizards-and-pumpkins/catalog

[Swahjak]

@rbayet @romainruaud As insiders, could you make an estimation of the work it would take to make this happen? I would consider spending some time on this, but without an in depth knowledge of ES it's hard to make an estimation on the time this would cost. If size isn't a problem we could replace / modify the flat tables to use elasticsearch as a backend. This would also resolve a problem for us related to the flat table rows being to large. Error Code: 1118. Row size too large (> 8126). Changing some columns to TEXT or BLOB may help. In current row format, BLOB prefix of 0 bytes is stored inline.

[Swahjak]

@bjoern-flagbit That's an interesting options. But to me it seems they aim to have a 'broad' support for catalogs while I think a Magento 2 specific solution would allow for full Magento 2 capability.

[badev-ajay]

Hi Swahjak,
i have enabled product listing from backend now I am getting bellow error

1 exception(s):
Exception #0 (Exception): Notice: Array to string conversion in /var/www/html/m23/vendor/magento/module-catalog/Model/View/Asset/Image.php on line 188

Exception #0 (Exception): Notice: Array to string conversion in /var/www/html/m23/vendor/magento/module-catalog/Model/View/Asset/Image.php on line 188
#0 /var/www/html/m23/vendor/magento/module-catalog/Model/View/Asset/Image.php(188): Magento\Framework\App\ErrorHandler->handler(8, 'Array to string...', '/var/www/html/m...', 188, Array)
#1 /var/www/html/m23/vendor/magento/module-catalog/Model/View/Asset/Image.php(95): Magento\Catalog\Model\View\Asset\Image->getImageInfo()
#2 /var/www/html/m23/vendor/magento/module-catalog/Block/Product/ImageFactory.php(151): Magento\Catalog\Model\View\Asset\Image->getUrl()
#3 /var/www/html/m23/vendor/magento/module-catalog/Block/Product/ImageBuilder.php(135): Magento\Catalog\Block\Product\ImageFactory->create(Object(Magento\Catalog\Model\Product\Interceptor), 'category_page_g...', Array)
#4 /var/www/html/m23/vendor/magento/module-catalog/Block/Product/AbstractProduct.php(513): Magento\Catalog\Block\Product\ImageBuilder->create(Object(Magento\Catalog\Model\Product\Interceptor), 'category_page_g...', Array)
#5 /var/www/html/m23/vendor/magento/framework/Interception/Interceptor.php(58): Magento\Catalog\Block\Product\AbstractProduct->getImage(Object(Magento\Catalog\Model\Product\Interceptor), 'category_page_g...', Array)
#6 /var/www/html/m23/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Catalog\Block\Product\ListProduct\Interceptor->___callParent('getImage', Array)
#7 /var/www/html/m23/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Catalog\Block\Product\ListProduct\Interceptor->Magento\Framework\Interception{closure}(Object(Magento\Catalog\Model\Product\Interceptor), 'category_page_g...')
#8 /var/www/html/m23/generated/code/Magento/Catalog/Block/Product/ListProduct/Interceptor.php(468): Magento\Catalog\Block\Product\ListProduct\Interceptor->___callPlugins('getImage', Array, Array)
#9 /var/www/html/m23/vendor/magento/module-catalog/view/frontend/templates/product/list.phtml(53): Magento\Catalog\Block\Product\ListProduct\Interceptor->getImage(Object(Magento\Catalog\Model\Product\Interceptor), 'category_page_g...')
#10 /var/www/html/m23/vendor/magento/framework/View/TemplateEngine/Php.php(59): include('/var/www/html/m...')
#11 /var/www/html/m23/vendor/magento/framework/View/Element/Template.php(271): Magento\Framework\View\TemplateEngine\Php->render(Object(Magento\Catalog\Block\Product\ListProduct\Interceptor), '/var/www/html/m...', Array)
#12 /var/www/html/m23/generated/code/Magento/Catalog/Block/Product/ListProduct/Interceptor.php(557): Magento\Framework\View\Element\Template->fetchView('/var/www/html/m...')
#13 /var/www/html/m23/vendor/magento/framework/View/Element/Template.php(301): Magento\Catalog\Block\Product\ListProduct\Interceptor->fetchView('/var/www/html/m...')
#14 /var/www/html/m23/vendor/magento/framework/View/Element/AbstractBlock.php(668): Magento\Framework\View\Element\Template->_toHtml()
#15 /var/www/html/m23/generated/code/Magento/Catalog/Block/Product/ListProduct/Interceptor.php(882): Magento\Framework\View\Element\AbstractBlock->toHtml()

@bjoern-flagbit That's an interesting options. But to me it seems they aim to have a 'broad' support for catalogs while I think a Magento 2 specific solution would allow for full Magento 2 capability.

[Swahjak]

@badev-ajay This was by no means meant as a 'out of the box' working PR. We've got a very specific environment for development so I currently do not have a 'vanilla' Magento environment for testing. This is a very (very) rough sketch of what could be.

On a general note (@romainruaud) first tests (based on the Smile Debug Toolbar) do not really seem to have a performance improvement. But then again Elasticsearch was running in a single container on a local machine (so was mysql, but the second seems more suitable on a 'small' development environment).

[romainruaud]

I close this one since it's a non-issue.

But I keep in mind (and the PR is here to remind) that @Swahjak provided a POC.

Regards