All store pages are uncached after one order with one product?

[andidhouse]

Hi there,

we are facing a huge problem regarding the magento own caching.

We installed version 2.0.5 - all ok so far.
We installed sample data medium - all ok so far.

We crawled the hole site so every page is in the magento own cache.

But if we only place one order with one product ALL pages of the whole shop are uncached again?

Is this a bug or the normal magento cache behavior. In version 1.7 only the pages are uncached which belong to the order (products and category of the products).

Many thanks!

[pboisvert]

Chuck--can you get this to the team to test if they are seeing the same?

[rafaelstz]

What is your deploy mode?

[andidhouse]

you mean magento mode?

Current application mode: default.

[joebusby]

I am getting the same or very similar with each order invalidating the page cache. With all the variations (over 300) we have for a configurable product, flushing the cache means the customer will wait quite a while to get the first product to respond after an order is placed by anyone. The varnish cache can't help with this. Any ideas for a workaround or what I have wrong?

Production mode with paypal flo pro

Running 2600 products on 2.0.5 and debian 8 on an LXC container with 3gb, 6 cores and php 5.6

[joanhe]

@andidhouse @joebusby What kind of cache do you use: builtin or Varnish?

[choukalos]

Created internal ticket: MAGETWO-52605 for tracking; assigned as bug.

[joebusby]

http goes through varnish https goes to the server direct via forwarding ports 80 and 443

As required, the internal cache is enabled, but varnish is selected and configured via:
Stores/Configuration/Advanced/System/Full Page Cache with the VCL file exported and plugged into varnish. It works fine except the problematic breaks in the "Page Cache" validation, and the occasional problem with paypal "transparent redirect" tech. As I recall, the problem exists with or without varnish, but I can test this if necessary. This was a "luma" with sample data re-build job and almost all the sample data is gone.

Order still flow into the system, but we get to have the "Please go to Cache Management and refresh cache types." banner to remind us to "flush and crawl" the cache and site to get it going again.

I'm not sure if it matters, but the web upgrade process won't work, so I have been upgrading via the command line process. php bin/magento setup:upgrade and php bin/magento setup:static-content:deploy as I recall

Hope that helps

[andidhouse]

Hi there,

we use the built in cache.
The same happens if we save a category - all pages are uncached after saving.

Is there maybe a way of a wrong setup in magento backend that can cause this or a wrong mode like developer mode the system is running on etc?

[andidhouse]

We had a magento developer looking over this topic and it really looks like magento itself is clearing all cache data after each order:

"On checkout success page, magento clear category tags that related with purchased product (catalog_category_226,catalog_category). Catalog inventory module dispatch cache clean task."

Can someone other confirm this bug on as well on a 2.0.5 system?

[Vinai]

I think I saw this issue in Magento 1.14, too. The top menu was cached with all category tags associated with the cache entry. Not sure if it was core behavior or because of a customization though.

[andidhouse]

Yes i think it has something to to with this:
Top menu (that visible on all pages), related with tag catalog_category.

[joebusby]

I'm not sure what happened, but orders are coming in now and all system caches are remaining validated. I did reboot once or twice, but tried both Payflo pro and payflo link and they are both working on quite a few orders with no disruption. I will update this thread if it starts happening again.

[joanhe]

@joebusby Thanks for your information. Could you check if Varnish is used now?

[joebusby]

@joanhe Yes, Varnish is caching all the http output, https/ssl is routed around the varnish server (direct to Magento 2) using port forwarding. All Magento caches are enabled and have been (suprise) staying that way. CDN was set up last week and running about a 70% hit rate, using both media and static entries.

[andidhouse]

Hi,

we tested on another 2.0.5 environment and here the same with magento built in cache.
Can someone other also confirm this to make shure it is not our setup somehow?

Many thanks!

[andidhouse]

We just realized that also saving one product in the backend causes the magento cache to be flushed completely.

Is this really a bug or are we in a wrong setup/mode?

[joanhe]

@andidhouse @joebusby I spent some time investigate this problem. This issue only appears in Magento 2 with Built-in Cache. Everything works fine with Varnish. This is due to absence of ESI protocol support in Built-in Cache. As a results update of single product can lead to invalidation of category data that triggers correspondingly invalidation of menu, last one lead to almost all catalog pages invalidation. Built-in Cache currently has a few limitations. This is one of the limitations. We are working on a document to explain the different level of support of Varnish and Built-in cache in Magento 2. I hope this will help user to decide which one to choose. Currently Varnish is the one recommended for production.

[choukalos]

I'm converting this to a feature request and putting this under our internal story ( MAGETWO-36845 ). @Vinai I believe this exists in M1x . From a timeline perspective this is pretty low priority - focus more heavily on Varnish for caching ( hence our recommendation ) and other performance work at this point.

[joebusby]

If it's easy, please check to see if this happens when you enter an order on the backend through SSL. If so, it's still a bug since Varnish does not support SSL. We should not have to set up a secure reverse proxy to use the admin interface.

[joebusby]

I just entered an order through the backend and it invalidated the cache. Still a bug in my mind.

[andidhouse]

First - thanks all for your answers and response.

So for us - magento internal chache solution is not usable at the moment due to this at all. A caching cleaning every cache at one backend action or frontend order is not an option.

For varnish:
Can some magento internal confirm that magento CE has the same varnish functions as magento enterprise? I read some time ago that varnish was not usable for CE in 2.0 as some important features where missing.
I also found no document showing if Magento 2.0 CE and Varnish is supporting clean "by tags" so when action is taken (for example a product edit at the backend or a order at frontend/backend) only the products / categories are uncached which are effected?
There is also a big discussion here internally when the varnish will clean a product / category cache:

• If product qty changes OR if stock status changes

Or where to set this settings?

I think for now it would also be good to have some statement here in the official documentation that all caches are cleaned at any action - we spent nearly two days investigation this as we thought an state of the art commerce system could not have such a bug (and yes in my opinion it is a big bug because it makes the internal fpc not usable at all).

Many thanks!

[joebusby]

Update:
Made changes to descriptions and on some products and the Mage page cache started invalidating on each order again. All caches were cleared, no help. System was re-indexed, no help. Re-deployed static files, no help. Reboot has fixed the problem so far. Possibly restarting the web server would have worked also. Using Magento 2 with varnish on http traffic is working again. An order submitted through the admin interface using https did NOT invalidate the page cache after the reboot. It appears we will need to restart the server after product or configuration changes to stop the cache invalidation. Otherwise its looking good.

[andidhouse]

Just installed and tested varnish version 4.x. - very rocking fast! But also here: the cache gets invalidated from time to time.. also no orders or backend action is taken. This happens regularly but i can´t figure out what it is that invalidates the varnish as well.

Any suggestions here? (Maybe this is an own topic)

[joebusby]

I too have problems with the page cache getting invalidated with or without Varnish. It's better with Varnish, but I have to re-index and clear the cache and reboot to get it to work, and even then it only lasts for a while. Any configuration change at all seems to make it start invalidating the cache on every order again. I have been trying to figure out what specific steps make it work again, but it seems to be the re-index, clear cache, clear CDN and reboot.

[andidhouse]

@joebusby i think it is normal that varnish purges on configuration changes (see magento documentation for this). BUT it is not normal that varnish invalidates without any configuration change completely. At our side varnish purges also from time to time also no orders or backend action is made which makes it not usable. we also set the malloc to 3g what should be more then enough space for the cache....

[joebusby]

@andidhouse Yes, it should invalidate the page cache on configuration changes, but I guess I was expecting that clearing the cache would allow you to take orders again without the page cache invalidating each time again until re-index and reboot.

[andidhouse]

hi @joebusby @joanhe,

i created a new issue regarding varnish cache functions and in my opinion no documentation and bug: #4688

I hope to find answers here because i think it is also lack of documentation on varnish caching which caches are invalidated on which actions for default and how you can change this.

[daniel-ifrim]

@joebusby @andidhouse See what Ivan Chepurnyi says about top menu cache tags in Magento 1. It's the same for Magento 2. https://www.youtube.com/watch?v=7XwvjLWV1O0
Magento 2 sends to Varnish the identities, see in topmenu block addIdentity and getIdentities functions.
About identities in Magento 2 and Varnish:
http://magento.stackexchange.com/questions/112162/what-is-identityinterface-in-magento2

[piotrekkaminski]

Thank you for your submission.

We recently made some changes to the way we process GitHub submissions to more quickly identify and respond to core code issues.

Feature Requests and Improvements should now be submitted to the new Magento 2 Feature Requests and Improvements forum (see details here).

We are closing this GitHub ticket and have moved your request to the new forum.

[TimQSO]

Does someone has a solution yet?

[TimQSO]

@andidhouse werre you able to solve this already?