cron_zencache_cleanup never executing!

[xberg]

Hi
Using Pro version 160103.
The _cron_zencache_cleanup never executes. I use Crontrol to monitor the wp_cron and there's very strange behavior: The "next run" time keeps changing!!! I have it set to run every 3 hours, but the next run time is always in just 1 minute time! When I refresh the crontrol page I see your CRON is still schedule to run in a minute. and it NEVER NEVER runs. So this just fills my cache and I have several times a day to manually delete the cache!
Same bahavior with _cron_zencache_auto_cache but I don't use this.

I tried deleting and reinstalling: same problem.

And you CRONs are the ONLY ones I cannot delete or edit with crontrol. It's like your system is continusouly deleting and recreating new ones!

Please help me fix this ASAP.

[xberg]

I do NOT have the Disable Cache Expiration If Server Load Average is High enabled, but it seems to behave like it is and that my server is high.

[raamdev]

@xberg I'm not able to reproduce this behavior. The cronjobs are behaving as expected for me:

40 seconds before running

While running

Just after running

---

I suggest disabling deletion safeguards (ZenCache → Plugin Options → Plugin Deletion Safeguards) and then deactivating and deleting the ZenCache plugin. Then install it fresh and see if you still have issues.

(Note that if you have a custom plugin configuration that you want to backup, you can use ZenCache → Plugin Options → Import/Export Options.)

Also, it's normal for ZenCache to recreate the cron events if you delete them. ZenCache needs those events to be present for the plugin to work properly, so if ZenCache is active and it detects the events are missing, it will recreate them. If you don't want ZenCache to recreate the events, simply deactivate the plugin.

[raamdev]

@xberg Also, if you want to change the schedule at which the cache cleanup occurs, you can create a custom schedule with WP Crontrol (Dashboard → Settings → Cron Schedules) and then select that custom schedule in ZenCache → Plugin Options → Cache Expiration Time → Cache Cleanup Schedule:

[xberg]

Raam,
Just because you cannot reproduce the issue does not mean that the issue is not there.
I am experienced with CRON jobs and I can tell you that there is a major problem on my setup with just your plugin, and this only started a few weeks ago (at least 2 releases ago).
Yes, I can change the schedule to whatever I want, but your cleanup NEVER runs. It keeps reseting itself to run in a minute (but NEVER runs).

Let's try to find a temporary workaround: is there any code I could run thru a PHP to replicate the behavior of the cleanup? ie, I would be creating my own CRON?

Because I am using a 2 GB RAMDRIVE it fills in less than a day and then starts producing a massive number of errors. As such your plugin in unusable now.

[xberg]

http://prntscr.com/9oo1oz
http://prntscr.com/9oo1t5
http://prntscr.com/9oo1wq
http://prntscr.com/9oo1zu
http://prntscr.com/9oo22i
http://prntscr.com/9oo25s
http://prntscr.com/9oo2b0

Here I am refreshing CRONTROL every 20 seconds. You can see that your CRONs are alway reset to run in 1 minute. In the meanwhile you can see that another cron, the NXS decrements correctly.

You have a problem and you need to acknowledge it.

[raamdev]

@xberg Have you reproduced this issue in a clean WordPress installation? See Testing ZenCache in a Clean WordPress Installation

If you have, then please provide me with a list of steps to reproduce this issue on my site.

[xberg]

Raamdev,

Do you have anything in your program that resets the counter to 1 minute? It would not surprise me at all that your "Disable Cache Expiration If Server Load Average is High" is creating the problem.

Otherwise is there any instruction I could run to replicate the behavior of your CRON?

Without this, what is your reimbursement policy as I can no longer use your caching with this bug.

[xberg]

Ah!! I found your code that was creating the bug. I cannot believe you did not immediately find it yourself after my description!

It's code you added 20151220 which resets the CRON to run in a minute: exactly the description I made initially!!!

It's in CronUtils.php and here it is:

/*

• Checks Cron setup, validates schedules, and reschedules events if necessary.
  *

• @attaches-to init hook.
  *

• @SInCE 151220 Improving WP Cron setup and validation of schedules
  /
  $self->checkCronSetup = function () use ($self) {
  if ($self->options['crons_setup'] < 1439005906
  || $self->options['crons_setup_on_namespace'] !== NAMESPACE
  || $self->options['crons_setup_with_cache_cleanup_schedule'] !== $self->options['cache_cleanup_schedule']
  || $self->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))
  || !wp_next_scheduled('cron'.GLOBAL_NS.'cleanup')
  /[pro strip-from="lite"]/ // Auto-cache engine.
  || !wp_next_scheduled('cron'.GLOBAL_NS.'auto_cache')
  /[/pro]*/
  ) {

  wp_clear_scheduled_hook('_cron_'.GLOBAL_NS.'_cleanup');
  wp_schedule_event(time() + 60, $self->options['cache_cleanup_schedule'], '_cron_'.GLOBAL_NS.'_cleanup');
  
  /*[pro strip-from="lite"]*/ // Auto-cache engine.
  wp_clear_scheduled_hook('_cron_'.GLOBAL_NS.'_auto_cache');
  wp_schedule_event(time() + 60, 'every15m', '_cron_'.GLOBAL_NS.'_auto_cache');
  /*[/pro]*/
  
  $self->updateOptions(
    array(
      'crons_setup'                             => time(),
      'crons_setup_on_namespace'                => __NAMESPACE__,
      'crons_setup_with_cache_cleanup_schedule' => $self->options['cache_cleanup_schedule'],
      'crons_setup_on_wp_with_schedules'        => sha1(serialize(wp_get_schedules()))
    )
  );
  

  }
  };

/*

• Resets crons_setup and clears WP-Cron schedules.
  *
• @SInCE 151220 Fixing bug with Auto-Cache Engine cron disappearing in some scenarios
  *
• @note This MUST happen upon uninstall and deactivation due to buggy WP_Cron behavior. Events with a custom schedule will disappear when plugin is not active (see http://bit.ly/1lGdr78).
  /
  $self->resetCronSetup = function ( ) use ($self) {
  if (is_multisite()) { // Main site CRON jobs.
  switch_to_blog(get_current_site()->blog_id);
  /[pro strip-from="lite"]/ // Auto-cache engine.
  wp_clear_scheduled_hook('cron'.GLOBAL_NS.'auto_cache');
  /[/pro]/
  wp_clear_scheduled_hook('cron'.GLOBAL_NS.'cleanup');
  restore_current_blog(); // Restore current blog.
  } else { // Standard WP installation.
  /[pro strip-from="lite"]_/ // Auto-cache engine.
  wp_clear_scheduled_hook('cron'.GLOBAL_NS.'auto_cache');
  /[/pro]*/
  wp_clear_scheduled_hook('cron'.GLOBAL_NS.'_cleanup');
  }
  $self->updateOptions(
  array( // Reset so that crons are rescheduled upon next activation
  'crons_setup' => $self->default_options['crons_setup'],
  'crons_setup_on_namespace' => $self->default_options['crons_setup_on_namespace'],
  'crons_setup_with_cache_cleanup_schedule' => $self->default_options['crons_setup_with_cache_cleanup_schedule'],
  'crons_setup_on_wp_with_schedules' => $self->default_options['crons_setup_on_wp_with_schedules']
  )
  );
  };

[xberg]

Remving this line fixed your bug:

|| $self->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))

[jaswrks]

@xberg Watch it with the tone! Show some respect here, and we will do the same.

I cannot believe you did not immediately find it yourself after my description!

Please cleanup the code that you posted and I will review as well. So far it looks like you solved the problem, but I'm not clear on why you're calling this a 🐛 yet. Please explain in greater details and we can work together at solving it.

[xberg]

Youi're right: I'm sorry about the tone.

[raamdev]

@xberg writes...

$self->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))`

The only reason that line would cause problems is if wp_get_schedules() changes on every page load (and that should never happen, which again, is why I asked you to reproduce this in a clean installation of WordPress to rule out issues with other plugins or custom code).

That line basically says, "if there has been a change to the WordPress Cron Schedules (i.e., you added/removed/changed a cron schedule), then set up the ZenCache cron events again". This is necessary because if you changed or delete a cron schedule that ZenCache was using, ZenCache must reconfigure its own cron events to make sure that it keeps working as expected.

I'm not seeing any bug in that code. I have tested it and it is working as expected.

[xberg]

Raamdev,
I am not a developer so I cannot help you. I just know that removing this line fixed my problem. With the line there was a continuous loop which kept creating a new cron job that was to execute in a minute, but immediately got deleted and recreated.

I understand it's hard to debug without being able to replicate, but I am convinced I am not the only one who will have this problem so at least now you know what has helped me fix things. The only reason I noticed this problem is that I am using a 2 GB Ramdrive for zencache and this filled up fast. If I were using harddrive it could have been weeks before I noticed a problem. And I also have crontrol installed which allowed me to see this problem: most users will not have crontrol.

[raamdev]

@xberg Again, have you reproduced this in a clean WordPress environment? I'm guessing you did not, because if you did I have a strong feeling you'd see that there is no bug there. :-)

[lkraav]

@raamdev I'm seeing the same thing on my multisite installation subsites. This cron schedule gets bumped to 1min away on every page load. There doesn't seem to be anything that would be messing with cron schedules that would cause the hash to change. I'll investigate some more.

EDIT !wp_next_scheduled('_cron_'.GLOBAL_NS.'_auto_cache') always evaluates to true for me, causing the ever-reset symptom.

EDIT I'm just going to flat out return from $self->checkCronSetup() right now pending further instructions. This eliminates slow update queries on every page load.

[xberg]

Thanks ikraav for sharing your fix. Concretly, what file did you edit?

[lkraav]

Thanks ikraav for sharing your fix. Concretly, what file did you edit?

./src/includes/closures/Plugin/CronUtils.php the same thing you already found. It's not a fix, just a stopgap until people are able to figure out what's happening.

[raamdev]

@lkraav The wp_next_scheduled() function (see docs) just checks if a given action hook, in this case _cron_zencache_auto_cache, has been scheduled.

If the conditional where that check exists evaluates to true (i.e., the event has not been scheduled), then ZenCache schedules the event with this line:

wp_schedule_event(time() + 60, 'every15m', '_cron_'.GLOBAL_NS.'_auto_cache');

So once the event has been scheduled, !wp_next_scheduled('_cron_'.GLOBAL_NS.'_auto_cache') should return false, since the event has now been scheduled.

https://github.com/websharks/zencache-pro/blob/160103/src/includes/closures/Plugin/CronUtils.php#L23-L58

I can't explain why that's not working properly for you, but in all of our tests it works as expected.

[lkraav]

I will re-evaluate each of the conditions again. It may not be about just wp_next_scheduled() here.

[xberg]

Today I upgraded to V160222 Pro and the issue was back.
I returned to
/wp-content/plugins/zencache-pro/src/includes/closures/Plugin/CronUtils.php
and commented out as I did before:
|| $self->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))

but this time it was not enough, and the CRON kept resetting itself to 1 minute.

But also commenting out:
|| $self->options['crons_setup_with_cache_cleanup_schedule'] !== $self->options['cache_cleanup_schedule']
worked and now the CRON is no longer reset.

Just FYI, as I have no time to dig further into this matter and that my fix solves the problem for me. Good luck.

[raamdev]

@xberg @lkraav I'm reopening this issue.

@lkraav reported in #667 (comment) that this issue occurred again with the Comet Cache update, which tells me this bug is most likely related to a certain combination of events that are not being considered in the checkCronSetup() routine, which is resulting in that routine updating the database repeatedly.

I have yet to figure out what combination of events is causing this, but since I have seen this happen during testing in the past (it happened to me once or twice, but then the problem disappeared and I was not able to reproduce it again at the time).

I'm going to reopen this GitHub issue so that we can continue tracking reports and so that it can be researched.

[xberg]

Hi: quick update. Today I upgraded from Zencache Pro to Comet cache Pro V 160227 and the problem re-appeared.
So this time I had to comment out 3 conditions, as commenting out only 2 still continusouly resets the CRON to 60 seconds.
The additional condition I had to comment out was:
!wp_next_scheduled('_cron_'.GLOBAL_NS.'_cleanup')

in addition to:

  //|| $self->options['crons_setup_with_cache_cleanup_schedule'] !== $self->options['cache_cleanup_schedule']
        //|| $self->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))

Good luck, now that I know the "trick" it's not a big inconvenience to comment this out with each upgrade.

[raamdev]

@renzms Thanks. I'm closing this for now and will reopen if we get any more information.

[teknofilo]

@raamdev, I am running Comet Cache v160917 and I am also seeing this behaviour

_cron_comet_cache_cleanup job is always to scheduled to run in 1 minute, but it never runs. If run it manually, files are properly deleted.

[raamdev]

@teknofilo Could you provide us with some information about your system? WordPress version, PHP version, web server type (Apache/Nginx), and a list of active plugins that you have in WordPress would be helpful.

[teknofilo]

@raamdev sure, this is the info:

Linux 4.4.0-36-generic #55-Ubuntu SMP x86_64
PHP Version 7.0.8-0ubuntu0.16.04.3
Apache/2.4.18 (Ubuntu)

Plugins:

• Ad Inserter
• Advanced Content Pagination Pro
• Akismet
• All in One Buttons
• All in one Favicon
• Amazon Link
• AMP
• Arqam
• Better Internal Link Search
• Cloudflare
• Comet Cache
• Custom Menu Shortcode
• Disqus Comment System
• Easy Add Thumbnail
• Google Analytics by MonsterInsights
• Google News Sitemap
• Heartbeat Control
• Jetpack por WordPress.com
• Linkify Text
• Liveblog
• MailPoet Newsletters
• MCE Table Buttons
• New RoyalSlider
• NextScripts: Social Networks Auto-Poster
• No Self Pings
• Oomph Clone Widgets
• Page Links a
• Postman SMTP
• Priority Shortcodes
• Random Text
• Regenerate Thumbnails
• RJ Quickcharts
• Search Meter
• Set All First Images As Featured
• SI CAPTCHA Anti-Spam
• TablePress
• Theme Test Drive
• Use Google Libraries
• VaultPress
• WP Crontrol
• WP Edit
• WP Server
• WP-Memory-Usage
• WP-Optimize

[raamdev]

@teknofilo That's a lot of active plugins. Have you tried reproducing the issue in a clean WordPress installation?

Also, is this WP Multisite or just a standard single-site installation?

[teknofilo]

@raamdev

No, I have not tried to reproduce it, that is not easy.

It is a single-site installation.

I have been playing with the cleanup schedule and I have set it to 'hourly'. With this frequency, the cron job seems to be properly scheduled because now I see that the job will be executed in 60 mins, 40 mins, 20 mins etc. as time goes by.

However, the files in the directory are not deleted even though lots them are old (expiration time is 2 hours, so there should not be files >3 hours). It looks like the job is not executed or, if it is, it does not execute properly. Is there anything I can do to verify weather the clean up job gets executed?

[lkraav]

I can also confirm on multisite this issue is still there on v160917.

[xberg]

Hi team,
I just wanted to let you know that this issue is still there.
Every single version I need to comment out a few lines so that _cron_comet_cache_cleanup actually runs, otherwise it resets itself without ever running.
I installed today V170220 and the bug is still there.
Not sure if this is actively explored anymore...
thanks

In this version I was able to get it working by commenting out a single line (perhaps this will allow you to narrow down root causes):

This line is:
|| $this->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))
|| !wp_next_scheduled('cron'.GLOBAL_NS.'_cleanup')

in:
if ((!get_transient('doing_cron') && $this->options['crons_setup'] < 1439005906)
|| $this->options['crons_setup_on_namespace'] !== NAMESPACE
|| $this->options['crons_setup_with_cache_cleanup_schedule'] !== $this->options['cache_cleanup_schedule']
//|| $this->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))
|| !wp_next_scheduled('cron'.GLOBAL_NS.'_cleanup')
/[pro strip-from="lite"]/ // Auto-cache engine.
|| !wp_next_scheduled('cron'.GLOBAL_NS.'_auto_cache')
/[/pro]/
)

[xberg]

I only have 5 common plugins with tecnofilo above:
Comet Cache
MailPoet Newsletters
NextScripts: Social Networks Auto-Poster
TablePress
WP Crontrol

[raamdev]

@xberg Thanks for the additional information. The problem I'm having is reproducing this issue consistently. Until I can reproduce it consistently, it's rather difficult to identify the cause.

@renzms Could you try installing the above plugins with the latest version of Comet Cache and see if you can reproduce the problem described in this issue?

[renzms]

@raamdev

Unable to reproduce bug

List of Plugins:

Tested Using:

WordPress Version: 4.7.2
Current WordPress Theme: Twenty Seventeen version 1.1

PHP Version: 7.0.10-2+deb.sury.org~xenial+1
MySQL Version: 10.0.29-MariaDB-0ubuntu0.16.04.1

Active Plugins:
Comet Cache Version 170220
MailPoet Newsletters Version 2.7.7
NextScripts: Social Networks Auto-Poster Version 3.7.16
TablePress Version 1.7
WP Crontrol Version 1.4

for the purpose of multiple tests and time, the Cron: _cron_comet_cache_cleanup, was allowed to run on its own automatically set at 15 min intervals

WP CRONTROL

1 Min before running

While Running

After Running

Observations to consider

Manually Modifying the Cron Job via WP Crontrol does not necessarily update the Recurrence of the Cron Job until it actually runs the already in progress Cron Job (15min)

Executing the Cron Job Manually via WP Control does not reset the counter.

Manually Modifying the Cron Job via the Comet Cache Plugin, (Dashboard → Plugin Options → Cache Expiration Time) does not necessarily update the Recurrence of the Cron Job until it actually runs the already in progress Cron Job (15min)

---

Alternate Test with: Advanced Crontrol Manager Plugin

Advanced Crontrol Manager Plugin

1 Min before running

While Running

After Running

[raamdev]

@renzms Thanks for the testing!

---

I just did another review of this entire GitHub issue and the associated source code and I'm totally out of ideas. @xberg was unable to reproduce the issue in a cloned version of the site, but at least 3 other people have reported this issue, even after we implemented an attempted fix. The issue occurs in both Multisite and Single-Site installations.

The issue appeared to be related to one of (or both of) these two conditionals:

            || $this->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))
            || !wp_next_scheduled('_cron_'.GLOBAL_NS.'_cleanup')

I'm completely fresh out of ideas. :-) It would be nice to have access to a site that is experiencing this issue.

@jaswsinc Any ideas?

[jaswrks]

From our research in this issue, we have learned that WordPress schedules and then re-schedules events, which can lead to a loss of the custom timeframe that we add to WordPress via filters. That's a bug in core, which can result in a number of CRON events becoming corrupt.

I see two problems in the existing release:

• This bug that exists in WordPress core is potentially the underlying cause.
• Our recent attempts to work around that bug have led to some overly complex hacks that are actually making this more difficult for us to maintain and troubleshoot further; i.e., those conditionals that you referenced in the last comment are tripping my hackometer.

I see two possible avenues from here:

1. Work to correct this in core and submit a patch.
2. Or, avoid WP Cron altogether and deal with this on our own. The objective here is just to run the ACE every few minutes automatically. If we remove WP Cron from the picture, that's a pretty simple and straightforward task that is easily accomplished in PHP without any library.

[xberg]

Yes: this is very very strange behavior: on mysite.com I have the issue each time I update the plugin as this removes my commented-out line.
However on the clone which is hosted on the same server with exactly (*) the same setup / plugins but just a different URL : dev.mysite.com it works perfectly.
I just did this 2 days ago: a new clone, updated Comet Cache from V 161227 to the current version and it worked fine out of the box.

I'm sorry as I know none of this is helpful for your debugging. I cannot give you access to my production platform due to customer confidentiality.
However I am sure I am just the top of the iceberg: very few people care or know about 1. what's happening in their wp-cron and 2. the size of their cache folder. I'm a sort of alignment of stars in that I use limited memory to cache comet cache and I have wp-crontrol installed. Perhaps having a window in comet cache showing cron execution will bring a lot more tickets and data :)

(*) only difference is that the dev platform runs an additional plugin "stop email" to avoid sending out emails to users. And in DEV we define wp_home and wp_siteurl directly in wp-config.php... Oh: and DEV does NOT use memory to store comet cache folder

[raamdev]

@jaswsinc writes...

This bug that exists in WordPress core is potentially the underlying cause.

That reminded me of this todo I had in my task list from a long time ago that I never got around to, related to bugging WP Cron. It was filed in my task manager as unrelated to this issue, but now that you mentioned buggy WP Cron, I was reminded of it. There's likely more background related to this issue here: #613 (comment)

[ogessel]

I'm having the same issues, as Raam already knows. https://forums.wpsharks.com/t/cron-jobs-seem-to-be-running-always/2636

And I'm happy to give you admin access to my set up if you want to look around.

[ogessel]

And I can confirm that solution proposed by xberg of February 22nd does do the trick.

Since I'm not a developer, I'm not sure what it now is doing or not doing anymore, but after having commented out that line the Cron jobs seem to perform like the other ones. In English: normally.

Of course I still hope that about a bug fix will be available shortly.
Greets, Olaf

[xberg]

@ogessel Don't hold your breath for the fix: I have to do my manual fix with each update for the last 1.5 years. But indeed if you are willing to give the devs an admin access they might finally have a platform in which they can reproduce the bug.

[raamdev]

Until this issue can be reproduced in a testing environment, or until we can clearly identify what's causing the problem, it's not possible to put out a fix. The code that @xberg mentioned commenting out is designed to to make Comet Cache compatible with custom WP Cron schedules and to ensure that the Comet Cache WP Cron events are scheduled.

The following is an explanation of the purpose of the two statements that @xberg is commenting out.

---

$this->options['crons_setup_on_wp_with_schedules'] !== sha1(serialize(wp_get_schedules()))

This line checks if the WP Cron Schedules have changed by comparing a stored SHA1 hash in the database ($this->options['crons_setup_on_wp_with_schedules']) with the SHA1 hash of the current WP Cron Schedules (sha1(serialize(wp_get_schedules()))). If the hashes are different, that means that the WP Cron Schedules have been modified in some way and Comet Cache needs to go through the process of setting up its WP Cron events again to ensure that they are scheduled properly. Any WordPress site owner can modify the default WP Cron schedules. Other WordPress plugins can also modify WP Cron Schedules. In either event, Comet Cache needs to ensure that its own event is scheduled properly, because if somebody modifies a WP Cron Schedule that Comet Cache is using, Comet Cache needs to reschedule the even to ensure that the modified schedule is used.

---

!wp_next_scheduled('cron'.GLOBAL_NS.'_cleanup')

This line simply checks to see if the Comet Cache cleanup event has been scheduled. If it has not been scheduled, then Comet Cache needs to run through the setup routine to make sure that it's scheduled. Once it has been scheduled, this line should evaluate to false. However, as @lkraav noted in an earlier comment, he found that line was always evaluating to true, which was causing an endless loop for him. That makes little sense to me because it would indicate that Comet Cache is never able to properly schedule the cleanup event, or the wp_next_scheduled() WordPress function is returning an incorrect value.

---

Again, I have not been able to consistently reproduce this issue at all on my end. It seems to be somehow related to specific hosting environments, but unless I can get full access (admin, FTP, and database) to a server where this is occurring, it's simply not possible for me to troubleshoot the issue.