[4.1] poor man’s cron 3rd attempt #29592

alikon · 2020-06-13T08:31:03Z

The last iteration of #25022, porting #22310 to 4.x

Goal

a way to run scheduled "quick" tasks in approximately regular intervals

Summary of Changes

a new system plugin scheduler
two new console command

How it works

In a normal hosting situation that would be job for cron, but if your hosting doesn’t offer cron, then a common solution in these cases is “poor man’s cron:” on every page load (onAfterRespond) the scheduler system plugin check if itself is expected to run and then trigger a new event onExecuteScheduledTask then any job plugin wich listen to that event check if itself is expected to run and if so execute his task.

Testing Instructions (web)

apply PR
System -> database -> Update Structure (there is a new table)
System panel -> discover -> you'll find a new system plugin scheduler
install it
set options, publish the plugin and save
for test only
Frequency (in minutes) set = 0 mean run every (onAfterRespond)

install these 2 silly job plugin:
plg_job_jobthree.zip
plg_job_jobone.zip
this one is an example on how to schedule an sql export
plg_job_jobexportdb.zip
job one simply sleep for 1 seconds
job two simply sleep for 3 seconds
set options, publish the plugin and save plugin ONE
Frequency (in unit) set 1
Unit of time set minute

means that this plugin run 1xminute = every minute
if onExecuteScheduledTask has been triggered

set options, publish the plugin and save plugin TWO
maybe use different settings than ONE

Expected result

depending on your (plugins) settings
take a coffe/smoke/whatever 😄
randomly click a couple of frontend pages
check the log file at yoursite/administrator/logs/joomla_web.php
you'll see something like this

#
#<?php die('Forbidden.'); ?>
#Date: 2019-05-30 17:24:39 UTC
#Software: Joomla! 4.0.0-alpha10-dev Development [ Amani ] 16-May-2019 19:22 GMT

#Fields: datetime	priority clientip	category	message
2019-05-30T17:24:39+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2019-05-30T17:24:40+00:00	INFO 127.0.0.1	scheduler	Job>jobone Task#221 Processing Time: 1.01 seconds.
2019-05-30T17:24:43+00:00	INFO 127.0.0.1	scheduler	Job>jobtwo Task#166 Processing Time: 3.01 seconds.
2019-05-30T17:24:43+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#1564 Total Processing Time: 4.04 seconds.
2019-05-30T17:24:47+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2019-05-30T17:24:47+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#1565 Total Processing Time: 0.04 seconds.

1st run
the scheduler plugin (wich runs always frequency=0) trigger the onExecuteScheduledTask event
so plugin ONE and plugin TWO are executed (cause no previous run)

2nd run
the scheduler plugin (wich runs always frequency=0) trigger the onExecuteScheduledTask event
but this time both plugin have an expected scheduled time (for example ONE each 1min TWO each 5min)
the 2nd run was at 17:24:47+00:00
the 1st run was at 17:24:43+00:00
they have only 4 seconds of difference form the previous execution so both plugin are not expected to run

3rd run
it depends on how fast you click & settings

Testing Instructions (webcron)

on scheduler system plugin
set Webcron to yes/enable
set Key to mywebcronactivationkey
then use yoursiteurl/index.php?webcronkey=mywebcronactivationkey as the url for webcron services to be triggred

Testing Instructions (cli/cron)

Go to the cli folder of your website and run
php joomla.php job:run

check the log file at yoursite/administrator/logs/joomla_cli.php

php joomla.php job:list

php joomla.php job:run --help

Actual result

N/A

Additional comments

it can be used for tasks like:

deleteExpiredConsents
remindExpiringConsents
logRotation
updateNotifications
sessionGC
etc...

Documentation Changes Required

yes

ceford · 2020-06-13T09:48:53Z

I am getting this in the joomla_web.php log:

#
#Date: 2020-06-13 09:28:59 UTC
#Software: Joomla! 4.0.0-beta2-dev Development [ Mañana ] 30-May-2020 21:13 GMT

#Fields: datetime priority clientip category message
2020-06-13T09:28:59+00:00 INFO ::1 scheduler PLG_SYSTEM_SCHEDULER_LOCKED
2020-06-13T09:30:05+00:00 INFO ::1 scheduler PLG_SYSTEM_SCHEDULER_LOCKED
2020-06-13T09:32:07+00:00 INFO ::1 scheduler PLG_SYSTEM_SCHEDULER_LOCKED
2020-06-13T09:36:45+00:00 INFO ::1 scheduler PLG_SYSTEM_SCHEDULER_LOCKED

The CLI gives this:

Scheduler

Scheduler started

Job: jobthree In Error
Job: jobone In Error
Job: jobexportdb In Error
[OK] Scheduler finished in 0.009

joomla_cli.php log has this:

#
#Date: 2020-06-13 09:38:03 UTC
#Software: Joomla! 4.0.0-beta2-dev Development [ Mañana ] 30-May-2020 21:13 GMT

#Fields: datetime priority clientip category message
2020-06-13T09:38:03+00:00 INFO - scheduler Scheduler started
2020-06-13T09:38:03+00:00 INFO - scheduler Scheduler tooks:0.008

My System - Job Scheduler plugin is showing Yes/No radio buttons, not a slider as in your screen grab. So has this mornings Nightly Build update not worked properly for me?

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/29592.}

alikon · 2020-06-13T09:52:05Z

@ceford most probably i've messed up something making the pr....let me check/re-check , and sorry for the extra work

richard67 · 2020-06-13T10:30:30Z

Update SQL script for adding the new table seems to be missing.

alikon · 2020-06-13T10:51:38Z

are this https://github.com/joomla/joomla-cms/pull/29592/files#diff-fedf44f6151687d314fc4f3cab068494 not enough

richard67 · 2020-06-13T12:00:10Z

@alikon Yes, seems I've missed them when checking on smartphone. But the insert statements into the extensions table are missing the meanwhile (since your last version of the PR) invented new column "locked". Could you check and if necessary correct?

richard67 · 2020-06-13T12:56:58Z

@alikon See alikon#69.

particthistle · 2020-06-13T15:08:04Z

I have tested this item ✅ successfully on 1c1c740

Tested web version. Had to apply database fix using System > Database first.

Changed the frequency to 5 min on Job 3 to test the frequency cycle, and sleep time to 10 seconds, which is why the figures for Job>jobthree Task#3 is delayed compared to the other instances.

Refreshed page a few times a few minutes apart. Log shown below.

#Date: 2020-06-13 14:47:47 UTC
#Software: Joomla! 4.0.0-beta2-dev Development [ Ma単ana ] 30-May-2020 21:13 GMT

#Fields: datetime	priority clientip	category	message
2020-06-13T14:47:47+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:47:52+00:00	INFO 101.116.160.42	scheduler	Job>jobthree Task#1 Return Code is: 0 Processing Time: 5.06 seconds.
2020-06-13T14:47:53+00:00	INFO 101.116.160.42	scheduler	Job>jobone Task#1 Return Code is: 0 Processing Time: 1.06 seconds.
2020-06-13T14:47:53+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 6.17 seconds.
2020-06-13T14:48:05+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:48:05+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 0.03 seconds.
2020-06-13T14:48:57+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:49:01+00:00	INFO 101.116.160.42	scheduler	PLG_SYSTEM_SCHEDULER_LOCKED
2020-06-13T14:49:02+00:00	INFO 101.116.160.42	scheduler	Job>jobthree Task#2 Return Code is: 0 Processing Time: 5.03 seconds.
2020-06-13T14:49:03+00:00	INFO 101.116.160.42	scheduler	Job>jobone Task#2 Return Code is: 0 Processing Time: 1.04 seconds.
2020-06-13T14:49:03+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 6.11 seconds.
2020-06-13T14:49:05+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:49:05+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 0.04 seconds.
2020-06-13T14:50:42+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:50:43+00:00	INFO 101.116.160.42	scheduler	Job>jobone Task#3 Return Code is: 0 Processing Time: 1.02 seconds.
2020-06-13T14:50:43+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 1.05 seconds.
2020-06-13T14:51:42+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:51:42+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 0.06 seconds.
2020-06-13T14:51:56+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:51:57+00:00	INFO 101.116.160.42	scheduler	Job>jobone Task#4 Return Code is: 0 Processing Time: 1.04 seconds.
2020-06-13T14:51:57+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 1.07 seconds.
2020-06-13T14:53:40+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:53:41+00:00	INFO 101.116.160.42	scheduler	Job>jobone Task#5 Return Code is: 0 Processing Time: 1.05 seconds.
2020-06-13T14:53:41+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 1.11 seconds.
2020-06-13T14:59:30+00:00	INFO 101.116.160.42	scheduler	Starting Scheduler
2020-06-13T14:59:40+00:00	INFO 101.116.160.42	scheduler	Job>jobthree Task#3 Return Code is: 0 Processing Time: 10.03 seconds.
2020-06-13T14:59:41+00:00	INFO 101.116.160.42	scheduler	Job>jobone Task#6 Return Code is: 0 Processing Time: 1.02 seconds.
2020-06-13T14:59:41+00:00	INFO 101.116.160.42	scheduler	Scheduler tooks 11.10 seconds.

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/29592.}

administrator/language/en-GB/plg_system_scheduler.ini

plugins/system/scheduler/scheduler.xml

Co-authored-by: Tobias Zulauf <[email protected]>

zero-24

Looks fine to me. Now we have to get two tests so this can be finally merged.

zero-24 · 2020-10-01T07:49:56Z

I have just updated 4.1-dev with 4.0 and updated this PR acordingly so once drone is happy this can be fully tested :)

alikon · 2020-10-01T16:08:47Z

the "Download — Prebuilt packages are available for download." is missing from the 4.1-dev pipeline ?

bembelimen · 2020-11-05T09:30:51Z

I really like the idea of having a Cron fallback in the Joomla! and it's also a feature on the roadmap, so really thank you for bringing this code in.

As we have still some time for Joomla! 4.1 I really would love to see to go a few steps further, so that we get a solution where the end user can manage crons over an interface and at the end the entries will be saved in a cron tab like way in the database.

It would also be great if cron is available on the server there could be some integration.
Also a CLI functionality and some API (helper?) would really be great.

Last but not least probably we should also take care of the user executing the Cron. In the worse case it's a user who has not the privileges.

(Thanks to @nibra and @HLeithner for a lot of valuable feedback here)

tl;tr; I really like what you did but I think more is possible, would you be up to spend some more time and enhance your great start to bring it on the next level?

alikon · 2020-11-11T13:41:27Z

I think more is possible

of course,

but what we have now in this pr is a good start.....

the 1st step when and if this pr will be merged should be to use the poor man cron's for tasks like :

deleteExpiredConsents
remindExpiringConsents
logRotation
updateNotifications
sessionGC
etc...

but i'll not spent more time on this, if this pr will not be merged (i've proposed this before 3.9 goes stable)

p.s
you don't need to manage complex crontab entry
something like this is enough

* * * * * /usr/local/bin/php /shared/httpd/j4/htdocs/cli/joomla.php job:run >> /shared/httpd/cron.log

alikon · 2021-06-26T11:43:06Z

if there is still someone interested i've decoupled it to be an external component https://github.com/alikon/testcom

bembelimen · 2021-06-26T14:00:04Z

Would it be possible that you recover the branch for a few days, so I can copy it into my repository? I still like the base idea and implementation.

alikon added 16 commits June 13, 2020 09:19

cli command

e62bdd2

cli command

73ef62b

joblist cli command

2c04462

cli scheduler command

9263dc6

the scheduler plugin

ab72f7a

the scheduler plugin manifest

a8d7666

sql

4b43675

sql

d236b9c

sql

fe1e9ae

sql

7919c0f

the jobs plugin

8f86ce8

log the run of the job plugin

3db81a9

option to log the job execution

bb03509

lang

ca2ab32

lang

5edf32c

lang

1c1c740

joomla-cms-bot added Language Change This is for Translators PR-4.0-dev labels Jun 13, 2020

alikon mentioned this pull request Jun 13, 2020

[4.0] poor man’s cron #25022

Closed

Fix extensions table inserts in update sql, fix semicolon

5801238

Remove factory

ae1ba3d

Quy reviewed Jun 13, 2020

View reviewed changes

administrator/language/en-GB/plg_system_scheduler.ini Outdated Show resolved Hide resolved

Quy reviewed Jun 13, 2020

View reviewed changes

administrator/language/en-GB/plg_system_scheduler.ini Outdated Show resolved Hide resolved

zero-24 reviewed Oct 1, 2020

View reviewed changes

plugins/system/scheduler/scheduler.xml Show resolved Hide resolved

alikon and others added 14 commits October 1, 2020 09:23

Update administrator/components/com_plugins/src/Jobs/JobsPlugin.php

2414996

Co-authored-by: Tobias Zulauf <[email protected]>

Update administrator/components/com_plugins/src/Jobs/JobsPlugin.php

9e00057

Co-authored-by: Tobias Zulauf <[email protected]>

Update administrator/components/com_plugins/src/Jobs/JobsPlugin.php

f9dd5d3

Co-authored-by: Tobias Zulauf <[email protected]>

Update administrator/components/com_plugins/src/Jobs/JobsPlugin.php

064b491

Co-authored-by: Tobias Zulauf <[email protected]>

Update administrator/components/com_plugins/src/Jobs/JobsPlugin.php

f3a7f0b

Co-authored-by: Tobias Zulauf <[email protected]>

Update libraries/src/Console/SchedulerCommand.php

be4020a

Co-authored-by: Tobias Zulauf <[email protected]>

Update plugins/actionlog/joomla/joomla.php

76f30ea

Co-authored-by: Tobias Zulauf <[email protected]>

Update plugins/system/scheduler/scheduler.php

7b6b693

Co-authored-by: Tobias Zulauf <[email protected]>

Update plugins/system/scheduler/scheduler.xml

88bfe75

Co-authored-by: Tobias Zulauf <[email protected]>

Update administrator/language/en-GB/plg_system_scheduler.ini

3bb4ad2

Co-authored-by: Tobias Zulauf <[email protected]>

Update plugins/actionlog/joomla/joomla.php

ea2d2cd

Co-authored-by: Tobias Zulauf <[email protected]>

Update administrator/language/en-GB/plg_system_scheduler.sys.ini

69a2698

Co-authored-by: Tobias Zulauf <[email protected]>

Update libraries/src/Console/JobListCommand.php

0e5ab83

Co-authored-by: Tobias Zulauf <[email protected]>

Update libraries/src/Console/JobListCommand.php

1bbf39a

Co-authored-by: Tobias Zulauf <[email protected]>

zero-24 approved these changes Oct 1, 2020

View reviewed changes

Merge branch '4.1-dev' into patch-121

56a5201

alikon added 3 commits October 1, 2020 09:59

cs

22b1179

cs

ea573a2

cs

776b3af

alikon closed this Jun 26, 2021

alikon deleted the patch-121 branch June 26, 2021 11:43

This comment was marked as abuse.

Sign in to view

ditsuke mentioned this pull request Aug 15, 2021

[4.1] [SOC 21] Add a Scheduled Tasks Infrastructure to Joomla #35143

Merged

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[4.1] poor man’s cron 3rd attempt #29592

[4.1] poor man’s cron 3rd attempt #29592

alikon commented Jun 13, 2020 •

edited

Loading

ceford commented Jun 13, 2020

alikon commented Jun 13, 2020

richard67 commented Jun 13, 2020

alikon commented Jun 13, 2020

richard67 commented Jun 13, 2020

richard67 commented Jun 13, 2020

particthistle commented Jun 13, 2020 •

edited

Loading

zero-24 left a comment

zero-24 commented Oct 1, 2020

alikon commented Oct 1, 2020

bembelimen commented Nov 5, 2020 •

edited

Loading

alikon commented Nov 11, 2020

alikon commented Jun 26, 2021

bembelimen commented Jun 26, 2021

This comment was marked as abuse.

[4.1] poor man’s cron 3rd attempt #29592

[4.1] poor man’s cron 3rd attempt #29592

Conversation

alikon commented Jun 13, 2020 • edited Loading

Goal

Summary of Changes

How it works

Testing Instructions (web)

Expected result

Testing Instructions (webcron)

Testing Instructions (cli/cron)

Actual result

Additional comments

Documentation Changes Required

ceford commented Jun 13, 2020

Scheduler

alikon commented Jun 13, 2020

richard67 commented Jun 13, 2020

alikon commented Jun 13, 2020

richard67 commented Jun 13, 2020

richard67 commented Jun 13, 2020

particthistle commented Jun 13, 2020 • edited Loading

zero-24 left a comment

Choose a reason for hiding this comment

zero-24 commented Oct 1, 2020

alikon commented Oct 1, 2020

bembelimen commented Nov 5, 2020 • edited Loading

alikon commented Nov 11, 2020

alikon commented Jun 26, 2021

bembelimen commented Jun 26, 2021

This comment was marked as abuse.

alikon commented Jun 13, 2020 •

edited

Loading

particthistle commented Jun 13, 2020 •

edited

Loading

bembelimen commented Nov 5, 2020 •

edited

Loading