Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1.x] Queue Overhaul #456

Merged
merged 43 commits into from
Dec 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
14c1fde
Slight refactor for readability
srtfisher Aug 31, 2023
b66bb65
Refactor to queue model
srtfisher Aug 31, 2023
b82ed9a
Use the query builder internally
srtfisher Aug 31, 2023
464245a
WIP
srtfisher Aug 31, 2023
7d623aa
Add phpstan typings
srtfisher Sep 1, 2023
499a9e8
Continued work
srtfisher Sep 1, 2023
087f4d6
Starting out on count()
srtfisher Sep 1, 2023
af4cd98
Adding query builder count() method
srtfisher Sep 1, 2023
96866ce
Merge branch 'query-count' into queue-performance
srtfisher Sep 1, 2023
434167c
Todo for tomorrow
srtfisher Sep 5, 2023
7959eec
Set the meta when creating the job
srtfisher Sep 5, 2023
2b99103
Improving the logic
srtfisher Sep 6, 2023
cbfa715
Wrapping up until tmw
srtfisher Sep 7, 2023
a1a071b
Merge remote-tracking branch 'origin/0.12.x' into queue-performance
srtfisher Sep 8, 2023
68ed908
Continued work
srtfisher Sep 13, 2023
d709c5e
Merge branch 'date-query' into queue-performance
srtfisher Sep 20, 2023
09a7677
Use the date query builder to get the proper queue items
srtfisher Sep 20, 2023
eb7de25
Merging in date-query
srtfisher Sep 20, 2023
48b6094
Adjust cron logic to properly scale up to meet demand of the queue
srtfisher Sep 22, 2023
92a5d3c
Adding tests for cron queue
srtfisher Sep 22, 2023
8e76bbc
Linting fixes
srtfisher Sep 22, 2023
964a2f7
Wrapping up fixes for queue
srtfisher Sep 22, 2023
ae3024f
Merge branch 'date-query' into queue-performance
srtfisher Sep 22, 2023
3a699f7
Raise memory limit, skip queue tests on 8.0
srtfisher Sep 22, 2023
cde02d2
Prevent queue service provider on 8.1
srtfisher Sep 22, 2023
038e010
Remove bogus version
srtfisher Sep 22, 2023
5589efa
Merge branch '0.12.x' into queue-performance
srtfisher Sep 22, 2023
8198633
Merge branch '1.x' into queue-performance
srtfisher Sep 22, 2023
e6e0ad0
Fixing tests
srtfisher Sep 22, 2023
a6684c2
Remove bogus return
srtfisher Sep 22, 2023
b45579b
Add a log of queue job events
srtfisher Sep 22, 2023
2149fe5
Filling in the queue log
srtfisher Sep 22, 2023
e82e393
Adding a database collection and passing along the found_rows to the …
srtfisher Sep 28, 2023
42d1bc9
Allow the value to be passed
srtfisher Sep 28, 2023
8e15474
Merge branch 'found-posts-results' into queue-performance
srtfisher Sep 28, 2023
e052d23
Merge branch '1.x' into queue-performance
srtfisher Oct 2, 2023
40a7065
Merge branch '1.x' into queue-performance
srtfisher Nov 14, 2023
50b55a2
Merging in 1.x
srtfisher Nov 21, 2023
c4b945b
Queue UI (#458)
srtfisher Nov 29, 2023
480ece0
Preserve queue job items and cleanup after some time (#472)
srtfisher Nov 29, 2023
9100de2
Merge branch '1.x' into queue-performance
srtfisher Nov 29, 2023
0816b89
Remove checks for <8.1 now that were on PHP 8.1+ always
srtfisher Nov 29, 2023
eb2f897
[1.x] Allow queue jobs to be run in-admin, allow delays for closure j…
srtfisher Nov 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions config/queue.php
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
| Define the queue provider used in the application.
|
*/
'default' => environment( 'QUEUE_CONNECTION', 'wordpress' ),
'default' => environment( 'QUEUE_CONNECTION', 'wordpress' ),

/*
|--------------------------------------------------------------------------
Expand All @@ -25,7 +25,40 @@
| The amount of items handled in one run of the queue.
|
*/
'batch_size' => environment( 'QUEUE_BATCH_SIZE', 100 ),
'batch_size' => environment( 'QUEUE_BATCH_SIZE', 5 ),

/*
|--------------------------------------------------------------------------
| Maximum number of concurrent batches
|--------------------------------------------------------------------------
|
| The maximum number of batches that can be run concurrently. For example,
| if 1000 queue jobs are dispatched and this is set to 5 with a batch size
| of 100, then 5 batches of 100 will be run concurrently and take two runs
| of the queue to complete.
|
*/
'max_concurrent_batches' => environment( 'QUEUE_MAX_CONCURRENT_BATCHES', 1 ),

/*
|--------------------------------------------------------------------------
| Delete failed or processed queue items after a set time
|--------------------------------------------------------------------------
|
| Delete failed or processed queue items after a set time in seconds.
|
*/
'delete_after' => environment( 'QUEUE_DELETE_AFTER', 60 * 60 * 24 * 7 ),

/*
|--------------------------------------------------------------------------
| Enable the Queue Admin Interface
|--------------------------------------------------------------------------
|
| Enable the queue admin interface to display queue jobs.
|
*/
'enable_admin' => environment( 'QUEUE_ENABLE_ADMIN', true ),

/*
|--------------------------------------------------------------------------
Expand All @@ -35,7 +68,7 @@
| Control the configuration for the queue providers.
|
*/
'wordpress' => [
'wordpress' => [
// Delay between queue runs in seconds.
'delay' => environment( 'QUEUE_DELAY', 0 ),
],
Expand Down
2 changes: 1 addition & 1 deletion src/mantle/application/class-app-service-provider.php
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ public function __construct( Application $app ) {
*/
protected function boot_scheduler() {
$this->app->singleton(
Schedule::class,
'scheduler',
fn ( $app ) => tap(
new Schedule( $app ),
fn ( Schedule $schedule ) => $this->schedule( $schedule ),
Expand Down
31 changes: 17 additions & 14 deletions src/mantle/application/class-application.php
Original file line number Diff line number Diff line change
Expand Up @@ -401,20 +401,23 @@ protected function register_base_bindings() {
*/
protected function register_core_aliases() {
$core_aliases = [
'app' => [ static::class, \Mantle\Contracts\Application::class ],
'config' => [ \Mantle\Config\Repository::class, \Mantle\Contracts\Config\Repository::class ],
'events' => [ \Mantle\Events\Dispatcher::class, \Mantle\Contracts\Events\Dispatcher::class ],
'files' => [ \Mantle\Filesystem\Filesystem::class ],
'filesystem' => [ \Mantle\Filesystem\Filesystem_Manager::class, \Mantle\Contracts\Filesystem\Filesystem_Manager::class ],
'log' => [ \Mantle\Log\Log_Manager::class, \Psr\Log\LoggerInterface::class ],
'queue' => [ \Mantle\Queue\Queue_Manager::class, \Mantle\Contracts\Queue\Queue_Manager::class ],
'redirect' => [ \Mantle\Http\Routing\Redirector::class ],
'request' => [ \Mantle\Http\Request::class, \Symfony\Component\HttpFoundation\Request::class ],
'router' => [ \Mantle\Http\Routing\Router::class, \Mantle\Contracts\Http\Routing\Router::class ],
'router.entity' => [ \Mantle\Http\Routing\Entity_Router::class, \Mantle\Contracts\Http\Routing\Entity_Router::class ],
'url' => [ \Mantle\Http\Routing\Url_Generator::class, \Mantle\Contracts\Http\Routing\Url_Generator::class ],
'view.loader' => [ \Mantle\Http\View\View_Finder::class, \Mantle\Contracts\Http\View\View_Finder::class ],
'view' => [ \Mantle\Http\View\Factory::class, \Mantle\Contracts\Http\View\Factory::class ],
'app' => [ static::class, \Mantle\Contracts\Application::class ],
'config' => [ \Mantle\Config\Repository::class, \Mantle\Contracts\Config\Repository::class ],
'events' => [ \Mantle\Events\Dispatcher::class, \Mantle\Contracts\Events\Dispatcher::class ],
'files' => [ \Mantle\Filesystem\Filesystem::class ],
'filesystem' => [ \Mantle\Filesystem\Filesystem_Manager::class, \Mantle\Contracts\Filesystem\Filesystem_Manager::class ],
'log' => [ \Mantle\Log\Log_Manager::class, \Psr\Log\LoggerInterface::class ],
'queue' => [ \Mantle\Queue\Queue_Manager::class, \Mantle\Contracts\Queue\Queue_Manager::class ],
'queue.worker' => [ \Mantle\Queue\Worker::class ],
'queue.dispatcher' => [ \Mantle\Queue\Dispatcher::class, \Mantle\Contracts\Queue\Dispatcher::class ],
'redirect' => [ \Mantle\Http\Routing\Redirector::class ],
'request' => [ \Mantle\Http\Request::class, \Symfony\Component\HttpFoundation\Request::class ],
'router' => [ \Mantle\Http\Routing\Router::class, \Mantle\Contracts\Http\Routing\Router::class ],
'router.entity' => [ \Mantle\Http\Routing\Entity_Router::class, \Mantle\Contracts\Http\Routing\Entity_Router::class ],
'scheduler' => [ \Mantle\Scheduling\Schedule::class ],
'url' => [ \Mantle\Http\Routing\Url_Generator::class, \Mantle\Contracts\Http\Routing\Url_Generator::class ],
'view.loader' => [ \Mantle\Http\View\View_Finder::class, \Mantle\Contracts\Http\View\View_Finder::class ],
'view' => [ \Mantle\Http\View\Factory::class, \Mantle\Contracts\Http\View\Factory::class ],
];

foreach ( $core_aliases as $key => $aliases ) {
Expand Down
12 changes: 10 additions & 2 deletions src/mantle/contracts/queue/interface-provider.php
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ interface Provider {
* @param mixed $job Job instance.
* @return bool
*/
public function push( $job );
public function push( $job ): bool;

/**
* Get the next set of jobs in the queue.
Expand All @@ -37,5 +37,13 @@ public function pop( string $queue = null, int $count = 1 ): Collection;
* @param string $queue Queue to compare against.
* @return bool
*/
public function in_queue( $job, string $queue = null ): bool;
public function in_queue( mixed $job, string $queue = null ): bool;

/**
* Retrieve the number of pending jobs in the queue.
*
* @param string $queue Queue name, optional.
* @return int
*/
public function pending_count( string $queue = null ): int;
}
2 changes: 2 additions & 0 deletions src/mantle/database/model/class-model.php
Original file line number Diff line number Diff line change
Expand Up @@ -175,12 +175,14 @@ public function refresh() {
}

$instance = static::find( $this->get( 'id' ) );

if ( ! $instance ) {
return null;
}

$this->exists = true;
$this->set_raw_attributes( $instance->get_raw_attributes() );

return $this;
}

Expand Down
10 changes: 5 additions & 5 deletions src/mantle/database/query/class-builder.php
Original file line number Diff line number Diff line change
Expand Up @@ -859,8 +859,8 @@ public function chunk_by_id( int $count, callable $callback, string $attribute =
/**
* Execute a callback over each item while chunking.
*
* @param callable(\Mantle\Support\Collection<int, TModel>): mixed $callback Callback to run on each chunk.
* @param int $count Number of items to chunk by.
* @param callable(TModel): mixed $callback Callback to run on each chunk.
* @param int $count Number of items to chunk by.
* @return boolean
*/
public function each( callable $callback, int $count = 100 ) {
Expand All @@ -878,9 +878,9 @@ public function each( callable $callback, int $count = 100 ) {
/**
* Execute a callback over each item while chunking by ID.
*
* @param callable(\Mantle\Support\Collection<int, TModel>): mixed $callback Callback to run on each chunk.
* @param int $count Number of items to chunk by.
* @param string $attribute Attribute to chunk by.
* @param callable(TModel): mixed $callback Callback to run on each chunk.
* @param int $count Number of items to chunk by.
* @param string $attribute Attribute to chunk by.
* @return boolean
*/
public function each_by_id( callable $callback, int $count = 100, string $attribute = 'id' ) {
Expand Down
3 changes: 2 additions & 1 deletion src/mantle/database/query/class-post-query-builder.php
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
* @method \Mantle\Database\Query\Post_Query_Builder<TModel> whereId( int $id )
* @method \Mantle\Database\Query\Post_Query_Builder<TModel> whereName( string $name )
* @method \Mantle\Database\Query\Post_Query_Builder<TModel> whereSlug( string $slug )
* @method \Mantle\Database\Query\Post_Query_Builder<TModel> whereStatus( string $status )
* @method \Mantle\Database\Query\Post_Query_Builder<TModel> whereStatus( string[]|string $status )
* @method \Mantle\Database\Query\Post_Query_Builder<TModel> whereTitle( string $title )
* @method \Mantle\Database\Query\Post_Query_Builder<TModel> whereType( string $type )
*/
Expand All @@ -49,6 +49,7 @@ class Post_Query_Builder extends Builder {
'post_author' => 'author',
'post_name' => 'name',
'slug' => 'name',
'status' => 'post_status',
];

/**
Expand Down
9 changes: 7 additions & 2 deletions src/mantle/queue/autoload.php
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,15 @@
/**
* Dispatch a job to the queue.
*
* @template TJob of \Mantle\Contracts\Queue\Job|\Closure
*
* @param \Mantle\Contracts\Queue\Job|\Closure $job Job instance.
* @return Pending_Dispatch|Pending_Closure_Dispatch
* @return Pending_Closure_Dispatch|Pending_Dispatch
*
* @phpstan-param TJob|\Mantle\Contracts\Queue\Job|\Closure|\Closure $job Job instance.
* @phpstan-return (TJob is \Closure ? Pending_Closure_Dispatch : Pending_Dispatch)<TJob>
*/
function dispatch( $job ): Pending_Dispatch {
function dispatch( $job ): Pending_Dispatch|Pending_Closure_Dispatch {
return $job instanceof \Closure
? new Pending_Closure_Dispatch( Closure_Job::create( $job ) )
: new Pending_Dispatch( $job );
Expand Down
30 changes: 22 additions & 8 deletions src/mantle/queue/class-closure-job.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,36 +2,39 @@
/**
* Closure_Job class file
*
* phpcs:disable Squiz.Commenting.VariableComment.Missing
*
* @package Mantle
*/

namespace Mantle\Queue;

use Closure;
use DateTimeInterface;
use Laravel\SerializableClosure\SerializableClosure;
use Mantle\Contracts\Queue\Can_Queue;
use ReflectionFunction;
use Throwable;

/**
* Abstract Queue Job
* Closure Job
*
* To be extended by provider-specific queue job classes.
* Storage of the closure-based queue job.
*/
class Closure_Job implements Can_Queue {
/**
* Serializable closure instance.
* The delay before the job will be run.
*
* @var SerializableClosure
* @var int|DateTimeInterface
*/
public SerializableClosure $closure;
public int|DateTimeInterface $delay;

/**
* The callbacks that should be run on failure.
*
* @var array
*/
public $failure_callbacks = [];
public array $failure_callbacks = [];

/**
* Create a new job instance.
Expand All @@ -48,8 +51,7 @@ public static function create( Closure $closure ): Closure_Job {
*
* @param SerializableClosure $closure Serialized closure to wrap.
*/
public function __construct( SerializableClosure $closure ) {
$this->closure = $closure;
public function __construct( public SerializableClosure $closure ) {
}

/**
Expand All @@ -61,6 +63,18 @@ public function handle() {
$callback();
}

/**
* Set the delay before the job will be run.
*
* @param DateTimeInterface|int $delay Delay in seconds or DateTime instance.
* @return static
*/
public function delay( DateTimeInterface|int $delay ) {
$this->delay = $delay;

return $this;
}

/**
* Add a callback to be executed if the job fails.
*
Expand Down
40 changes: 22 additions & 18 deletions src/mantle/queue/class-dispatcher.php
Original file line number Diff line number Diff line change
Expand Up @@ -11,54 +11,58 @@
use Mantle\Contracts\Container;
use Mantle\Contracts\Queue\Can_Queue;
use Mantle\Contracts\Queue\Queue_Manager;
use Mantle\Queue\Events\Job_Queued;

/**
* Queue Dispatcher
*
* Executes jobs from the queue.
*/
class Dispatcher {
/**
* Container instance.
*
* @var Container
*/
protected $container;

/**
* Constructor.
*
* @param Container $container Container instance.
*/
public function __construct( Container $container ) {
$this->container = $container;
}
public function __construct( protected Container $container ) {}

/**
* Dispatch the job to the queue.
*
* @param mixed $job Job instance.
* @return mixed
* @return void
*/
public function dispatch( $job ) {
public function dispatch( mixed $job ): void {
if ( ! $this->should_command_be_queued( $job ) ) {
return $this->dispatch_now( $job );
$this->dispatch_now( $job );

return;
}

$manager = $this->container->make( Queue_Manager::class );
/**
* Provider instance.
*
* @var \Mantle\Contracts\Queue\Provider
*/
$provider = $this->container->make( Queue_Manager::class )->get_provider();

// Send the job to the queue.
$manager->get_provider()->push( $job );
$provider->push( $job );

// Dispatch the job queued event.
$this->container['events']->dispatch(
new Job_Queued( $provider, $job ),
);
}

/**
* Dispatch a job in the current process.
*
* @param mixed $job Job instance.
* @return mixed
* @return void
*/
public function dispatch_now( $job ) {
return $this->container->call( [ $job, 'handle' ] );
public function dispatch_now( mixed $job ): void {
$this->container->call( [ $job, 'handle' ] );
}

/**
Expand Down
Loading