Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[1.x] Queue Overhaul #456

Merged
merged 43 commits into from
Dec 1, 2023
Merged

[1.x] Queue Overhaul #456

merged 43 commits into from
Dec 1, 2023

Conversation

srtfisher
Copy link
Member

@srtfisher srtfisher commented Sep 22, 2023 •
edited
Loading

Overhauling the queue to make it more performant.

  • Adds locking, job failures, delaying, and job retrying.
  • Refactor to dogfood the Mantle models from within the queue provider.
  • Supports concurrent jobs with configurable limits. For example, if 100 jobs are dispatched and 10 concurrent batches can be run, 10 cron events will be dispatched to handle the queued jobs.
  • Built with PHP 8.1 in mind (enums) and is slated for a 1.0 release feature. The queue package will require 8.1 but the rest of the framework still supports 8.0.

Remaining:

srtfisher added 22 commits August 31, 2023 16:07
@srtfisher
Slight refactor for readability
14c1fde
@srtfisher
Refactor to queue model
b66bb65
@srtfisher
Use the query builder internally
b82ed9a
@srtfisher
WIP
464245a
@srtfisher
Add phpstan typings
7d623aa
@srtfisher
Continued work
499a9e8
@srtfisher
Starting out on count()
087f4d6
@srtfisher
Adding query builder count() method
af4cd98
@srtfisher
Merge branch 'query-count' into queue-performance
96866ce
@srtfisher
Todo for tomorrow
434167c
@srtfisher
Set the meta when creating the job
7959eec
@srtfisher
Improving the logic
2b99103
@srtfisher
Wrapping up until tmw
cbfa715
@srtfisher
Merge remote-tracking branch 'origin/0.12.x' into queue-performance
a1a071b
@srtfisher
Continued work
68ed908
@srtfisher
Merge branch 'date-query' into queue-performance
d709c5e
@srtfisher
Use the date query builder to get the proper queue items
09a7677
@srtfisher
Merging in date-query
eb7de25
@srtfisher
Adjust cron logic to properly scale up to meet demand of the queue
48b6094
@srtfisher
Adding tests for cron queue
92a5d3c
@srtfisher
Linting fixes
8e76bbc
@srtfisher
Wrapping up fixes for queue
964a2f7
@srtfisher srtfisher marked this pull request as draft September 22, 2023 19:29
@srtfisher srtfisher changed the base branch from 0.12.x to 1.x September 22, 2023 19:32
@srtfisher srtfisher mentioned this pull request Sep 27, 2023
Merged
@coderabbitai coderabbitai
Copy link

coderabbitai bot commented Nov 14, 2023 •
edited
Loading

Important

Auto Review Skipped

Auto reviews are disabled on base/target branches other than the default branch. Please add the base/target branch pattern to the list of additional branches to be reviewed in the settings.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Tips

Chat with CodeRabbit Bot (@coderabbitai)

  • If you reply to a review comment from CodeRabbit, the bot will automatically respond.
  • To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment
  • Note: Review comments are made on code diffs or files, not on the PR overview.
  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Note: For conversation with the bot, please use the review comments on code diffs or files.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

@srtfisher srtfisher mentioned this pull request Nov 16, 2023
Merged
@srtfisher srtfisher changed the title Queue Performance Improvements [1.x] Queue Overhaul Nov 27, 2023
@srtfisher
Queue UI (#458)
c4b945b 
* Continued work on queue UI=

* Wrapping up the core of the queue UI

* Working table for queue UI

* Adding counts to the queue

* Switch to use wp_count_posts for performance

* Style the row table

* Adding single view for queue job

* Adding queue UI with fixes for retrying
@srtfisher
Preserve queue job items and cleanup after some time (#472)
480ece0 
* Continued work on queue UI=

* Wrapping up the core of the queue UI

* Working table for queue UI

* Adding counts to the queue

* Switch to use wp_count_posts for performance

* Style the row table

* Adding single view for queue job

* Adding queue UI with fixes for retrying

* Serialize enums when asserting against them

* Keep queue job around after it is complete for logging

* Fixing tests and adjusting comment

* Fixing linting

* Stubbing the cleanup command

* Cleanup the processed/failed jobs after some time

* Fixing phpcs issue
@srtfisher
Merge branch '1.x' into queue-performance
9100de2
@srtfisher
Remove checks for <8.1 now that were on PHP 8.1+ always
0816b89
@srtfisher
[1.x] Allow queue jobs to be run in-admin, allow delays for closure j…
eb2f897 
…obs (#481)

* Add support for displaying completed queue job status

* Allow a closure job to be delayed

* Allow queue jobs to be run in-admin, allow closure jobs to be delayed

* Add a back link on the queue job page

* Cleanup running jobs with the cleanup command

* Remove the superfluous take()
@srtfisher srtfisher marked this pull request as ready for review December 1, 2023 15:10
@srtfisher
Copy link
Member Author

Reviewed in subsequent pull requests.

@srtfisher srtfisher merged commit 2e60851 into 1.x Dec 1, 2023
11 checks passed
@srtfisher srtfisher deleted the queue-performance branch December 1, 2023 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@srtfisher