Skip to content

Commit

Permalink
feat(queue, worker): resolve conflict
Browse files Browse the repository at this point in the history
  • Loading branch information
fgozdz committed Sep 4, 2024
2 parents 2e3fe37 + 69c1296 commit c4cb517
Show file tree
Hide file tree
Showing 22 changed files with 509 additions and 256 deletions.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2018 taskforcesh
Copyright (c) 2018 BullForce Labs AB.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
50 changes: 20 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,16 +75,6 @@ Sign up at [Taskforce.sh](https://taskforce.sh)
better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more <a href="https://www.dragonflydb.io/docs/integrations/bullmq">here</a> on how to use it with BullMQ.
</td>
</tr>
<tr>
<td>
<a href="https://dashboard.memetria.com/new?utm_campaign=BULLMQ">
<img src="https://www.memetria.com/images/logo/memetria-logo.svg" width=350 alt="Memetria for Redis" />
</a>
</td>
<td>
If you need high quality production Redis instances for your BullMQ project, please consider subscribing to <a href="https://dashboard.memetria.com/new?utm_campaign=BULLMQ">Memetria for Redis</a>, leaders in Redis hosting that works perfectly with BullMQ. Use the promo code "BULLMQ" when signing up to help us sponsor the development of BullMQ!
</td>
</tr>
</table>

# Used by
Expand Down Expand Up @@ -230,26 +220,26 @@ This is just scratching the surface, check all the features and more in the offi

Since there are a few job queue solutions, here is a table comparing them:

| Feature | [BullMQ-Pro](https://bullmq.io/#bullmq-pro) | [BullMQ](https://bullmq.io) | Bull | Kue | Bee | Agenda |
| :------------------------ | :-------------: | :-------------: | :-------------: | :---: | -------- | ------ |
| Backend | redis | redis | redis | redis | redis | mongo |
| Observables || | | | | |
| Group Rate Limit || | | | | |
| Group Support || | | | | |
| Batches Support || | | | | |
| Parent/Child Dependencies || | | | | |
| Priorities || ||| ||
| Concurrency || |||||
| Delayed jobs || ||| ||
| Global events || ||| | |
| Rate Limiter || || | | |
| Pause/Resume || ||| | |
| Sandboxed worker || || | | |
| Repeatable jobs || || | ||
| Atomic ops || || || |
| Persistence || |||||
| UI || ||| ||
| Optimized for | Jobs / Messages | Jobs / Messages | Jobs / Messages | Jobs | Messages | Jobs |
| Feature | [BullMQ-Pro](https://bullmq.io/#bullmq-pro) | [BullMQ](https://bullmq.io) | Bull | Kue | Bee | Agenda |
| :------------------------ | :-----------------------------------------: | :-------------------------: | :-------------: | :---: | -------- | ------ |
| Backend | redis | redis | redis | redis | redis | mongo |
| Observables | | | | | | |
| Group Rate Limit | | | | | | |
| Group Support | | | | | | |
| Batches Support | | | | | | |
| Parent/Child Dependencies | | | | | | |
| Priorities | | ||| ||
| Concurrency | | |||||
| Delayed jobs | | ||| ||
| Global events | | ||| | |
| Rate Limiter | | || | | |
| Pause/Resume | | ||| | |
| Sandboxed worker | | || | | |
| Repeatable jobs | | || | ||
| Atomic ops | | || || |
| Persistence | | |||||
| UI | | ||| ||
| Optimized for | Jobs / Messages | Jobs / Messages | Jobs / Messages | Jobs | Messages | Jobs |

## Contributing

Expand Down
221 changes: 111 additions & 110 deletions docs/gitbook/SUMMARY.md
Original file line number Diff line number Diff line change
@@ -1,130 +1,131 @@
# Table of contents

* [What is BullMQ](README.md)
* [Quick Start](<README (1).md>)
* [API Reference](https://api.docs.bullmq.io)
* [Changelogs](changelog.md)
* [v4](changelogs/changelog-v4.md)
* [v3](changelogs/changelog-v3.md)
* [v2](changelogs/changelog-v2.md)
* [v1](changelogs/changelog-v1.md)
- [What is BullMQ](README.md)
- [Quick Start](<README (1).md>)
- [API Reference](https://api.docs.bullmq.io)
- [Changelogs](changelog.md)
- [v4](changelogs/changelog-v4.md)
- [v3](changelogs/changelog-v3.md)
- [v2](changelogs/changelog-v2.md)
- [v1](changelogs/changelog-v1.md)

## Guide

* [Introduction](guide/introduction.md)
* [Connections](guide/connections.md)
* [Queues](guide/queues/README.md)
* [Auto-removal of jobs](guide/queues/auto-removal-of-jobs.md)
* [Adding jobs in bulk](guide/queues/adding-bulks.md)
* [Global Concurrency](guide/queues/global-concurrency.md)
* [Removing Jobs](guide/queues/removing-jobs.md)
* [Workers](guide/workers/README.md)
* [Auto-removal of jobs](guide/workers/auto-removal-of-jobs.md)
* [Concurrency](guide/workers/concurrency.md)
* [Graceful shutdown](guide/workers/graceful-shutdown.md)
* [Stalled Jobs](guide/workers/stalled-jobs.md)
* [Sandboxed processors](guide/workers/sandboxed-processors.md)
* [Pausing queues](guide/workers/pausing-queues.md)
* [Jobs](guide/jobs/README.md)
* [FIFO](guide/jobs/fifo.md)
* [LIFO](guide/jobs/lifo.md)
* [Job Ids](guide/jobs/job-ids.md)
* [Job Data](guide/jobs/job-data.md)
* [Debouncing](guide/jobs/debouncing.md)
* [Delayed](guide/jobs/delayed.md)
* [Repeatable](guide/jobs/repeatable.md)
* [Prioritized](guide/jobs/prioritized.md)
* [Removing jobs](guide/jobs/removing-job.md)
* [Stalled](guide/jobs/stalled.md)
* [Getters](guide/jobs/getters.md)
* [Flows](guide/flows/README.md)
* [Adding flows in bulk](guide/flows/adding-bulks.md)
* [Get Flow Tree](guide/flows/get-flow-tree.md)
* [Fail Parent](guide/flows/fail-parent.md)
* [Remove Dependency](guide/flows/remove-dependency.md)
* [Ignore Dependency](guide/flows/ignore-dependency.md)
* [Remove Child Dependency](guide/flows/remove-child-dependency.md)
* [Metrics](guide/metrics/metrics.md)
* [Rate limiting](guide/rate-limiting.md)
* [Parallelism and Concurrency](guide/parallelism-and-concurrency.md)
* [Retrying failing jobs](guide/retrying-failing-jobs.md)
* [Returning job data](guide/returning-job-data.md)
* [Events](guide/events.md)
* [QueueScheduler](guide/queuescheduler.md)
* [Redis™ Compatibility](guide/redis-tm-compatibility/README.md)
* [Dragonfly](guide/redis-tm-compatibility/dragonfly.md)
* [Redis™ hosting](guide/redis-tm-hosting/README.md)
* [AWS MemoryDB](guide/redis-tm-hosting/aws-memorydb.md)
* [AWS Elasticache](guide/redis-tm-hosting/aws-elasticache.md)
* [Architecture](guide/architecture.md)
* [NestJs](guide/nestjs/README.md)
* [Producers](guide/nestjs/producers.md)
* [Queue Events Listeners](guide/nestjs/queue-events-listeners.md)
* [Going to production](guide/going-to-production.md)
* [Migration to newer versions](guide/migration-to-newer-versions.md)
* [Troubleshooting](guide/troubleshooting.md)
- [Introduction](guide/introduction.md)
- [Connections](guide/connections.md)
- [Queues](guide/queues/README.md)
- [Auto-removal of jobs](guide/queues/auto-removal-of-jobs.md)
- [Adding jobs in bulk](guide/queues/adding-bulks.md)
- [Global Concurrency](guide/queues/global-concurrency.md)
- [Removing Jobs](guide/queues/removing-jobs.md)
- [Workers](guide/workers/README.md)
- [Auto-removal of jobs](guide/workers/auto-removal-of-jobs.md)
- [Concurrency](guide/workers/concurrency.md)
- [Graceful shutdown](guide/workers/graceful-shutdown.md)
- [Stalled Jobs](guide/workers/stalled-jobs.md)
- [Sandboxed processors](guide/workers/sandboxed-processors.md)
- [Pausing queues](guide/workers/pausing-queues.md)
- [Jobs](guide/jobs/README.md)
- [FIFO](guide/jobs/fifo.md)
- [LIFO](guide/jobs/lifo.md)
- [Job Ids](guide/jobs/job-ids.md)
- [Job Data](guide/jobs/job-data.md)
- [Debouncing](guide/jobs/debouncing.md)
- [Delayed](guide/jobs/delayed.md)
- [Repeatable](guide/jobs/repeatable.md)
- [Prioritized](guide/jobs/prioritized.md)
- [Removing jobs](guide/jobs/removing-job.md)
- [Stalled](guide/jobs/stalled.md)
- [Getters](guide/jobs/getters.md)
- [Flows](guide/flows/README.md)
- [Adding flows in bulk](guide/flows/adding-bulks.md)
- [Get Flow Tree](guide/flows/get-flow-tree.md)
- [Fail Parent](guide/flows/fail-parent.md)
- [Remove Dependency](guide/flows/remove-dependency.md)
- [Ignore Dependency](guide/flows/ignore-dependency.md)
- [Remove Child Dependency](guide/flows/remove-child-dependency.md)
- [Metrics](guide/metrics/metrics.md)
- [Rate limiting](guide/rate-limiting.md)
- [Parallelism and Concurrency](guide/parallelism-and-concurrency.md)
- [Retrying failing jobs](guide/retrying-failing-jobs.md)
- [Returning job data](guide/returning-job-data.md)
- [Events](guide/events.md)
- [QueueScheduler](guide/queuescheduler.md)
- [Redis™ Compatibility](guide/redis-tm-compatibility/README.md)
- [Dragonfly](guide/redis-tm-compatibility/dragonfly.md)
- [Redis™ hosting](guide/redis-tm-hosting/README.md)
- [AWS MemoryDB](guide/redis-tm-hosting/aws-memorydb.md)
- [AWS Elasticache](guide/redis-tm-hosting/aws-elasticache.md)
- [Architecture](guide/architecture.md)
- [NestJs](guide/nestjs/README.md)
- [Producers](guide/nestjs/producers.md)
- [Queue Events Listeners](guide/nestjs/queue-events-listeners.md)
- [Going to production](guide/going-to-production.md)
- [Migration to newer versions](guide/migration-to-newer-versions.md)
- [Troubleshooting](guide/troubleshooting.md)

## Patterns

* [Adding jobs in bulk across different queues](patterns/adding-bulks.md)
* [Manually processing jobs](patterns/manually-fetching-jobs.md)
* [Named Processor](patterns/named-processor.md)
* [Flows](patterns/flows.md)
* [Idempotent jobs](patterns/idempotent-jobs.md)
* [Throttle jobs](patterns/throttle-jobs.md)
* [Process Step Jobs](patterns/process-step-jobs.md)
* [Failing fast when Redis is down](patterns/failing-fast-when-redis-is-down.md)
* [Stop retrying jobs](patterns/stop-retrying-jobs.md)
* [Redis Cluster](patterns/redis-cluster.md)
- [Adding jobs in bulk across different queues](patterns/adding-bulks.md)
- [Manually processing jobs](patterns/manually-fetching-jobs.md)
- [Named Processor](patterns/named-processor.md)
- [Flows](patterns/flows.md)
- [Idempotent jobs](patterns/idempotent-jobs.md)
- [Throttle jobs](patterns/throttle-jobs.md)
- [Process Step Jobs](patterns/process-step-jobs.md)
- [Failing fast when Redis is down](patterns/failing-fast-when-redis-is-down.md)
- [Stop retrying jobs](patterns/stop-retrying-jobs.md)
- [Timeout jobs](patterns/timeout-jobs.md)
- [Redis Cluster](patterns/redis-cluster.md)

## BullMQ Pro

* [Introduction](bullmq-pro/introduction.md)
* [Install](bullmq-pro/install.md)
* [Observables](bullmq-pro/observables/README.md)
* [Cancelation](bullmq-pro/observables/cancelation.md)
* [Groups](bullmq-pro/groups/README.md)
* [Getters](bullmq-pro/groups/getters.md)
* [Rate limiting](bullmq-pro/groups/rate-limiting.md)
* [Concurrency](bullmq-pro/groups/concurrency.md)
* [Local group concurrency](bullmq-pro/groups/local-group-concurrency.md)
* [Max group size](bullmq-pro/groups/max-group-size.md)
* [Pausing groups](bullmq-pro/groups/pausing-groups.md)
* [Prioritized intra-groups](bullmq-pro/groups/prioritized.md)
* [Sandboxes for groups](bullmq-pro/groups/sandboxes-for-groups.md)
* [Batches](bullmq-pro/batches.md)
* [NestJs](bullmq-pro/nestjs/README.md)
* [Producers](bullmq-pro/nestjs/producers.md)
* [Queue Events Listeners](bullmq-pro/nestjs/queue-events-listeners.md)
* [API Reference](https://nestjs.bullmq.pro/)
* [Changelog](bullmq-pro/nestjs/changelog.md)
* [API Reference](https://api.bullmq.pro)
* [Changelog](bullmq-pro/changelog.md)
* [Support](bullmq-pro/support.md)
- [Introduction](bullmq-pro/introduction.md)
- [Install](bullmq-pro/install.md)
- [Observables](bullmq-pro/observables/README.md)
- [Cancelation](bullmq-pro/observables/cancelation.md)
- [Groups](bullmq-pro/groups/README.md)
- [Getters](bullmq-pro/groups/getters.md)
- [Rate limiting](bullmq-pro/groups/rate-limiting.md)
- [Concurrency](bullmq-pro/groups/concurrency.md)
- [Local group concurrency](bullmq-pro/groups/local-group-concurrency.md)
- [Max group size](bullmq-pro/groups/max-group-size.md)
- [Pausing groups](bullmq-pro/groups/pausing-groups.md)
- [Prioritized intra-groups](bullmq-pro/groups/prioritized.md)
- [Sandboxes for groups](bullmq-pro/groups/sandboxes-for-groups.md)
- [Batches](bullmq-pro/batches.md)
- [NestJs](bullmq-pro/nestjs/README.md)
- [Producers](bullmq-pro/nestjs/producers.md)
- [Queue Events Listeners](bullmq-pro/nestjs/queue-events-listeners.md)
- [API Reference](https://nestjs.bullmq.pro/)
- [Changelog](bullmq-pro/nestjs/changelog.md)
- [API Reference](https://api.bullmq.pro)
- [Changelog](bullmq-pro/changelog.md)
- [Support](bullmq-pro/support.md)

## Bull

* [Introduction](bull/introduction.md)
* [Install](bull/install.md)
* [Quick Guide](bull/quick-guide.md)
* [Important Notes](bull/important-notes.md)
* [Reference](https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md)
* [Patterns](bull/patterns/README.md)
* [Persistent connections](bull/patterns/persistent-connections.md)
* [Message queue](bull/patterns/message-queue.md)
* [Returning Job Completions](bull/patterns/returning-job-completions.md)
* [Reusing Redis Connections](bull/patterns/reusing-redis-connections.md)
* [Redis cluster](bull/patterns/redis-cluster.md)
* [Custom backoff strategy](bull/patterns/custom-backoff-strategy.md)
* [Debugging](bull/patterns/debugging.md)
* [Manually fetching jobs](bull/patterns/manually-fetching-jobs.md)
- [Introduction](bull/introduction.md)
- [Install](bull/install.md)
- [Quick Guide](bull/quick-guide.md)
- [Important Notes](bull/important-notes.md)
- [Reference](https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md)
- [Patterns](bull/patterns/README.md)
- [Persistent connections](bull/patterns/persistent-connections.md)
- [Message queue](bull/patterns/message-queue.md)
- [Returning Job Completions](bull/patterns/returning-job-completions.md)
- [Reusing Redis Connections](bull/patterns/reusing-redis-connections.md)
- [Redis cluster](bull/patterns/redis-cluster.md)
- [Custom backoff strategy](bull/patterns/custom-backoff-strategy.md)
- [Debugging](bull/patterns/debugging.md)
- [Manually fetching jobs](bull/patterns/manually-fetching-jobs.md)

## Bull 3.x Migration

* [Compatibility class](bull-3.x-migration/compatibility-class.md)
* [Migration](bull-3.x-migration/migration.md)
- [Compatibility class](bull-3.x-migration/compatibility-class.md)
- [Migration](bull-3.x-migration/migration.md)

## Python

* [Introduction](python/introduction.md)
* [Changelog](python/changelog.md)
- [Introduction](python/introduction.md)
- [Changelog](python/changelog.md)
15 changes: 14 additions & 1 deletion docs/gitbook/changelog.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
## [5.12.13](https://github.com/taskforcesh/bullmq/compare/v5.12.12...v5.12.13) (2024-09-03)


### Bug Fixes

* **repeat:** replace delayed job when updating repeat key ([88029bb](https://github.com/taskforcesh/bullmq/commit/88029bbeab2a58768f9c438318f540010cd286a7))

## [5.12.12](https://github.com/taskforcesh/bullmq/compare/v5.12.11...v5.12.12) (2024-08-29)


### Bug Fixes

* **flows:** throw error when queueName contains colon ([#2719](https://github.com/taskforcesh/bullmq/issues/2719)) fixes [#2718](https://github.com/taskforcesh/bullmq/issues/2718) ([9ef97c3](https://github.com/taskforcesh/bullmq/commit/9ef97c37663e209f03c501a357b6b1a662b24d99))

## [5.12.11](https://github.com/taskforcesh/bullmq/compare/v5.12.10...v5.12.11) (2024-08-28)


Expand Down Expand Up @@ -75,7 +89,6 @@
### Bug Fixes

* **job:** consider passing stackTraceLimit as 0 ([#2692](https://github.com/taskforcesh/bullmq/issues/2692)) ref [#2487](https://github.com/taskforcesh/bullmq/issues/2487) ([509a36b](https://github.com/taskforcesh/bullmq/commit/509a36baf8d8cf37176e406fd28e33f712229d27))
* **job:** make sure json.dumps return JSON compliant JSON [python] ([#2683](https://github.com/taskforcesh/bullmq/issues/2683)) ([4441711](https://github.com/taskforcesh/bullmq/commit/4441711a986a9f6a326100308d639eb0a2ea8c8d))

# [5.12.0](https://github.com/taskforcesh/bullmq/compare/v5.11.0...v5.12.0) (2024-08-01)

Expand Down
Empty file removed docs/gitbook/do-not-commit-.md
Empty file.
4 changes: 4 additions & 0 deletions docs/gitbook/guide/flows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ The above call will return instances for all the jobs added to the queue.
Note that the parent queue does not need to be the same queue as the one used for the children.
{% endhint %}

{% hint style="warning" %}
If a jobId option is provided, make that it does not contain a colon **:** as this is considered a separator.
{% endhint %}

When the parent job is processed it is possible to access the results generated by its child jobs. For example, lets assume the following worker for the child jobs:

{% tabs %}
Expand Down
Loading

0 comments on commit c4cb517

Please sign in to comment.