Need documentation

[jcoyne]

I have several questions about Qyu and I suspect your documentation could be improved so that others don't raise the same questions.

Workers all must be able to connect to the single shared state store. They do this to retrieve job information needed for processing and to determine what state to move a job to when a task is complete. Is that right? Could this be a bottleneck? How do I change the workflow? What happens to jobs that are already in process when I change a workflow? How do I ensure all workers get the changes? Do they need to be restarted, or will they automatically pick them up?

What is the relationship of tasks to queues? Should each task have it's own queue or is it acceptable to use a single queue for all tasks? Can a worker service multiple queues? Is it up to the worker to determine what task type it is being asked to execute? What is the best way to do that?

How should exceptions be handled? Does Qyu provide any facility for retrying jobs? Is there any logging facilities in Qyu?

Do long running jobs ever timeout? Is the timeout configurable? What happens to jobs that timeout?

[oss92]

Workers all must be able to connect to the single shared state store. They do this to retrieve job information needed for processing and to determine what state to move a job to when a task is complete. Is that right? Could this be a bottleneck?

• Jobs do not have a state themselves, they have the initial payload and reference a workflow. The message queue contains the task ids. Each task has a status and Qyu does three actions: create, get, lock and update task statuses. It all depends on a lot of factors whether it can be a bottleneck.
• Also anyone has the freedom to implement custom qyu-store-* with their desired database, api, etc.. overriding the provided methods (ex: github.com/QyuTeam/qyu-store-activerecord) if they ever face a bottleneck.

How do I change the workflow?

• A workflow can be updated by updating the descriptor. Running tasks read the workflow every time they finish execution so if the workflow is changed runtime a task will create its subsequent tasks or conclude depending on the new workflow.

What is the relationship of tasks to queues? Should each task have it's own queue or is it acceptable to use a single queue for all tasks?

• Every worker is listening on a queue, that means, all tasks IDs in this queue will execute the same exact code therefore every different task must have their own queue.

Can a worker service multiple queues?

• Not yet, Qyu was designed to support an autoscaling microservice infrastructure, very specialized micro-applications and small code size. That said, it may allow multiple queues in a future update in case two flows differ really slightly or if some developers prefer more monolithic apps.

Is it up to the worker to determine what task type it is being asked to execute?

• The workers received a task ID, fetches its payload and runs the passed blocked of code in it so yes, according to the queue it listens to. Check out the examples at https://github.com/QyuTeam/qyu/tree/master/examples/simple and https://github.com/QyuTeam/qyu/tree/master/examples/split_n_sync/workers and let me know if it is still not clear.

How should exceptions be handled? Does Qyu provide any facility for retrying jobs?

• Qyu retries tasks indefinitely if an error is raised. If you don't want to retry the task then rescue the error. If you want to report/log errors and still retry, you can rescue the error, do what you need to do ant then re-raise it.

Is there any logging facilities in Qyu?

• Qyu logs all errors that cause a task to fail as before retrying.

worker error: ERROR_CLASS: ERROR_MESSAGE
backtrace: BACKTRACE

Qyu's logger is also accessible via Qyu.logger.info "say something" for example.

---

Thanks for your questions and interest. We will add more documentation soon.