Arq scheduler #20

sverhoeven · 2023-01-16T12:30:53Z

Use https://arq-docs.helpmanual.io/ to create a scheduler that uses Redis as a queue and multiple workers (bartender perform cli).

TODO:

ArqScheduler
Tests
Build from config.yaml
Document scheduler in README
Development documentation: start redis Docker container + start worker
bartender perform subcommand for worker
~~[ ] Add redis + workers to docker-compose~~ postponed to Docker compose deployment with arq workers #30

To test:

Create config file with uncommented copy of https://github.com/i-VRESSE/bartender/blob/arq/config-example.yaml#L53-L64 as the first destination
Start Redis container with docker run --detach --publish 6379:6379 redis:7
Start bartender serve
In another shell start bartender mix
Submit a job as described at https://github.com/i-VRESSE/bartender/tree/arq#word-count-example

Fixes #21
Should be merged after #17

Refs #20

Peter9192

Works well. I still get issues when the default job dir doesn't exist. Also, I have a question about the bartender mix cli and the assignment of one vs multiple workers.

src/bartender/__main__.py

src/bartender/schedulers/arq.py

src/bartender/schedulers/slurm.py

Peter9192 · 2023-02-03T10:35:19Z

src/bartender/__main__.py

+    if not configs:
+        raise ValueError("No destination found in config file using arq scheduler")
+
+    asyncio.run(run_workers(configs))


I'm a bit confused here. Do you need a separate destination for each worker? And each arqscheduler can have only one worker?

Sorry to hear that,
Each arqscheduler needs at least one worker., but multiple (on different machines) is used to handle more jobs running at the same time.

You can have multiple destinations, for example

destinations: quick: scheduler: type: arq redis_dsn: redis://localhost:6379 filesystem: type: local small: scheduler: type: arq redis_dsn: redis://localhost:6379 queue: small filesystem: type: dirac big: scheduler: type: arq redis_dsn: redis://bartender.uu.nl:6379 queue: big filesystem: type: sftp hostname: headnode.cluster.uu.nl

quick, could have worker on same machine as bartender web service. bartender perform --destination quick

small, could have workers run on a grid machines

big, could have workers on hpc cluster compute nodes.

While checking arq docs I saw it used some bad defaults, so I added max_jobs and job_timeout props to ArqSchedulerConfig.

Should such an example be part of the docs?

So you add more workers by passing the same scheduler address in different destinations? Would be good to add more documentation on the new configuration page in the docs, yes.

when redis is on internet a strong password should be added and some firewall rules

src/bartender/web/api/job/sync.py

src/bartender/web/api/job/views.py

tests/schedulers/test_arq.py

src/bartender/__main__.py

Peter9192 · 2023-02-03T12:43:27Z

Opened #48 to see what's required to apply google docstring format to this branch as well. There seem to be no conflicts between this branch and #45. After merging #45, the only changes needed in this branch are contained in this commit: 1921954

The built-in arq job_timeout default was 5 minutes which is too short. Set it to 1 hour.

sverhoeven

Thanks for reviewing, you have given me some nice improvements.

src/bartender/schedulers/arq.py

src/bartender/__main__.py

sverhoeven · 2023-02-06T15:29:44Z

src/bartender/__main__.py

+    if not configs:
+        raise ValueError("No destination found in config file using arq scheduler")
+
+    asyncio.run(run_workers(configs))


Sorry to hear that,
Each arqscheduler needs at least one worker., but multiple (on different machines) is used to handle more jobs running at the same time.

You can have multiple destinations, for example

destinations: quick: scheduler: type: arq redis_dsn: redis://localhost:6379 filesystem: type: local small: scheduler: type: arq redis_dsn: redis://localhost:6379 queue: small filesystem: type: dirac big: scheduler: type: arq redis_dsn: redis://bartender.uu.nl:6379 queue: big filesystem: type: sftp hostname: headnode.cluster.uu.nl

quick, could have worker on same machine as bartender web service. bartender perform --destination quick

small, could have workers run on a grid machines

big, could have workers on hpc cluster compute nodes.

While checking arq docs I saw it used some bad defaults, so I added max_jobs and job_timeout props to ArqSchedulerConfig.

src/bartender/__main__.py

src/bartender/schedulers/arq.py

src/bartender/web/api/job/sync.py

src/bartender/web/api/job/views.py

tests/schedulers/test_arq.py

sverhoeven · 2023-02-06T15:52:17Z

src/bartender/__main__.py

+    if not configs:
+        raise ValueError("No destination found in config file using arq scheduler")
+
+    asyncio.run(run_workers(configs))


Should such an example be part of the docs?

Peter9192

Thanks for following up. It would be nice to add some more explanation to the new configuration docs page in from #49

As after merge with origin/main the file was inconsistent

sverhoeven · 2023-02-24T12:39:44Z

Example was added to https://github.com/i-VRESSE/bartender/blob/arq/docs/configuration.md#example-of-running-jobs-with-multiple-destinations-and-workers

sverhoeven added 3 commits January 16, 2023 13:31

Add arq as dependency

e581d12

Refs #20

Make repr look same as __init__

dcdf166

Added ArqScheduler, untested and undocumented

7b9864a

sverhoeven changed the base branch from main to multi-scheduler January 16, 2023 14:15

sverhoeven and others added 9 commits January 17, 2023 16:51

Started adding tests for ArqScheduler

c65b49e

Simplify flake8 ignore globss for tests

f2c4f41

Moved todos to GH issue #26

c78937f

Coverage should be done on src

e91e173

Added working tests for ArqScheduler

33a4140

Added support to build arq scheduler from config file

263afd6

Added bartender mix subcommand + documented arq scheduler

408e2d1

Supply Redis dsn on testing build as ArqRedis connects on construction

986bb4c

Merge branch 'multi-scheduler' into arq

3bd16de

sverhoeven mentioned this pull request Jan 24, 2023

Docker compose deployment with arq workers #30

Closed

sverhoeven marked this pull request as ready for review January 24, 2023 10:54

sverhoeven requested a review from Peter9192 January 24, 2023 10:57

sverhoeven added 2 commits January 24, 2023 15:48

Merge remote-tracking branch 'origin/multi-scheduler' into arq

c07614a

Merge branch 'multi-scheduler' into arq

9b992dc

Base automatically changed from multi-scheduler to main January 27, 2023 12:06

Peter9192 reviewed Feb 3, 2023

View reviewed changes

src/bartender/__main__.py Show resolved Hide resolved

Peter9192 mentioned this pull request Feb 3, 2023

Apply google docstring formatting to code in branch arq #48

Closed

sverhoeven added 7 commits February 6, 2023 14:14

Merge remote-tracking branch 'origin/main' into arq

9307d86

Run docconvert --in-place -c docconvert-config.json src/

b9cdae5

Wait for job to error out

77a61c2

Rename bartender mix to bartender perform

b5708cf

Tyopo

1219ac6

Add --destination filter to bartender perform sub command.

3014398

Add max_jobs + job_timeout to ArqScheduler config

dc54101

The built-in arq job_timeout default was 5 minutes which is too short. Set it to 1 hour.

sverhoeven added 2 commits February 6, 2023 16:33

Fix repr

0f16b0d

Fix repr test

338851e

sverhoeven commented Feb 6, 2023

View reviewed changes

Peter9192 mentioned this pull request Feb 10, 2023

Cancelled state for arq scheduler? #53

Open

Peter9192 approved these changes Feb 10, 2023

View reviewed changes

sverhoeven added 3 commits February 24, 2023 11:53

Merge remote-tracking branch 'origin/main' into arq

1d05e97

poetry lock --no-update

a525aec

As after merge with origin/main the file was inconsistent

Add multiple destinations example

d40d62b

sverhoeven merged commit ebaa0f3 into main Feb 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arq scheduler #20

Arq scheduler #20

sverhoeven commented Jan 16, 2023 •

edited

Loading

Peter9192 left a comment

Peter9192 Feb 3, 2023

sverhoeven Feb 6, 2023 •

edited

Loading

sverhoeven Feb 6, 2023

Peter9192 Feb 10, 2023

sverhoeven Feb 23, 2023

Peter9192 commented Feb 3, 2023

sverhoeven left a comment

sverhoeven Feb 6, 2023 •

edited

Loading

sverhoeven Feb 6, 2023

Peter9192 left a comment

sverhoeven commented Feb 24, 2023

Arq scheduler #20

Arq scheduler #20

Conversation

sverhoeven commented Jan 16, 2023 • edited Loading

Peter9192 left a comment

Choose a reason for hiding this comment

Peter9192 Feb 3, 2023

Choose a reason for hiding this comment

sverhoeven Feb 6, 2023 • edited Loading

Choose a reason for hiding this comment

sverhoeven Feb 6, 2023

Choose a reason for hiding this comment

Peter9192 Feb 10, 2023

Choose a reason for hiding this comment

sverhoeven Feb 23, 2023

Choose a reason for hiding this comment

Peter9192 commented Feb 3, 2023

sverhoeven left a comment

Choose a reason for hiding this comment

sverhoeven Feb 6, 2023 • edited Loading

Choose a reason for hiding this comment

sverhoeven Feb 6, 2023

Choose a reason for hiding this comment

Peter9192 left a comment

Choose a reason for hiding this comment

sverhoeven commented Feb 24, 2023

sverhoeven commented Jan 16, 2023 •

edited

Loading

sverhoeven Feb 6, 2023 •

edited

Loading

sverhoeven Feb 6, 2023 •

edited

Loading