feat: add rq worker #1011

raphael0202 · 2022-12-08T09:47:46Z

use rq and redis queues to send tasks from API to workers
use redis locks to avoid concurrent product insight updates or concurrent product_update tasks
simplify job workflows
save nutriscore and nutrition-table object detection model results in image_prediction table

…product

- the relative path was different between prod and local, so it didn't work locally - use webnet network instead of robotoff_webnet in ml.yml

- split image import tasks into more granular tasks - save nutriscore detection object model results in DB - save nutrition-table detection object model results in DB

Taking care of DB connection is a good practice

we're going to use a distinct redis-stack for robotoff-ann for safety

for high and low priority jobs

I've checked that as the documentation says, a new atomic transaction is indeed started when using `with db:`

during DB connection

alexgarel

LGTM, here are my comments.

Only the change in deploy is potentially harmfull, we must discuss it or revert.

.deepsource.toml

.github/workflows/container-deploy-ml.yml

doc/how-to-guides/deployment/maintenance.md

doc/introduction/architecture.md

docker-compose.yml

robotoff/workers/tasks/import_image.py

robotoff/app/api.py

robotoff/workers/tasks/product_updated.py

raphael0202 added 16 commits December 6, 2022 13:01

feat: only accept predictions from one product during insight import

5513f7a

feat: use rq to send tasks to workers

727d561

feat: add redis lock to avoid concurrent insight import for the same …

61e77b0

…product

style: fix flake8 and isort errors

4b7cb33

feat: add a lock during product update jobs

ed663f5

fix: add default timeout during TF serving request

7617476

fix: add default TF_SERVING_HOST in .env file

34aaa4e

chore: fix config discrepancy between ML local/prod

1b2564d

- the relative path was different between prod and local, so it didn't work locally - use webnet network instead of robotoff_webnet in ml.yml

feat: simplify image import process

bdcb4ef

feat: create more atomic tasks during image import

f153652

- split image import tasks into more granular tasks - save nutriscore detection object model results in DB - save nutrition-table detection object model results in DB

fix: use autoconnect=False during DB connection

4c8fd58

Taking care of DB connection is a good practice

fix: fix DB connection in workers

2696288

feat: make object detection jobs idempotent

a349543

feat: add more logs during resource loading

2341127

fix: switch from redis-stack to classic redis

82630ee

we're going to use a distinct redis-stack for robotoff-ann for safety

feat: refresh cache when performing worker maintenance tasks

838f9e1

github-actions bot added category prediction GitHub Actions insights integration tests object detection predictions robotoff app robotoff-cli 🧪 tests unit tests workers neural category prediction labels Dec 8, 2022

github-actions bot assigned raphael0202 Dec 8, 2022

raphael0202 force-pushed the add-rq-worker branch from 67f3957 to 838f9e1 Compare December 8, 2022 09:51

fix: disable deepsource autofix

fa3e2ef

raphael0202 force-pushed the add-rq-worker branch from d914ae2 to fa3e2ef Compare December 8, 2022 10:08

raphael0202 requested a review from alexgarel December 8, 2022 10:09

fix: update poetry lock file

31d5b84

raphael0202 marked this pull request as ready for review December 8, 2022 10:30

raphael0202 requested a review from a team as a code owner December 8, 2022 10:30

raphael0202 added 11 commits December 8, 2022 12:07

feat: use queues for refresh_insight job on all DB

143cf94

fix: use db.connection_context in with_db decorator

1ea52de

feat: add new CLI commands to launch background tasks

cbb970e

feat: add two kind of workers: worker_high and worker_low

8cc2b22

for high and low priority jobs

fix: fix flake8 and isort errors

2809881

feat: use batches in refresh_insights job

09b5fdd

feat: remove orphan containers when doing make up

81357b7

fix: no need to use db.atomic() when db context manager is used

761030a

I've checked that as the documentation says, a new atomic transaction is indeed started when using `with db:`

feat: improve CLI import commands

06bf2e1

docs: improve documentation

fc1f2a4

fix: fix integration tests after adding autoconnect=False

e1c3132

during DB connection

raphael0202 temporarily deployed to robotoff-net December 9, 2022 09:11 — with GitHub Actions Inactive

fix: open DB connection when needed in scheduler

5a5194a

github-actions bot added the scheduler label Dec 9, 2022

fix: add with_db decorator to generate_fiber_quality_facet

a7eb556

raphael0202 temporarily deployed to robotoff-net December 9, 2022 09:57 — with GitHub Actions Inactive

raphael0202 merged commit c62f14c into master Dec 9, 2022

raphael0202 deleted the add-rq-worker branch December 9, 2022 10:26

raphael0202 mentioned this pull request Dec 12, 2022

Switch from multiprocessing Listener to Celery + Redis #1005

Closed

alexgarel reviewed Dec 12, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add rq worker #1011

feat: add rq worker #1011

raphael0202 commented Dec 8, 2022 •

edited

Loading

alexgarel left a comment

feat: add rq worker #1011

feat: add rq worker #1011

Conversation

raphael0202 commented Dec 8, 2022 • edited Loading

alexgarel left a comment

Choose a reason for hiding this comment

raphael0202 commented Dec 8, 2022 •

edited

Loading