EXECUTION_SWEEP job queue is growing

[emhelmich]

Hi,

Duplicates closed issue: #20255

Version 2.12.0 has reduced the frequency of the cron job to once per day (from once per hour), but we are still seeing a growing queue.

[chlins]

You mean the job queue is growing seen from the Jobservice Dashboard? Did you stop and then resume the EXECUTION_SWEEP queue after upgrade?

[emhelmich]

Yes, that's what I mean. I have now stopped the EXECUTION_SWEEP - this emptied the queue. But I believe a collegue did do this after the update. I guess tomorrow I can see if tonight's job completed.

[chlins]

Can you check the jobservice's task logs, the current execution frequency is once a day, and if there is still a buildup, it means that running that task once will take more than a day?

[emhelmich]

Indeed, the job queue keeps growing. Probably no EXECUTION_SWEEP finishes correctly (screenshot below).

I see a number of different errors in the jobservice container logs. Perhaps the most recurring variant is this type (searching for the job id 0598b5a1b4eef76e5b22b598@1725235200 results in many repeating errors):

2024-12-07T23:12:08Z [INFO] [/jobservice/worker/cworker/reaper.go:195]: Reaper: fire hook again for job 0598b5a1b4eef76e5b22b598@1725235200 as job status change is not ACKed: Scheduled(rev=1725235186)
2024-12-07T23:12:38Z [ERROR] [/jobservice/hook/hook_agent.go:155]: Retry: sending hook event error: Post "http://sro-harbor-voorportaal-prd-core:80/service/notifications/tasks/166562": context deadline exceeded (Client.Timeout exceeded while awaiting headers), evt=status change: job=0598b5a1b4eef76e5b22b598@1725235200, status=Scheduled, revision=1725235186->http://sro-harbor-voorportaal-prd-core:80/service/notifications/tasks/166562, duration=1.323674704s
2024-12-07T23:12:38Z [ERROR] [/jobservice/runtime/bootstrap.go:168]: trigger hook event error: Post "http://sro-harbor-voorportaal-prd-core:80/service/notifications/tasks/135145": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[stonezdj]

Can you please check the size of the task and execution table?

docker exec -it harbor-db bash
>psql -U postgres -d registry
>select count(1) from task;
>select count(1) from execution;

Maybe there is a execution_sweep job is running in background and blocking subsequent job.
Please restart harbor instance, and flush the jobservice redis db if the Harbor is installed with docker-compose, for kubernetes, you need to find out the redis db which has many key with {harbor_job_service_namespace} prefix. or just flush all database.

docker exec -it <redis_container_id> bash
>redis-cli 
# 2 is the db index of job service.
>select 2  
>flushdb

Then restart the Harbor, it could fix the dangling job blocking the subsequent job.

[emhelmich]

select count(1) from task; -> 827434
select count(1) from execution; -> 17715

In our case Harbor runs on Kubernetes. I want to schedule bringing the service down this Friday to try the rest of your suggestion.

[emhelmich]

After running "flushdb" from redis-cli in the redis container it appears last night's EXECUTION_SWEEP job completed successfully.

Thanks for the help, @stonezdj!

[emhelmich]

Closing because fixed.