Jobs stuck in matched when using SingularityCE

[chrisburr]

In LHCb we've been seeing jobs getting stuck in Matched when using SingularityCE instead of rescheduling when issues happen launching the container.

I suspect it's caused by the interplay between the PoolCE with an inner SingularityCE.

[aldbr]

Not entirely sure about that but from what I have observed so far, the JobAgent is not able to get the result of a job submission from a PoolCE.

As the PoolCE manages a pool of processes, it cannot return the status of a submission synchronously, so the PoolCE always returns S_OK() to the JobAgent.

https://github.com/DIRACGrid/DIRAC/blob/rel-v8r0/src/DIRAC/Resources/Computing/PoolComputingElement.py#L149

The results of the submissions are handled by a callback function named PoolCE.finalizeJob(), which only prints an error log in case of failure.

https://github.com/DIRACGrid/DIRAC/blob/rel-v8r0/src/DIRAC/Resources/Computing/PoolComputingElement.py#L201

Thus, the following blocks of code do not seem to be executed:

https://github.com/DIRACGrid/DIRAC/blob/rel-v8r0/src/DIRAC/WorkloadManagementSystem/Agent/JobAgent.py#L307-L315
https://github.com/DIRACGrid/DIRAC/blob/rel-v8r0/src/DIRAC/WorkloadManagementSystem/Agent/JobAgent.py#L660-L676

A suggestion:

• The JobAgent would pass the jobID to PoolCE.submitJob().
• The PoolCE would add the jobID to the PoolCE.taskResults.
• In each cycle, the JobAgent would check PoolCE.taskResults, would retrieve the job IDs of the failed submissions and would handle the failures properly.

This solution means that the JobAgent would be more dependent on the PoolCE, or at least its structure.
This is probably not an issue as it is a generic component that can always be used.

[fstagni]

Good analysis, I don't find it incorrect. It is maybe possible to verify it with a test, it would be useful also when coding the solution.
If you are right, then there's no interplay issue with SingularityCE, but everything is on PoolCE side.

Regarding your suggestion: IMHO, PoolCE might even be the default "Inner CE" out there (if it's bug-free...). The reason why we kept it as the not-default one was exactly because of these possible bugs.

[aldbr]

Another thing I am not entirely sure to understand.
Is there a mechanism that marks jobs as FAILED if the JobAgent does not finish well?
Because I don't see any, but there is a comment in the code that seems to indicate that: https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/WorkloadManagementSystem/Agent/JobAgent.py#L663

[chrisburr]

It was true at the time the comment was written and was removed in df805d5.