Slurm jobs completing but targets doesn't realise

[multimeric]

I have a large workflow with several thousand branches. When I run the workflow, I can see it starting the first 10 tasks (the number of workers I have), but then it hangs forever and never progresses:

• +1000 UTC 2022-09-26 18:03 58.33 start branch loaded_count_db_8cbc8dd9
• +1000 UTC 2022-09-26 18:04 00.78 start branch loaded_count_db_501b380f
• +1000 UTC 2022-09-26 18:04 03.43 start branch loaded_count_db_e11f32f5
• +1000 UTC 2022-09-26 18:04 06.06 start branch loaded_count_db_9d7b826b
• +1000 UTC 2022-09-26 18:04 08.39 start branch loaded_count_db_bba47662
• +1000 UTC 2022-09-26 18:04 11.54 start branch loaded_count_db_db6e5f6a
• +1000 UTC 2022-09-26 18:04 14.14 start branch loaded_count_db_269e0176
• +1000 UTC 2022-09-26 18:04 17.05 start branch loaded_count_db_050a3af7
• +1000 UTC 2022-09-26 18:04 20.18 start branch loaded_count_db_ab762fe6
• +1000 UTC 2022-09-26 18:04 23.46 start branch loaded_count_db_7d38b0d9

However, if I ask slurm how these jobs are going, they have all either finished or timed out:

$ sacct --format JobID,JobName%30,State,ExitCode --allocations
JobID                               JobName      State ExitCode 
------------ ------------------------------ ---------- -------- 
8260200            loaded_count_db_8cbc8dd9  COMPLETED      0:0 
8260201            loaded_count_db_501b380f  COMPLETED      0:0 
8260202            loaded_count_db_e11f32f5  COMPLETED      0:0 
8260203            loaded_count_db_9d7b826b  COMPLETED      0:0 
8260204            loaded_count_db_bba47662  COMPLETED      0:0 
8260205            loaded_count_db_db6e5f6a    TIMEOUT      0:0 
8260206            loaded_count_db_269e0176  COMPLETED      0:0 
8260207            loaded_count_db_050a3af7  COMPLETED      0:0 
8260208            loaded_count_db_ab762fe6  COMPLETED      0:0 
8260209            loaded_count_db_7d38b0d9  COMPLETED      0:0

My question is, how does this happen? How can a submitted job complete or fail, without targets realising? Is this a common problem? Are there known solutions?

More details below:

For reference, my _targets.R:

library(targets)
library(future)
library(future.batchtools)
library(tarchetypes)

# Set target options:
tar_option_set(
  packages = c("tibble"),
  format = "rds"
)

plan(batchtools_slurm, template="slurm")

#tar_source(files = "dev")
source("dev/counts_to_sql.R")

list(
  tar_files(
    name = sample_dirs,
    command = get_counts_dirs("/vast/projects/RCP/human_cell_atlas/splitted_light_data")
  ),
  tar_files(
    name = count_db,
    format = "file",
    command = create_counts_table("/vast/scratch/users/milton.m/hca.sqlite")
  ),
  tar_target(
    name = loaded_count_db,
    format = "file",
    command = counts_to_sql(sample_dirs, count_db),
    pattern = cross(sample_dirs, count_db),
    resources = tar_resources(
      future = tar_resources_future(
        plan = tweak(
          batchtools_slurm,
          resources = list(memory = 1024 * 50, walltime=60 * 3),
          template="slurm"
        )
      )
    ),
    cue = tar_cue(
      # Never re-run this task, to save time
      mode = "never"
    )
  )
)

My batchtools.conf.R:

cluster.functions = batchtools::makeClusterFunctionsSlurm(
  array.jobs = TRUE
)
default.resources = list(
  ncpus=1,
  walltime=60 * 60,
  memory=1000,
  partition="regular"
)

And my batchtools.slurm.tmpl:

#!/bin/bash

## Job Resource Interface Definition
##
## ntasks [integer(1)]:       Number of required tasks,
##                            Set larger than 1 if you want to further parallelize
##                            with MPI within your job.
## ncpus [integer(1)]:        Number of required cpus per task,
##                            Set larger than 1 if you want to further parallelize
##                            with multicore/parallel within each task.
## walltime [integer(1)]:     Walltime for this job, in seconds.
##                            Must be at least 60 seconds for Slurm to work properly.
## memory   [integer(1)]:     Memory in megabytes for each cpu.
##                            Must be at least 100 (when I tried lower values my
##                            jobs did not start at all).
##
## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.

<%
# relative paths are not handled well by Slurm
log.file = fs::path_expand(log.file)
-%>


#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
#SBATCH --error=<%= log.file %>
#SBATCH --time=<%= ceiling(resources$walltime / 60) %>
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<%= resources$ncpus %>
#SBATCH --mem-per-cpu=<%= resources$memory %>
#SBATCH --export ALL
<%= if (!is.null(resources$partition)) sprintf(paste0("#SBATCH --partition='", resources$partition, "'")) %>
<%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %>

## Initialize work environment like
## source /etc/profile
## module add ...
eval $(/usr/bin/modulecmd bash load R/4.2.1)

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

## Run R:
## we merge R output with stdout from SLURM, which gets then logged via --output option
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

[stuvet]

I've not noticed this exact behaviour, but something similar occurs when batchtools & future.batchtools fail to link to the running jobs - they'll continue on Slurm but will apparently end with an error as far as targets is concerned. I guess if an error were never triggered but there was a failure to link then you'd see your noted behaviour.

When run via future.batchtools (or tar_make_future) there were a few Slurm-only problems with the current CRAN release of batchtools (0.9.15) that were fixed with mllg/batchtools#276
& mllg/batchtools#277. It'd be worth trying with a subsequent git version of batchtools if you haven't already.

[multimeric]

Let's see, I'm using batchtools_0.9.15, future.batchtools_0.10.0, targets_0.13.4. I'll try updating all of the batchtools packages to their git versions to see if that helps. I wish the CRAN versions were up to date!

[stuvet]

The other one to mention is AFAIK true slurm array jobs aren't supported when run via future.batchtools (futureverse/future.batchtools#23 (comment)).

I still add that as a workaround for the gzfile error discussed here (#570 (comment)) where jobs would otherwise get dropped by future.batchtools if they have to be rescheduled even though Slurm would continue to run, but you can't submit more than one job per array. Not sure what would happen if you tried.

[multimeric]

Ah well I'm not actually using the array jobs, I thought I should just include it in the config since my HPC does support it. Anyway, updating those two packages to the git version did solve the issue, thanks for that!

[multimeric]

Actually not quite. I got one job completing but the rest still have issues. In particular any jobs that time out aren't currently reported.

[stemangiola]

Does this help for job timeout? #568