[help] Error: _hash_ no such index at level 2

[koefoeden]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi!

I'm getting the error

_hash_ no such index at level 2

from a target that dynamically branches over a list of input files, and splits it up into a possibly varying number of output files depending on the exact input file content. I've tried to reproduce the error in a minimal example, but I have not been succesful (i.e. the minimal example runs as expected), and I hypothesize that the problem is something a bit trivial on my side. Unfortunately, there is no workspace or traceback generated for the target branches for some reason, so it makes it a bit hard to debug - so if anyone has some intuition based on the specific error message, it would be greatly appreciated!

This is what the (subsetted) output of tar_meta() looks like for each of the 10 failed input file/dynamic branches.

and this is what the reporter-output looks like:

Below is a simplified example analogous to capture the overall structure in the problematic part of my pipeline. Here, I have removed two outer tar_map layers (which is the test.muscle. parts of the original erroring branch names, using the dot as delimiter), to perhaps aid your intuition in what might be going wrong.

library(targets)
library(tarchetypes)
tar_option_set(
  iteration = "list"
)

list(
  tar_target(
    directories,
    {
      dirs <- c("dir1", "dir2", "dir3")
      map(dirs, ~dir.create(.x))
      dirs
    }
  ),
  tar_file(
    input_file_paths,
    {
      write_lines(file=file.path(directories,"input_file.txt"), x="abc")
      file.path(directories,"input_file.txt")
    },
    pattern = map(directories)
  ),
  tar_target(
    sample_indices,
    sample(x=1:length(input_file_paths), size=2)
  ),
  tar_files(
    sampled_input_paths,
    input_file_paths[sample_indices]
  ),
  tar_file(
    replicate_of_erroring_target,
    {
      # slit input file into one or two files depending on branch

      out_path_1 <- paste0(sampled_input_paths_files,".out1")
      file.copy(sampled_input_paths_files, out_path_1)
      out_path_2 <- NULL

      if (str_detect(sampled_input_paths_files, "dir2")) {
        out_path_2 <- paste0(sampled_input_paths_files,".out2")
        file.copy(sampled_input_paths_files, out_path_2)
      }

      return(c(out_path_1, out_path_2))
    },
    pattern=map(sampled_input_paths_files)
  )
)

As you can see from running tar_read(replicate_of_erroring_target) below, this minimal example outputs exactly what I aim to get in my real pipeline.

Any suggestions on how to continue debugging this error is also greatly appreciated.

Finally, here is the sessionInfo extracted from the SLURM-controller worker output:

R version 4.3.3 (2024-02-29)
• Platform: x86_64-pc-linux-gnu (64-bit)
• Running under: Red Hat Enterprise Linux 8.10 (Ootpa)
•
• Matrix products: default
• BLAS/LAPACK: /maps/direct/software/openblas/0.3.24/lib/libopenblasp-r0.3.24.so; LAPACK version 3.11.0
•
• locale:
• [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
• [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
• [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
• [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
• [9] LC_ADDRESS=C LC_TELEPHONE=C
• [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
•
• time zone: Europe/Copenhagen
• tzcode source: system (glibc)
•
• attached base packages:
• [1] stats graphics grDevices datasets utils methods base
•
• other attached packages:
• [1] targets_1.9.1.9000 conflicted_1.2.0 magrittr_2.0.3 lubridate_1.9.3
• [5] forcats_1.0.0 stringr_1.5.1 dplyr_1.1.4 purrr_1.0.2
• [9] readr_2.1.5 tidyr_1.3.1 tibble_3.2.1 ggplot2_3.5.1
• [13] tidyverse_2.0.0
•
• loaded via a namespace (and not attached):
• [1] utf8_1.2.4 generics_0.1.3 renv_1.0.7
• [4] stringi_1.8.4 hms_1.1.3 grid_4.3.3
• [7] timechange_0.3.0 fastmap_1.2.0 autometric_0.0.5.9002
• [10] processx_3.8.4 backports_1.5.0 secretbase_1.0.1
• [13] promises_1.3.2 ps_1.8.0 BiocManager_1.30.23
• [16] fansi_1.0.6 scales_1.3.0 crew_0.10.2.9001
• [19] codetools_0.2-20 cli_3.6.3 rlang_1.1.4
• [22] munsell_0.5.1 withr_3.0.0 cachem_1.1.0
• [25] yaml_2.3.9 tools_4.3.3 tzdb_0.4.0
• [28] memoise_2.0.1 getip_0.1-4 nanonext_1.3.0
• [31] colorspace_2.1-0 base64url_1.4 vctrs_0.6.5
• [34] R6_2.5.1 lifecycle_1.0.4 pkgconfig_2.0.3
• [37] callr_3.7.6 later_1.4.1 pillar_1.9.0
• [40] gtable_0.3.5 Rcpp_1.0.13 glue_1.7.0
• [43] data.table_1.15.4 xfun_0.46 tidyselect_1.2.1
• [46] knitr_1.48 mirai_1.3.0 igraph_2.0.3
• [49] compiler_4.3.3

[wlandau]

Hard to say, but you can get tracebacks with workspaces: https://books.ropensci.org/targets/debugging.html#workspaces

Probably unrelated, but input_file_paths may invalidate directories because the latter tracks everything in those newly created folders. I recommend creating the directory in the same target you create the file:

tar_target(directories, c("dir1", "dir2", "dir3")),
tar_target(
  input_file_paths,
  {
    fs::dir_create(directories) # does not fail if the directory already exists
    writeLines("abc", file.path(directories, "input_file.txt"))
    file.path(directories, "input_file.txt")
  },
  pattern = map(directories)
)

[koefoeden]

Hi Will - unfortunately, it does not seem that any workspaces for these targets are saved, even though the option is set. I can confirm that workspaces are indeed saved for other targets on error - but not for these it seems. Do you know why this might be?

Regarding the errors - have you previously encountered these, and are they possibly related to the targets infrastructure, or is it more likely a third-party package? This info would help me a lot for debugging purposes.

And thanks for the directory tip - this is likely unrelated, as these folders already exist in the "real" pipeline.

[koefoeden]

Hmm - rolling back from the development version 1.9.1.9000 to 1.9.1 seems to have done the trick. Maybe it was a bit reckless to go straight for the development version when updating the package! Thanks for your help again!

[wlandau]

I may have fixed this as of bc09d7a. Please let me know if you still have this issue in targets >= 1.9.1.9003.