Unexpected rebuilds when using tar_group() + iteration = 'group'

[lindsayplatt]

In my pipeline, I have a step that first fetches information for a list of states I provide. Then, based on the returned information, I want to branch over individual state data and kick off a long data download per state. It is critical to me that my pipeline only rebuilds states when absolutely necessary. Below is a reproducible example of this pattern but without actual data retrieval code, just prints "Fetching data".

To tackle this pipeline challenge, I thought that using tar_group() plus iteration = 'group' for my dataset with all of the state data would make sense. I was surprised to learn that this approach didn't work when you add in a new state that comes before the existing states alphabetically (and no, sorting that vector didn't work either). When I reworked this to use pattern = map() + iteration = 'list', it did work as expected and didn't rebuild everything when I added a new state that came before the others alphabetically.

I am wondering why the tar_group() approach doesn't work. Am I misusing it? Have I uncovered a bug? What is the best way to choose between iteration types? Thanks!

Using tar_group() + iteration = 'group' rebuilds my pipeline unnecessarily

_targets.R file that does not operate as expected when I change the states target

options(tidyverse.quiet = TRUE)
library(targets)
library(tarchetypes)
library(tidyverse)

list(
  # I tried using `sort()` but that didn't matter.
  tar_target(states, c('MI', 'MN')), 
  
  tar_target(state_data, 
    as.tibble(state.x77) %>% 
      mutate(state_nm = rownames(state.x77), 
             state_abb = state.abb) %>% 
      relocate(state_nm, state_abb, .before = Population) %>% 
      filter(state.abb %in% states) %>% 
      # Set up to be a grouped target in the hopes that targets won't rebuild other groups
      group_by(state_abb) %>% 
      tar_group(),
    iteration = "group"
  ),
  
  tar_target(per_state_task, {
    message(sprintf("Fetching data for %s", unique(state_data$state_nm)))
    state_data$Area
  },
  pattern = map(state_data)
  )
)

Walk through of unexpected results

Running the above works ✔

> tar_make()
* start target states
* built target states
* start target state_data
* built target state_data
* start branch per_state_task_681e3696
Fetching data for Michigan
* built branch per_state_task_681e3696
* start branch per_state_task_ea91aeb2
Fetching data for Minnesota
* built branch per_state_task_ea91aeb2
* built pattern per_state_task
* end pipeline

Changing states to tar_target(states, c('MI', 'MN', 'PA')) also works ✔

> tar_make()
* start target states
* built target states
* start target state_data
* built target state_data
v skip branch per_state_task_681e3696
v skip branch per_state_task_ea91aeb2
* start branch per_state_task_c1949ea9
Fetching data for Pennsylvania
* built branch per_state_task_c1949ea9
* built pattern per_state_task
* end pipeline

Changing states to tar_target(states, c('MI', 'MN', 'PA', 'AL')) rebuilds all of the previous targets unexpectedly ❌

> tar_make()
* start target states
* built target states
* start target state_data
* built target state_data
* start branch per_state_task_cf209554
Fetching data for Alabama
* built branch per_state_task_cf209554
* start branch per_state_task_94228602
Fetching data for Michigan
* built branch per_state_task_94228602
* start branch per_state_task_7e7176c2
Fetching data for Minnesota
* built branch per_state_task_7e7176c2
* start branch per_state_task_9912688e
Fetching data for Pennsylvania
* built branch per_state_task_9912688e
* built pattern per_state_task
* end pipeline

Using pattern = map() + iteration = 'list' skips builds as expected

_targets.R file that operates as expected when I change the states target

options(tidyverse.quiet = TRUE)
library(targets)
library(tarchetypes)
library(tidyverse)

list(
  # I tried using `sort()` but that didn't matter.
  tar_target(states, c('MI', 'MN')), 
  
  tar_target(state_data, 
    as.tibble(state.x77) %>% 
      mutate(state_nm = rownames(state.x77), 
             state_abb = state.abb) %>% 
      relocate(state_nm, state_abb, .before = Population) %>% 
      filter(state.abb %in% states),
    pattern = map(states),
    iteration = "list"
  ),
  
  tar_target(per_state_task, {
    message(sprintf("Fetching data for %s", unique(state_data$state_nm)))
    state_data$Area
  },
  pattern = map(state_data)
  )
)

Walk through of expected results

Running the above works ✔

> tar_make()
* start target states
* built target states
* start branch state_data_13e7a07e
* built branch state_data_13e7a07e
* start branch state_data_a04292f3
* built branch state_data_a04292f3
* built pattern state_data
* start branch per_state_task_fc5e6bc7
Fetching data for Michigan
* built branch per_state_task_fc5e6bc7
* start branch per_state_task_98f4a757
Fetching data for Minnesota
* built branch per_state_task_98f4a757
* built pattern per_state_task
* end pipeline

Changing states to tar_target(states, c('MI', 'MN', 'PA')) also works ✔

> tar_make()
* start target states
* built target states
v skip branch state_data_13e7a07e
v skip branch state_data_a04292f3
* start branch state_data_dc829027
* built branch state_data_dc829027
* built pattern state_data
v skip branch per_state_task_fc5e6bc7
v skip branch per_state_task_98f4a757
* start branch per_state_task_e8d33826
Fetching data for Pennsylvania
* built branch per_state_task_e8d33826
* built pattern per_state_task
* end pipeline

Changing states to tar_target(states, c('MI', 'MN', 'PA', 'AL')) also works ✔

> tar_make()
* start target states
* built target states
v skip branch state_data_13e7a07e
v skip branch state_data_a04292f3
v skip branch state_data_dc829027
* start branch state_data_9cc7a469
* built branch state_data_9cc7a469
* built pattern state_data
v skip branch per_state_task_fc5e6bc7
v skip branch per_state_task_98f4a757
v skip branch per_state_task_e8d33826
* start branch per_state_task_af272fb9
Fetching data for Alabama
* built branch per_state_task_af272fb9
* built pattern per_state_task
* end pipeline

I am using targets v0.4.2.

[wlandau]

Could it be related to ropensci/tarchetypes#53? Would you try the development version of targets?

[lindsayplatt]

I just tried the development version but that didn't change the outcome. Trying to understand

if all the slices of the data are affected by the changes to dplyr grouping attributes

Technically the column I am grouping by changes each iteration because there are new unique values (so new groups). But what I don't understand is why it only seems to be an issue when I add new states that are alphabetically ahead of the ones I was already building this for. If this was an issue with dplyr grouping attributes, I would think it would happen when I add new groups every time and not just when they are at different points in the alphabet.

Could there be some kind of internal sorting going on that causes this?

[wlandau]

Sorry, I was not at a computer when I first responded.

You definitely found a bug. When you first ran the pipeline, the value of the tar_group column in the MI group was 1. When you inserted AL, tar_group for MI became 2, which changed the hash.

This is exactly why I tried to avoid numeric indexes for most of dynamic branching. If branch names looked like state_data_1 and state_data_2 instead of state_data_13e7a07e, we would have the exact same problem all the time, even when iteration is "list".

It should be a pretty easy fix on my end. I will just remove tar_group before hashing the slice whenever iteration is "group".

[wlandau]

Should be fixed in d6bac3a. Thanks for catching and reproducing the error, would have been difficult to find otherwise.

[wlandau]

Unfortunately, the solution invalidates all existing workflows that rely on iteration = "group".

[lindsayplatt]

That definitely fixed my example here! Still getting my actual pipeline to work, but there could be other factors contributing to that. Thanks for the quick response and action 💯