Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline uses cached process despite edited eval script #5470

Open
CormacKinsella opened this issue Nov 5, 2024 · 2 comments · May be fixed by #5517
Open

Pipeline uses cached process despite edited eval script #5470

CormacKinsella opened this issue Nov 5, 2024 · 2 comments · May be fixed by #5517
Assignees
Labels

Comments

@CormacKinsella
Copy link

Bug report

  • Despite editing an eval script, a cached version of the process will be used under certain circumstances

Expected behavior and actual behavior

  • Expected: editing an eval script (e.g. from multiqc --version to multiqc --version | sed "s/version//" ) will be picked up as an edit and therefore a cached version won't be used.

  • Actual behaviour: it works correctly if resume = true is stated within the main workflow script. However if -resume is provided on command line, or resume = true is given in nextflow.config, then the cached version is used after a change to the eval script

Steps to reproduce the problem

  1. Generate nextflow.config and main.nf from below example
  2. Run the pipeline
  3. Edit the eval statement to 'multiqc --version | sed "s/version//"'
  4. Rerun -> nothing changes about the output
resume = true
apptainer.enabled = true
apptainer.autoMounts = true
nextflow.enable.dsl=2
nextflow.preview.topic = true

process FOO {
    tag "${id}"
    container "quay.io/biocontainers/multiqc:1.14--pyhdfd78af_0"

    input:
    val(id)

    output:
    tuple val(task.process), eval('multiqc --version'), topic: versions

    script:
    """
    """
}

workflow {
    input_ch = Channel.from("Sample1") 

    FOO(input_ch)

    channel.topic('versions').view()
}

Program output

Before edit:
[FOO, multiqc, version 1.14]

After edit:
[FOO, multiqc, version 1.14]

Should be:
[FOO, multiqc, 1.14] # Note that this behaviour is achieved by moving resume = true from nextflow.config to main.nf

Environment

  • Nextflow version: version 24.10.0 build 5928
  • Java version: openjdk 11.0.24 2024-07-16
  • Operating system: Ubuntu 22.04.5 LTS
  • Bash version: GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

Additional context

@bentsherman bentsherman added the bug label Nov 5, 2024
@bentsherman
Copy link
Member

Good point, the eval script was never added to the task hash

@bentsherman
Copy link
Member

@jorgee this one should be pretty easy if you'd like to try. Basically we need to add the eval outputs to the task hash here:

// add all the input name-value pairs to the key generator
for( Map.Entry<InParam,Object> it : task.inputs ) {
keys.add( it.key.name )
keys.add( it.value )
}

Something like this:

        // add inputs ...
        // ...

        // add eval outputs
        for( Map.Entry<OutParam,Object> it : task.outputs ) {
            if( it.key instanceof CmdEvalParam )
                keys.add( it.key.getTarget(task.context) )
        }

@jorgee jorgee self-assigned this Nov 14, 2024
@jorgee jorgee linked a pull request Nov 18, 2024 that will close this issue
@jorgee jorgee linked a pull request Nov 18, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants