Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"ERROR ~ Unknown config attribute out" when adding functions to nextflow.config #1005

Closed
stevekm opened this issue Jan 21, 2019 · 5 comments
Closed

"ERROR ~ Unknown config attribute out" when adding functions to nextflow.config #1005

stevekm opened this issue Jan 21, 2019 · 5 comments
Labels
triage/invalid

Comments

@stevekm
Copy link
Contributor

stevekm commented Jan 21, 2019

Bug report

When adding custom functions to the nextflow.config file, I get an error like this:

ERROR ~ Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config

Expected behavior and actual behavior

Expected that the functions should not affect the nextflow.config file parsing

Steps to reproduce the problem

(Context: in the middle of trying to write a function to dynamically select the "best" SLURM queue to submit to...)

A base pipeline like this works just fine:

main.nf

import java.text.SimpleDateFormat
Date now = new Date()
SimpleDateFormat timestamp_fmt = new SimpleDateFormat("yyyy-MM-dd_HH-mm-ss")
def timestamp = timestamp_fmt.format(now)
params.outputDir = "output"

Channel.from(1..2).set { input_ch }

process make_file {
    tag "${x}"

    input:
    val(x) from input_ch

    output:
    file("${output_file}") into output_files

    script:
    output_file = "${x}.txt"
    """
    printf "${x}\t\${SLURM_JOB_PARTITION:-none}\t\$(hostname)\t\$(date +"%Y-%m-%d-%H-%M-%S")\n" > "${output_file}"
    """
}
output_files.collectFile(name: "log.${timestamp}.txt", storeDir: "${params.outputDir}").subscribe { println "${it}" }

nextflow.config

def get_best_queue (task){
    // function for selecting the best queue to submit to
    def task_hash = "${task.workDir.parent.baseName}/${task.workDir.baseName[0..5]}"
    println "(get_best_queue) [${task_hash}] getting queue..." // TODO: log.info for this instead..
    // TODO: add queue selection logic here....
    selected_queue = "cpu_short"
    println "(get_best_queue) [${task_hash}] got queue: ${selected_queue}"
    return("${selected_queue}")
}


process {
    executor = 'slurm'
    queue = { get_best_queue(task) }
    clusterOptions = '--ntasks-per-node=1 --export=NONE --export=NTHREADS --mem-bind=local'
}

output:

$ ./nextflow run main.nf
N E X T F L O W  ~  version 18.10.1
Launching `main.nf` [angry_perlman] - revision: e0f82ab739
[warm up] executor > slurm
(get_best_queue) [54/dd0d94] getting queue...
(get_best_queue) [11/720c49] getting queue...
(get_best_queue) [54/dd0d94] got queue: cpu_short
(get_best_queue) [11/720c49] got queue: cpu_short
[54/dd0d94] Submitted process > make_file (1)
[11/720c49] Submitted process > make_file (2)
output/log.2019-01-21_14-31-00.txt

However, when I start to add more functions to my nextflow.config file, things start breaking... For example:

nextflow.config

import java.text.SimpleDateFormat

def get_timestamp(){
    Date now = new Date()
    SimpleDateFormat timestamp_fmt = new SimpleDateFormat("yyyy-MM-dd_HH-mm-ss")
    def timestamp = timestamp_fmt.format(now)
    return(timestamp)
}
println "${get_timestamp()}"

def get_best_queue (task){
    // function for selecting the best queue to submit to
    def task_hash = "${task.workDir.parent.baseName}/${task.workDir.baseName[0..5]}"
    println "(get_best_queue) [${task_hash}] getting queue..." // TODO: log.info for this instead..
    // TODO: add queue selection logic here....
    selected_queue = "cpu_short"
    println "(get_best_queue) [${task_hash}] got queue: ${selected_queue}"
    return("${selected_queue}")
}

process {
    executor = 'slurm'
    queue = { get_best_queue(task) }
    clusterOptions = '--ntasks-per-node=1 --export=NONE --export=NTHREADS --mem-bind=local'
}

output:

$ ./nextflow run main.nf
N E X T F L O W  ~  version 18.10.1
Launching `main.nf` [fabulous_archimedes] - revision: e0f82ab739
2019-01-21_14-33-36
ERROR ~ Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config

 -- Check '.nextflow.log' file for details
make: *** [run] Error 1

^^^ note that the function get_timestamp() successfully runs, as can be seen in the stdout

.nextflow.log

$ cat .nextflow.log
Jan-21 14:33:35.765 [main] DEBUG nextflow.cli.Launcher - $> ./nextflow run main.nf
Jan-21 14:33:35.898 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 18.10.1
Jan-21 14:33:35.906 [main] INFO  nextflow.cli.CmdRun - Launching `main.nf` [fabulous_archimedes] - revision: e0f82ab739
Jan-21 14:33:35.918 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
Jan-21 14:33:35.919 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
Jan-21 14:33:35.936 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jan-21 14:33:36.222 [main] DEBUG nextflow.config.ConfigBuilder - In the following config object the attribute `out` is empty:
  process {
        executor='slurm'
        queue=_nf_config_5d3d5b42$_run_closure1$_closure2@460ebd80
        clusterOptions='--ntasks-per-node=1 --export=NONE --export=NTHREADS --mem-bind=local'
  }

Jan-21 14:33:36.282 [main] ERROR nextflow.cli.Launcher - Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
nextflow.exception.ConfigParseException: Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
        at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:249)
        at nextflow.config.ConfigBuilder.validate(ConfigBuilder.groovy:400)
        at nextflow.config.ConfigBuilder.validate(ConfigBuilder.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
        at nextflow.config.ConfigBuilder.merge0(ConfigBuilder.groovy:367)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:184)
        at nextflow.config.ConfigBuilder.buildConfig0(ConfigBuilder.groovy:316)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
        at nextflow.config.ConfigBuilder.buildGivenFiles(ConfigBuilder.groovy:282)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
        at nextflow.config.ConfigBuilder.buildConfigObject(ConfigBuilder.groovy:633)
        at nextflow.config.ConfigBuilder.build(ConfigBuilder.groovy:646)
        at nextflow.script.ScriptRunner.<init>(ScriptRunner.groovy:115)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:223)
        at nextflow.cli.Launcher.run(Launcher.groovy:428)
        at nextflow.cli.Launcher.main(Launcher.groovy:586)

Another example;

nextflow.config:

// a wrapper closure around executing a string
// can take either a string or a list of strings (for arguments with spaces)
// prints all output, complains and halts on error
// https://stackoverflow.com/questions/159148/groovy-executing-shell-commands
def runCommand = { strList ->
  assert ( strList instanceof String ||
           ( strList instanceof List && strList.each{ it instanceof String } ) \
)
  def proc = strList.execute()
  proc.in.eachLine { line -> println line }
  proc.out.close()
  proc.waitFor()

  print "[INFO] ( "
  if(strList instanceof List) {
    strList.each { print "${it} " }
  } else {
    print strList
  }
  println " )"

  if (proc.exitValue()) {
    println "gave the following error: "
    println "[ERROR] ${proc.getErrorStream()}"
  }
  assert !proc.exitValue()
}
runCommand("echo HELLO WORLD")

def get_best_queue (task){
    // function for selecting the best queue to submit to
    def task_hash = "${task.workDir.parent.baseName}/${task.workDir.baseName[0..5]}"
    println "(get_best_queue) [${task_hash}] getting queue..." // TODO: log.info for this instead..
    // TODO: add queue selection logic here....
    selected_queue = "cpu_short"
    println "(get_best_queue) [${task_hash}] got queue: ${selected_queue}"
    return("${selected_queue}")
}


process {
    executor = 'slurm'
    queue = { get_best_queue(task) }
    clusterOptions = '--ntasks-per-node=1 --export=NONE --export=NTHREADS --mem-bind=local'
}

output:

$ ./nextflow run main.nf
N E X T F L O W  ~  version 18.10.1
Launching `main.nf` [infallible_hypatia] - revision: e0f82ab739
HELLO WORLD
[INFO] ( echo HELLO WORLD )
ERROR ~ Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config

 -- Check '.nextflow.log' file for details

^^^ again you can see that the function runCommand() works

.nextflow.log

$ cat .nextflow.log
Jan-21 14:37:11.272 [main] DEBUG nextflow.cli.Launcher - $> ./nextflow run main.nf
Jan-21 14:37:11.414 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 18.10.1
Jan-21 14:37:11.423 [main] INFO  nextflow.cli.CmdRun - Launching `main.nf` [infallible_hypatia] - revision: e0f82ab739
Jan-21 14:37:11.434 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
Jan-21 14:37:11.435 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
Jan-21 14:37:11.452 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jan-21 14:37:11.756 [main] DEBUG nextflow.config.ConfigBuilder - In the following config object the attribute `out` is empty:
  process {
        executor='slurm'
        queue=_nf_config_e369962c$_run_closure2$_closure6@22bac7bc
        clusterOptions='--ntasks-per-node=1 --export=NONE --export=NTHREADS --mem-bind=local'
  }

Jan-21 14:37:11.808 [main] ERROR nextflow.cli.Launcher - Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
nextflow.exception.ConfigParseException: Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
        at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:249)
        at nextflow.config.ConfigBuilder.validate(ConfigBuilder.groovy:400)
        at nextflow.config.ConfigBuilder.validate(ConfigBuilder.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
        at nextflow.config.ConfigBuilder.merge0(ConfigBuilder.groovy:367)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:184)
        at nextflow.config.ConfigBuilder.buildConfig0(ConfigBuilder.groovy:316)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
        at nextflow.config.ConfigBuilder.buildGivenFiles(ConfigBuilder.groovy:282)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
        at nextflow.config.ConfigBuilder.buildConfigObject(ConfigBuilder.groovy:633)
        at nextflow.config.ConfigBuilder.build(ConfigBuilder.groovy:646)
        at nextflow.script.ScriptRunner.<init>(ScriptRunner.groovy:115)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:223)
        at nextflow.cli.Launcher.run(Launcher.groovy:428)
        at nextflow.cli.Launcher.main(Launcher.groovy:586)

Any ideas why this is breaking? Did I miss something?

Environment

  • Nextflow version: 18.10.1
  • Java version:
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
  • Operating system: RHEL 7
@stevekm
Copy link
Contributor Author

stevekm commented Jan 21, 2019 •
edited
Loading

looks like the error may be originating at this section of the code?

nextflow/modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy

Line 387 in b49e6d7

* Validate a config object verifying is does not contains unresolved attributes

@stevekm
Copy link
Contributor Author

stevekm commented Jan 21, 2019

Also just realized that the error can be reproduced with something as simple as this:

nextflow.config

process {
    executor = 'slurm'
    queue = "cpu_short" 
    clusterOptions = '--ntasks-per-node=1 --export=NONE --export=NTHREADS --mem-bind=local'
}

println "${process}"

output:

./nextflow run main.nf
N E X T F L O W  ~  version 18.10.1
Launching `main.nf` [intergalactic_wozniak] - revision: e0f82ab739
[executor:slurm, queue:cpu_short, clusterOptions:--ntasks-per-node=1 --export=NONE --export=NTHREADS --mem-bind=local]
ERROR ~ Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config

 -- Check '.nextflow.log' file for details
make: *** [run] Error 1

.nextflow.log

$ cat .nextflow.log
Jan-21 15:00:19.911 [main] DEBUG nextflow.cli.Launcher - $> ./nextflow run main.nf
Jan-21 15:00:20.022 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 18.10.1
Jan-21 15:00:20.029 [main] INFO  nextflow.cli.CmdRun - Launching `main.nf` [intergalactic_wozniak] - revision: e0f82ab739
Jan-21 15:00:20.040 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
Jan-21 15:00:20.041 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
Jan-21 15:00:20.058 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jan-21 15:00:20.327 [main] DEBUG nextflow.config.ConfigBuilder - In the following config object the attribute `out` is empty:
  process {
        executor='slurm'
        queue='cpu_short'
        clusterOptions='--ntasks-per-node=1 --export=NONE --export=NTHREADS --mem-bind=local'
  }

Jan-21 15:00:20.379 [main] ERROR nextflow.cli.Launcher - Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
nextflow.exception.ConfigParseException: Unknown config attribute `out` -- check config file: /gpfs/data/molecpathlab/development/nextflow-demos/slurm-queue/nextflow.config
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
        at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:249)
        at nextflow.config.ConfigBuilder.validate(ConfigBuilder.groovy:400)
        at nextflow.config.ConfigBuilder.validate(ConfigBuilder.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
        at nextflow.config.ConfigBuilder.merge0(ConfigBuilder.groovy:367)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:184)
        at nextflow.config.ConfigBuilder.buildConfig0(ConfigBuilder.groovy:316)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:176)
        at nextflow.config.ConfigBuilder.buildGivenFiles(ConfigBuilder.groovy:282)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
        at nextflow.config.ConfigBuilder.buildConfigObject(ConfigBuilder.groovy:633)
        at nextflow.config.ConfigBuilder.build(ConfigBuilder.groovy:646)
        at nextflow.script.ScriptRunner.<init>(ScriptRunner.groovy:115)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:223)
        at nextflow.cli.Launcher.run(Launcher.groovy:428)
        at nextflow.cli.Launcher.main(Launcher.groovy:586)

@Crabime
Copy link

Crabime commented Jan 23, 2019

I think there is nothing related Nextflow for your question, it's the matter of Groovy. You can replace println to System.out.println, and i have tried such way and ok.
Maybe there is a default Interceptable implementation of Groovy which will intercept all println method and convert it to System.out.println, only latter one can be accepted by JVM, and out here become a property for ConfigParser and value is null.

@pditommaso
Copy link
Member

pditommaso commented Jan 24, 2019 •
edited
Loading

Functions are not expected to be defined in the config file. This may be possible when #940 will be implemented. So far it's not a supported use case.

@stevekm
Copy link
Contributor Author

stevekm commented Jan 24, 2019 •
edited
Loading

Thanks for the feedback all.

The use-case here is that I want to have dynamic process directives (e.g. process.queue), but I need it defined in the nextflow.config because I only want it used as part of a particular profile (e.g. on our SLURM cluster), and the logic required to implement what I want is getting to be too complex to embed in a simple closure (e.g. read the sinfo and squeue and decide the best queue to use... but only do so every ~15min so as not to flood the scheduler with requests... and with error handling for when the sinfo/squeue commands return errors... and also make log messages about all this happening, etc. etc).

Also @Crabime I am guess I am not familiar enough with Groovy to understand exactly why using println inside the config is causing a problem.

edit: looks like you can actually print to a file just fine from within the nextflow.config, so I think that is also a good option in this case.

import java.text.SimpleDateFormat
SimpleDateFormat timestamp_fmt = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")

queue = {
    def timestamp = timestamp_fmt.format(new Date())
    def queue_log = new File("my_log_file.txt")
    queue_log.append("\n[${timestamp}] some log entry")

    params.queue
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
triage/invalid
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pditommaso @stevekm @Crabime