Understanding internals about how Nextflow runs containers

[alexhermida]

Hello! first thanks for the great tool you've made, I'm still learning most of the configurations and specifics but looks great, and the configuration is good as well.

This is more for understanding how Nextflow uses the underlying system where it runs rather than asking for help 😄 I came a across an issue while I was building some workflows which I'd like to understand better, and I coded a small and reproducible example.

With the following small Python script:

#!/usr/bin/env python3
import shutil
import tempfile
import json


def main():
    with tempfile.NamedTemporaryFile(mode="w") as tmp_file:
        metadata = {"name": "test", "version": "0.1.0"}
        json.dump(metadata, tmp_file) 
    
        result = shutil.move(tmp_file.name, 'metadata.json')

    return result
            
if "__main__" == __name__:
    main()

the following Dockerfile

FROM python:3.12.2-slim

WORKDIR /usr/app/src

COPY test.py ./

CMD ["/bin/bash"]

and the following Nextflow process

process testScript {
    debug true
    container 'python-script-test:latest'
    output:
    path 'metadata.json'
    """

    test.py
    """
}

I noticed that the behaviour running this directly in Docker and in Nextflow with the same Docker image is different.

In both cases, this will generate a metadata.json file, with Nextflow lives within the specific workflow run folder.
But for the case of running it with Nextflow, even though is running the same Docker image, the file it generates is empty whereas in with docker run it has content.

I'd expect this could happen if that code was running in different systems, as the behaviour for shutil.move could change between different systems but not happening running the same docker image.

This can be easily solvable in the code adding a tmp_file.flush() before moving the file, which I think it's more correct anyway.

What puzzles me is why this is happening, and what makes the code behaves differently if the underlying system is the same.

Thanks for your help!

Best.

[pditommaso]

You can see how nextflow run the container checking the docker command in the .command.run created by nextflow for the task

[VasLem]

This is generally not true for deployment on Google Batch.

[bentsherman]

For container-native executors like AWS Batch and Google Batch, you have to look at the batch job definition to see how the container is executed. The cloud executors generally don't give you direct control over the docker command