Development environment guide #5486

christopher-hakkaart · 2024-11-08T13:22:02Z

Add a new development environment page with the recommended minimal setup. Add tabs to help split Windows/macOS/Linux.

Development environment

VS Code
Extensions
- Nextflow
- Remote development
Docker
Git
WSL
Dev Containers
- Development Containers extension
- Create and run a dev container

@netlify /developer-env

Signed-off-by: Christopher Hakkaart <[email protected]>

netlify · 2024-11-08T13:22:18Z

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	`c839def`
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/6732659bde1c000008e6061c
😎 Deploy Preview	https://deploy-preview-5486--nextflow-docs-staging.netlify.app/developer-env
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

ewels

Very nice!

docs/developer-env.md

ewels · 2024-11-08T13:35:10Z

docs/developer-env.md

+
+# Environment setup
+
+Setting up a Nextflow development environment is a prerequisite for creating, testing, and optimizing data analysis pipelines. The steps below outline recommended tools for setting up an optimal Nextflow development environment.


A brief TOC here would help I think, not sure if that's non-standard for these docs though.

docs/developer-env.md

pditommaso · 2024-11-08T13:46:27Z

docs/developer-env.md

+### Development Containers extension
+
+The VS Code Dev Containers extension lets you use a container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of VS Code's full feature set.
+
+The Dev Containers extension is included as a part of the [Remote Development extension pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack). See {ref}`remote-development-ext` for installation instructions.
+
+### Create and run a dev container
+
+A `devcontainer.json` file in your project directory instructs VS Code how to access, create, and configure a development container. It can be used to run an application or to separate tools, libraries, or runtimes.
+
+The `devcontainer.json` can be used to:
+
+- Install additional tools in the container.
+- Automatically install extensions.
+- Forward or publish additional ports.
+- Set runtime arguments.
+- Reuse or extend your existing Docker Compose setup.
+- Add more Advanced container configuration.
+
+To create a dev container with an existing image:
+
+1. Create `.devcontainer/devcontainer.json` in the root of your project.
+1. Add an image with the Nextflow tooling and VS Code customizations to the `devcontainer.json`. For example:
+
+    ```json
+    {
+        "name": "Nextflow Dev Container",
+        "image": "nfcore/gitpod:latest",
+        "remoteUser": "vscode",
+        "runArgs": ["--privileged"],
+
+        // Configure tool-specific properties.
+        "customizations": {
+            // Configure properties specific to VS Code.
+            "vscode": {
+                // Set *default* container specific settings.json values on container create.
+                "settings": {
+                    "python.defaultInterpreterPath": "/opt/conda/bin/python"
+                },
+
+                // Add the IDs of extensions you want installed when the container is created.
+                "extensions": ["ms-python.python", "ms-python.vscode-pylance", "nf-core.nf-core-extensionpack", "nextflow.nextflow"]
+            }
+        }
+    }
+    ```
+
+    :::{note}
+    Instead of using a prebuilt image, a custom Dockerfile may also live in the `.devcontainer` folder. You can replace the image property in `devcontainer.json` with dockerfile and utilize the custom container. See [Create a Dev Container](https://code.visualstudio.com/docs/devcontainers/create-dev-container) for more information.
+    :::
+
+1. Enter **Dev Containers: Reopen in Container** in the VS Code Command Palette and reopen your project. You should now see the name of the container ("Nextflow Dev Container" if using the above example) in the bottom left corner of VS Code.
+
+:::{note}
+Dev Containers can also be used by GitHub Codespaces in VS Code or the browser. See [GitHub Codespaces](https://code.visualstudio.com/docs/remote/codespaces) for more information.


not sure this makes much sense in the context of nextflow workflow in which each process can have its own container. Our best practice is Conda + Wave for container reproducibility and portability

Fair point. I'm not committed to keeping this. It's very thin on details and can get complicated quickly. I'll comment it out.

I want to push back on p's argument. The intent of a devcontainer isn't to have a single container with all processes for your pipeline (I agree conda + wave is best-practice for that). Instead the devcontainer has all of the pre-requisites like conda, nextflow, nf-core, nf-test, etc installed and updated to their latest versions.

Devcontainers works really well for the following use cases:

People who are new to developing in Nextflow and don't have any of these tools installed on their computer

Folks who only occasionally develop Nextflow pipelines and don't want all of those tools cluttering up their system when they're not working on nextflow stuff.

Teams who want to ensure everyone is working off the same version of all the base tools.

Especially with devcontainers being supported by both gitpod and Codespaces there's also major benefits to being able to boot up into a repo-specific cloud environment quite easily. (part of why I'd love Data Studios to support devcontainers.json down the line too).

Yeah agree with this. I think that devcontainers for the pipeline development environment is not incompatible with using conda / wave etc for actually running the pipeline.

Signed-off-by: Christopher Hakkaart <[email protected]>

pditommaso · 2024-11-08T16:22:09Z

Not sure this is material for nextflow documentation, this is more material for tutorial or blog post

kenibrewer · 2024-11-09T00:24:57Z

Not sure this is material for nextflow documentation, this is more material for tutorial or blog post

I've had four people in the last week (1 at summit and 3 at ASHG) tell me that they had a hard time figuring out how to install Nextflow from the documentation. Two of them gave up and quit and never used Nextflow, one of them needed personal help from me, and one was running an ancient version of nextflow that came pre-installed as a workaround.

Those user experiences make me think this is a very important addition to the main docs. I also think it would be better featured prominently as one of the first pages new users encounter on the docs page. A separate tutorial page or blog post will be harder to find and won't get continued ongoing attention to make sure we keep it up to date.

Signed-off-by: Christopher Hakkaart <[email protected]>

pditommaso · 2024-11-11T11:40:41Z

I've had four people in the last week ..

Understand and it makes sense. However, I'm contrary having dev containers in this context. Provided it could help, it should not make part of the best practices we are promoting for container usage both for dev and production deployment

justinegeffen

This is awesome!

Please don't be alarmed at the number of comments - the content is great - just a few editorial tweaks here and there. Happy to workshop the feedback if it's unclear. :)

docs/developer-env.md

justinegeffen · 2024-11-11T19:10:25Z

docs/developer-env.md

+
+### Development Containers extension
+
+The VS Code Dev Containers extension lets you use a container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of VS Code's full feature set.


We've said "full-featured" 3 times in the past few paragraphs. Maybe we can remove these two or use a different adjective? I'm coming up blank right now though!

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Signed-off-by: Christopher Hakkaart <[email protected]>

ewels

Content is looking great, though I think that the VSCode part can be condensed a bit to help with general readability and maybe yank the devcontainers into a blog post. Otherwise, comments are very minor.

Great work! 👏🏻

ewels · 2024-11-13T22:08:29Z

docs/developer-env.md

+- [Apptainer/Singularity](https://marketplace.visualstudio.com/items?itemName=onnovalkering.vscode-singularity) - Provides syntax highlighting for Apptainer/Singularity definition files.
+- [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker) - A basic spell checker that works well with camelCase code.
+- [Docker](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker) - Makes it easy to create, manage, and debug containerized applications.
+- [EditorConfig](https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig) - Support for EditorConfig project files for code standardisation.
+- [indent-rainbow](https://marketplace.visualstudio.com/items?itemName=oderwat.indent-rainbow) - Highlight indentation levels, especially if they're inconsistent.
+- [Nextflow](https://marketplace.visualstudio.com/items?itemName=nextflow.nextflow) - Nextflow language support.
+- [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) - Code formatter using Prettier.
+- [Rainbow CSV](https://marketplace.visualstudio.com/items?itemName=mechatroner.rainbow-csv) - Highlight columns in .CSV files in different colors.
+- [Ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) - An extremely fast Python linter and code formatter, written in Rust.
+- [Todo Tree](https://marketplace.visualstudio.com/items?itemName=Gruntfuggly.todo-tree) - Show TODO, FIXME, etc. comment tags in a tree view.
+- [YAML](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) - YAML Language Support by Red Hat, with built-in Kubernetes syntax support.
+- [Markdown Extended](https://marketplace.visualstudio.com/items?itemName=jebbs.markdown-extended) - Gives Markdown previews, including admonitions.


I'm slightly wary of actually listing them out like this, as it's the kind of thing that could get stale quite quickly. Would prefer something more generic like "a number of useful extensions that come recommended from community members, such as ..." then like 2 or 3 specific examples.

I can see your point. I'll shrink this (and the other extensions) down

docs/developer-env.md

ewels · 2024-11-13T22:10:56Z

docs/developer-env.md

+
+See {ref}`vs-code-page` for more information about the Nextflow extension.
+
+````{tabs}


Can we have just one of these for "installing VSCode extensions"? Feels bulky to have essentially the identical content repeated for each set of VSCode extensions.

At the moment the VSCode section is like, most of the page. I think making each subsection shorter would help.

ewels · 2024-11-13T22:12:09Z

docs/developer-env.md

+- [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) - Work with a separate toolchain or container based application by opening any folder mounted into or inside a container.
+- [WSL](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl) - Get a Linux-powered development experience from the comfort of Windows by opening any folder in the Windows Subsystem for Linux.


I'm a bit confused by these as they're not really remote..? Maybe worth addressing this given the heading?

docs/developer-env.md

ewels · 2024-11-13T22:15:20Z

docs/developer-env.md

+
+(wsl)=
+
+## Windows Subsystem for Linux


This still feels to me to be more part of the Nextflow installation, rather than Dev Environment.

Personally I'd rather remove it from this page entirely.

ewels · 2024-11-13T22:17:23Z

docs/developer-env.md

+
+See [Set up a WSL development environment](https://learn.microsoft.com/en-us/windows/wsl/setup/environment) for more information.
+
+## Development Containers


I wonder if this section would be better off as a blog post that we can point to in an admonition from this page.

(a) This part feels quite a lot more detailed than the rest of the page and (b) I feel that devcontainers are more of a "this can be a nice thing" rather than a requirement. I, and I guess the majority, of Nextflow developers have never used this.

I think it's emerging but not especially common. I'll drop it from this page and add it to the list of possible blog posts (maybe something @kenibrewer would want to write)

We also have a "Tutorials" section in the Nextflow docs as a temporary home for tutorial-like content until we have something better on seqera.io, so if you want something more permanent than a blog post you can put it there.

Blog post or tutorial would be more appropriate for dev containers for now IMO

Okay, great. I think it's agreed that it should be removed from the PR. I'll keep the text in case we can recycle it for a blog or tutorial. Depending on how it's pitched, it could be wrapped up into a tutorial or blog. Either way, we can put it aside for now.

christopher-hakkaart added 12 commits November 5, 2024 14:29

First draft developer environment

33103ac

Signed-off-by: Christopher Hakkaart <[email protected]>

Merge branch 'nextflow-io:master' into docs-user-guide

1d199a2

Adding WSL skeleton and fixes

a1dfb9f

Signed-off-by: Christopher Hakkaart <[email protected]>

Adding dev containers section

633f71b

Signed-off-by: Christopher Hakkaart <[email protected]>

Finish draft

ef32325

Signed-off-by: Christopher Hakkaart <[email protected]>

Fix minor formatting issue

49b7f6f

Signed-off-by: Christopher Hakkaart <[email protected]>

Improve clarity

493b2e9

Signed-off-by: Christopher Hakkaart <[email protected]>

Improve clarity again

9a5a1fd

Signed-off-by: Christopher Hakkaart <[email protected]>

Fix bullet points

72cb41d

Signed-off-by: Christopher Hakkaart <[email protected]>

Try new phrasing

57e0024

Signed-off-by: Christopher Hakkaart <[email protected]>

Try new phrasing

d75d3bb

Signed-off-by: Christopher Hakkaart <[email protected]>

Merge branch 'nextflow-io:master' into docs-user-guide

6a58a07

ewels reviewed Nov 8, 2024

View reviewed changes

pditommaso reviewed Nov 8, 2024

View reviewed changes

christopher-hakkaart added 2 commits November 8, 2024 15:46

Incorporate comments from draft review

ba8175d

Signed-off-by: Christopher Hakkaart <[email protected]>

Incorporate comments from draft review

89d87e3

Signed-off-by: Christopher Hakkaart <[email protected]>

Uncomment dev containers and add Gitpod note

9cb8f7b

Signed-off-by: Christopher Hakkaart <[email protected]>

justinegeffen reviewed Nov 11, 2024

View reviewed changes

christopher-hakkaart and others added 8 commits November 11, 2024 20:46

Update docs/developer-env.md

fdd9ca2

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

73b66a7

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

c4d733e

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

a7cc667

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

f3b10be

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

6686425

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

55db7ab

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

9f9147e

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

christopher-hakkaart and others added 11 commits November 11, 2024 20:48

Update docs/developer-env.md

2a7ebc2

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

177f2a9

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

7304074

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

1a8e1e5

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

e6349f1

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

b3c00c5

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

d046d2f

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

e4832f7

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Update docs/developer-env.md

2dd42b9

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Apply suggestions from code review

461a157

Co-authored-by: Justine Geffen <[email protected]> Signed-off-by: Christopher Hakkaart <[email protected]>

Add suggestions from review

c839def

Signed-off-by: Christopher Hakkaart <[email protected]>

bentsherman added dependencies Pull requests that update a dependency file docs and removed dependencies Pull requests that update a dependency file labels Nov 13, 2024

ewels reviewed Nov 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Development environment guide #5486

Development environment guide #5486

christopher-hakkaart commented Nov 8, 2024 •

edited by ewels

Loading

netlify bot commented Nov 8, 2024 •

edited

Loading

ewels left a comment

ewels Nov 8, 2024

pditommaso Nov 8, 2024

christopher-hakkaart Nov 8, 2024

kenibrewer Nov 8, 2024

ewels Nov 8, 2024

pditommaso commented Nov 8, 2024

kenibrewer commented Nov 9, 2024

pditommaso commented Nov 11, 2024

justinegeffen left a comment

justinegeffen Nov 11, 2024

ewels left a comment

ewels Nov 13, 2024

christopher-hakkaart Nov 14, 2024

ewels Nov 13, 2024

ewels Nov 13, 2024

ewels Nov 13, 2024

ewels Nov 13, 2024

christopher-hakkaart Nov 14, 2024

bentsherman Nov 14, 2024

christopher-hakkaart Nov 14, 2024


		# Environment setup

		Setting up a Nextflow development environment is a prerequisite for creating, testing, and optimizing data analysis pipelines. The steps below outline recommended tools for setting up an optimal Nextflow development environment.


		### Development Containers extension

		The VS Code Dev Containers extension lets you use a container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of VS Code's full feature set.


		See {ref}`vs-code-page` for more information about the Nextflow extension.

		````{tabs}

		- [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) - Work with a separate toolchain or container based application by opening any folder mounted into or inside a container.
		- [WSL](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl) - Get a Linux-powered development experience from the comfort of Windows by opening any folder in the Windows Subsystem for Linux.


		See [Set up a WSL development environment](https://learn.microsoft.com/en-us/windows/wsl/setup/environment) for more information.

		## Development Containers

Development environment guide #5486

Are you sure you want to change the base?

Development environment guide #5486

Conversation

christopher-hakkaart commented Nov 8, 2024 • edited by ewels Loading

netlify bot commented Nov 8, 2024 • edited Loading

✅ Deploy Preview for nextflow-docs-staging ready!

ewels left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pditommaso commented Nov 8, 2024

kenibrewer commented Nov 9, 2024

pditommaso commented Nov 11, 2024

justinegeffen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewels left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christopher-hakkaart commented Nov 8, 2024 •

edited by ewels

Loading

netlify bot commented Nov 8, 2024 •

edited

Loading