Remote container with Singularity

[statiksof]

Is it possible to use singularity containers instead of Docker? We only provide Singularity on our remote machines.

[Chuxel]

Not today. Only Docker is fully supported today. Kubernetes has attach support and Podman works with some limitations.

[statiksof]

@Chuxel thanks for the quick answer. Actually, I though that I can overcome the glibc issue by using containers. Is there any other workaround to make the remote ssh works? I have glibc 2.12.

[Chuxel]

Not today - Node, V8, and even Chrome require recent versions of glibc unfortunately.

[statiksof]

Just an update, I can start the vscode-server inside the singularity container. It starts without any issue. The problem now is how to forward ssh or config the ssh extension locally to use this server.

[oschulz]

I would love to see support for Singularity containers, together with other container systems like Shifter and Charliecloud that are popular in scientific (resp. HPC/HTC) computing environments.

I currently use a very clunky workaround:

• I use a custom SSH script (via VSCode config option"remote.SSH.path") that sets an env var LC__SINGULARITY_IMAGE (different images can be chosen depending on the name of the target system) and then does exec ssh "$@". Env vars starting with LC_* (and only those) are usually forwarded by SSH.

• I modify "$HOME/.vscode-server/bin/*/server.sh", and add change "$ROOT/node ..." to singularity run "$LC__SINGULARITY_IMAGE" $ROOT/node ..."

It works, but it's not very convenient. "server.sh" needs to be modified after every update of vscode-server, and since the server instance gets reused, only one container instance can be active on a given remote system at the same time.

[oschulz]

A first step to make it easier for users to use custom container runtimes might be to add a configuration option like "remote.SSH.runwith". It would allow the user to specify a wrapper script for the vscode server. That user-specified wrapper script would be required to end with exec $@, so that "remote.SSH.runwith" = "run_in_my_container.sh" would result in run_in_my_container.sh "$ROOT/node" ${INSPECT:-} "$ROOT/out/vs/server/main.js" "$@" or run_in_my_container.sh /path/to/server.sh ....

If VS Code would also set some env vars before calling that wrapper script, forwarding information like workspace path, then the wrapper script could decide which container image to run, and how, depending on the application.

[statiksof]

@oschulz thank you for sharing your workaround. I finally got it to work after multiple refinements. For me, forwarding the LC* didn't work. So, I spent most of the time modifying server.sh. I used a default centos 7 image just for testing, there are some warnings when the agent starts, I am thinking to provide dedicated images for that purpose.

[oschulz]

For me, forwarding the LC* didn't work.

Unfortunately, that depends on the SSH server config - I guess some sites disable it, and users don't have control over the server config, of course.

[RobertRosca]

The instructions for using Podman with the remote containers are here: https://code.visualstudio.com/docs/remote/containers#_can-i-use-podman-instead-of-docker

It just suggests pointing the docker executable path to the podman path, and it works as podman and docker have similar CLI commands.

One solution would be to create a wrapper for Singularity which parses and 'translates' the docker commands into Singularity commands. This is similar to the workaround mentioned here int this comment above #3066 (comment) but it's slightly less clunky.

I'll add this to my list of cool things to maybe work on, but probably won't get to it. If anybody here is interested I'm willing to help out though.

[oschulz]

Thanks @RobertRosca , I'll definitely try that, such a wrapper should be possible. Will post if it works.

[david-macleod]

Another workaround (#4474) using a modified version of the remote extension (which will hopefully make it into the default extension soon) allows specifying the singularity exec in .ssh/config e.g.

Host blah
    User blah
    HostName blah
    RemoteCommand "/usr/local/bin/singularity exec $CONTAINER_IMAGE /bin/bash"

[oschulz]

I guess the ideal solution would be a config option for remote-ssh that allows the user to specify an arbitrary command that gets inserted before "$ROOT/node [...]. That way, one could use workspace-dependent containers, etc.

Was trying to prototype something like that, but to my knowledge the source code of the remote-ssh extension isn't available in unmangled form. Should be trivial to do for the VS code team, though.

[oschulz]

Thanks @RobertRosca , I'll definitely try that, such a wrapper should be possible. Will post if it works.

I played a bit with this, but it's tricky because Singularity (and similar scientific container runtimes like Shifter and Charliecloud) are very different from docker - they don't have a background service, they just run the container directly, there's only one process. So things like docker ps -q don't really exist. It's much closer to a glorified chroot, instead.

[RobertRosca]

@oschulz Hmm good point, there aren't analogs for all of the commands you can do in docker in singularity, but most things are possible, singularity actually has decent OCI Runtime Support built in already.

In Singularity you can run containers as a background service, this is called a container instance instance, you can also run containers as OCI compliant instances which gives you JSON-format outputs for the state of the container which have the pattern OCI expects.

So if you start a container with singularity instance start ..., it runs as a background process, and you can then do singularity instance list to get a list of running instances which is pretty similar to docker ps -q.

Played around a bit with the vscode remote docker extension and the calls it makes are usually like:

[254405 ms] Start: Run: docker context show
[254423 ms] Start: Run: docker context show
[254442 ms] Start: Run: docker context inspect --format {{.Name}}
[254464 ms] Start: Run: docker context inspect --format {{.Name}}
[254496 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[254531 ms] 1.41
[254532 ms] Start: Run: docker -v
[254561 ms] Start: Run: docker events --format {{json .}} --filter event=create --filter event=destroy --filter event=die --filter event=health-status --filter event=kill --filter event=pause --filter event=rename --filter event=resize --filter event=restart --filter event=start --filter event=stop --filter event=unpause --filter event=update --filter event=exec_create
[254568 ms] Start: Run: docker ps -q -a
[254616 ms] Start: Run: docker inspect --type container  # Ran on all containers

That's just done as the setup, now when you select a container to attach to it runs:

[299720 ms] Start: Run: docker inspect --type container aba69f4...
[299760 ms] Start: Run: docker start #  Ran on the container you selected to attach to
[301149 ms] Start: Run: docker inspect --type container aba69f4...

On the 'local' client, and then the following runs on the 'remote' client:

[3 ms] Remote-Containers 0.161.0 in VS Code 1.54.0-insider (a699ffaee62010c4634d301da2bbdb7646b8d1da).
[9 ms] Start: Run: docker context show
[53 ms] Start: Run: docker context inspect --format {{.Name}}
[84 ms] Start: Run: docker context show
[87 ms] Start: Resolving Remote
[88 ms] Setting up container: /vscode_test_container
[89 ms] Start: Run: docker inspect --type container /vscode_test_container
[119 ms] Start: Run: docker context inspect --format {{.Name}}
[133 ms] Start: Check Docker is running
[133 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[170 ms] Server API version: 1.41
[171 ms] Start: Inspecting container
[171 ms] Start: Run: docker inspect --type container aba69f4...
[198 ms] Start: Run: docker exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=8033a654-1097-4d65-9b42-b6472cfb27df1613987151778 aba69f4... /bin/sh
[201 ms] Start: Run in container: uname -m
[231 ms] Start: Run in container: cat /etc/passwd

You can place various additional flags to be passed to this in a runArgs variable, either with docker-compose or devcontainer files, which would let you set some additional flags for Singularity (e.g. containall).

I've made a (empty) repo here: https://github.com/RobertRosca/singularity-vscode-devcontainer

And an initial dev issue collecting some info: RobertRosca/singularity-vscode-devcontainer#2

[oschulz]

In Singularity you can run containers as a background service

Hm, good point, have to look into that (and check if it requires permissions that plain users on scientific compute clusters might not get, they tend to be restrictive).

[RobertRosca]

Hm, good point, have to look into that (and check if it requires permissions that plain users on scientific compute clusters might not get, they tend to be restrictive).

Should be fine, I've only glanced through the code but as far as I understand it instances work in a pretty simple way. When you start an instance it starts a container as a background process and then (by default) writes the PID and some other info to the users home directory (e.g. ~/.singularity/instances/sing/$HOSTNAME/$USERNAME) in JSON files.

After that all the commands that interact with instances (e.g. list, stop, log, run, attach, etc...) look through that directory to find the PID/other relevant info, and use that PID to communicate with the background container. As long as the user can start singularity containers and the instance JSON files can be written then it should work fine. I currently use singularity instances on the cluster at work and haven't had any issues with them.

[oschulz]

Great, thanks for the details!

[PavelSosin-320]

Hi Robert, It is very interesting! I don't understand what is the goal of the herd race toward already obsolete Client-Server architecture in the development environment. It requires extra hardware, extra software, extra network equipment, and extra configuration per user without any obvious benefits. It is possible and very easy to run any dockerless OCI runtime locally using natively on the Linux Desktop or on the Windows using WSL. The total memory footprint doesn't exceed 0.5 Gb plus 2Gb taken by WSL service in the case of Buildah.OCI runtimes and OCI image repositories are not Docker's monopoly and can be found today almost in every corner of Global Cloud infrastructure free of charge in many cases. Could you share links to Singularity documentation, Installation guide, and Hardware&Software requirements?

…

On Mon, Feb 22, 2021 at 4:53 PM Oliver Schulz ***@***.***> wrote: Great, thanks for the details! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3066 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM7YVKFUU5ENCFLWAIYMZJLTAJVVBANCNFSM4NNB5SOQ> .

[oschulz]

Should be fine, I've only glanced through the code but as far as I understand it instances work in a pretty simple way.

This might indeed work nicely with singularity. The problem is that other relevant container runtimes like Shifter and Charliecloud do (AFAIK) not support instances.

In the end, a simple option in VS-code that allows for an arbitrary prefix commands before the vscode-server starts (so like it was put before "$ROOT/node [...] in the server start script) would probably be the most flexible solution.

[mathiaswagner]

I used a separate ssh-key and in the remote host's authorized_keys I prefix that special public key key with

command="singularity run image.sif $SSH_ORIGINAL_COMMAND"

This seems to work.

On my local host I have in my .ssh/config

Host vscoderemote
HostName remote
IdentityFile /path/to/ssh-key-for-code

[oschulz]

Neat!

[DraTeots]

@PavelSosin-320 I don't understand what is the goal of the herd race toward already obsolete Client-Server architecture in the development environment

Well... You don't understand =) Since singularity is in play, it is highly probably a scientific environment, where software part is usually understaffed and software consists of crappy monster packages like CERN ROOT. Some software from the stack is new and demanding (c++17, newest compilers, look at ACTS tracking), some other software is 50 years old, all developers are passed away and nobody touches it as it works but requires old versions of Fortran. End everything is wired together with lots of environment variables, hacks and patches. Building such stack from scratch make take days of fails and gotchas. Images are 7-8GB stuffed of scientific dependencies where "everything works, don't touch". Understaffed maintainers cannot provide sane stably working installation solutions other than containers. Please, try to build something like sPHENIX software stack and then come back to this issue as new person who now understands all.

Docker container support in VSCode is superb and it is pleasure to develop in the container. But in many scientific communities docker is given up in favor of Singularity. Such plugin would greatly improve situation for many scientists.

[oschulz]

I don't understand what is the goal of the herd race toward already obsolete Client-Server architecture in the development environment [...]It requires extra hardware

It's not so much about plain software development for generic applications (write on your local system, test on your local system, deploy to the cloud or so).

As it turns out, VS code is an great tool for scientific coding and analysis. So code is written, but it's constantly changed, plots are made, data is analysed, code is adapted, and to on. That often requires systems or compute clusters with capabilities far beyond a laptop or desktop machine, and indeed sometimes special hardware like state of the art GPUs, etc.

[oschulz]

Ah, I forgot - also, in typical scientific computing environments (central home directories, no root privileges for users on desktops or severs) Docker can't really be used safely, so Singularity & friends (locally and remote) have become very common.

[PavelSosin-320]

Singularity, Docker, Podman, Buildah, Kubernetes & friends share the same Image standard - OCI. The Image built using one of the listed tools can be run in any other tool almost without effort. The OCI runtime can run on any scale, from Podman Linux CLI to Kubernetes clusters. The client-Server architecture is the personal developer choice, there is no herd race here today. Every developer can make the choice to use everything on a single laptop or use a Cloud-based development pipeline. Base docker images for NVidia GPU are available from NVidia, they can be used on almost any hardware with NVidia GPU, from the laptop to Cloud infrastructure like AWS, GCP. So Creation of the Docker image is a one-time investment that makes software portable forever. My choice today is Podman at home and RedHat dev pipeline for big projects with complex integration.
Tlalks about Docker safety is bullshit today, Banks and Government defense bodies use Kubernetes because it offers better security frameworks.

[PavelSosin-320]

Oliver, the Podman development is lead by the person with the nickname "SELinux man". So, you can use it safely or report your doubts directly.

…

On Sun, May 23, 2021 at 10:02 AM Oliver Schulz ***@***.***> wrote: Ah, I forgot - also, in typical scientific computing environments (central home directories, no root privileges for users on desktops or severs) Docker can't really be used safely, so Singularity & friends (locally and remote) have become very common. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3066 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM7YVKDOVC3JOY35V6WPNHTTPCR7PANCNFSM4NNB5SOQ> .

[DebajyotiS]

@patgo25, did you also have a proxy jump? The issue is it works normally on a terminal. So it really just looks like VSCode doesn't recognise my remoteCommand

[DebajyotiS]

@oschulz Interestingly, if I explicitly set useLocalServer to true, I can't seem to use ssh at all through vscode (note, that I had this checked in the settings).

It is only after setting it to false that I can ssh in.

    "editor.inlineSuggest.enabled": true,
    "remote.SSH.connectTimeout": 30,
    "remote.SSH.useLocalServer": false,
    "remote.SSH.enableRemoteCommand": true,
    "remote.SSH.logLevel": "trace",

[oschulz]

if I explicitly set useLocalServer to true, I can't seem to use ssh at

I have to admit I don't know what's going on there ...

[DebajyotiS]

quick question on your custom run script that you use to export XDG runtime: is that path dependent?
Can I keep it in my home/dir
and then just have remoteCommand be /home/dir/script
or am I misunderstanding something?

[gipert]

Sorry I cannot help a lot since I don't know Windows either... But, one additional piece of information: I had set up the thing on a student's Windows laptop a couple of months ago and it stopped working yesterday, after an Remote.SSH extension update (suspicious...). Might be related to your issue, I did not investigate further though.

[ravusairam]

@DebajyotiS I am also facing the same issue. I am able to ssh using windows powershell and it goes to the container on my remote machine. But, it doesn't work on vscode..Have you resolved the issue?

[DebajyotiS]

@ravusairam , Unfortunately not. See issues 6598 and 6086

[adinriv]

@DebajyotiS have you tried to set up a RemoteCommand that does a resource allocation (for instance srun or salloc) first and then spins the singularity container within that node?

I'm trying something on these lines and I'm seeing port forwarding errors. Yet the solution with singularity as in your example works.

[DebajyotiS]

@adinriv I haven't. I usually request a node from the terminal first, and then modify the config file accordingly which is a lot of steps for it to not work ultimately. Are you using windows or linux?

[adinriv]

I'm on linux. I wanted to spin VSCode within the container to ease the job of running jupyter notebooks within the containers.

If not, VSCode doesn't see the kernels inside it and do not work as well (at least I haven't made it work).

[MiguelRodo]

To note, I've been able to run VSC inside a singularity container on a compute node on an HPC using the Visual Studio Code Server. All I did was add the install command (available at the link) to the image definition file and run singularity run <image> code-server in a compute node terminal.

[oschulz]

Oh, that's indeed an interesting new option!

[casjogreen]

Miguelrodo did you was able to install ms python extension?

[casjogreen]

miguelrodo, the complete question:
Did you install the ms python extension in your vscode-server inside your singularity image? I was able to install vscode desktop and run inside the singularity image. Still, I couldn't install the ms-python extension to debug with it inside the singularity image.

[casjogreen]

To note, I've been able to run VSC inside a singularity container on a compute node on an HPC using the Visual Studio Code Server. All I did was add the install command (available at the link) to the image definition file and run singularity run <image> code-server in a compute node terminal.

Did you install the ms python extension in your vscode-server inside your singularity image? I was able to install vscode desktop and run inside the singularity image. Still, I couldn't install the ms-python extension to debug with it inside the singularity image.

[MiguelRodo]

Hi @casjogreen. Nope, sorry, I did not - the images I've used were for R data analyses. For what it's worth, I didn't install any VSC extensions inside the image itself or during the image build. I think VSC extensions are installed to ~/.vscode-server-insiders, so the VSC server should be able to install to there after the image is created and then run them. But I don't really know the nitty gritty of extensions, so that could be inaccurate or not the whole story.

[casjogreen]

Hi @casjogreen. Nope, sorry, I did not - the images I've used were for R data analyses. For what it's worth, I didn't install any VSC extensions inside the image itself or during the image build. I think VSC extensions are installed to ~/.vscode-server-insiders, so the VSC server should be able to install to there after the image is created and then run them. But I don't really know the nitty gritty of extensions, so that could be inaccurate or not the whole story.

Thank you

[ChristianEschen]

Hi,
I am trying to install the IDE vscode inside a singularity container. To do so I have specified a Dockerfile as:

FROM nvcr.io/nvidia/pytorch:21.12-py3
ARG DEBIAN_FRONTEND=noninteractive
RUN apt update
RUN apt install software-properties-common apt-transport-https wget -y &&
wget -q https://packages.microsoft.com/keys/microsoft.asc -O- | apt-key add -
RUN add-apt-repository "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main"
RUN apt update -y
RUN apt install code -y

From the dockerfile I build the vscode_remote_development.sif
The next step is to run the singularity container: $singularity shell --nv -B vscode_remote_development.sif
I can start vscode in the singularity container with: $code .
Now, I need to manually install the python extension (I install it manually using "Install from VSIX.." using the GUI).
The installation succeeds but inside the run and debug tab the panel is empty (see attached image).

[casjogreen]

ChristianEschen The same happened to me, even though I tried to install the Python extension for vscode while creating the singularity image.

[R4dicalEdward]

Hi,
my workaround currently is to build a container with full linux distribution (ubuntu18.04) and install VS Code as well.

Bootstrap: docker
From: nvidia/cuda:11.3.1-devel-ubuntu18.04

I downloaded the . deb file first and made it available via

%files 
 code_1.73.1-1667967334_amd64.deb

but I think the wget-approach should lead to the same outcome.

In my %post-section I install everything I need (e.q. nano, git etc.) and then I install VS Code via:

apt-get install ./code_1.73.1-1667967334_amd64.deb -y

and then build my python env.

I'm able to open VS Code via:

singularity run /path/to/image/image.sif code

and install all extensions in the usual way via GUI. They will be installed into home/username/.vscode on your node.

This works just fine in my case.

NOTE: To open VS Code on compute node you need to connect to that node with X11 forwarding enabled (ssh -X).

Hope that helped.

[data-panda]

Has anyone tried to install the python extension to enable a debug session within the container? I tried the steps listed using the RemoteCommand method which worked fine but only suffices to edit and run the code. I installed the python extension but the activation of it fails with the following error:

Any help will be appreciated!

Edit1:
This is what I can see while I try to use developer tools and look at the console to figure out what is causing the problem. To me it seems like there is "socket" related error but I am far from typing/understanding a single line of typescript code to say it conculsively. Image attached below:

Edit2:
This turned out more to be a bug related to the VScode Insiders version that I was using. I figured it out when I was greeted with the same error while debugging a python code on my local machine. I tried running the same steps with VSCode stable and the python extension installed fine and got successfully activated to debug in all scenarios: local, remote, remote singularity container.

[dcat52]

Hello @Chuxel
There seems to have been substantial development and ways people have gotten things to at least partially work. Also since the creation of this issue, Singularity has been accepted into the Linux Foundation community under the name Apptainer (https://www.linuxfoundation.org/press/press-release/new-linux-foundation-project-accelerates-collaboration-on-container-systems-between-enterprise-and-high-performance-computing-environments).

How can we work towards official support for Apptainer with this extension? If needing community-driven development, could you provide some information as to what developmentally needs to be completed to meet this goal?

[Chuxel]

The dev container spec is now open at https://containers.dev, and the reference implementation is at https://github.com/devcontainers/cli. The CLI is used under the hood by the Remote - Containers / Dev Containers extension and would also generally be by other dev container spec supporting tools. The vast majority of the VS Code server is in the https://github.com/microsoft/vscode repo as well. Those would be the best places to look to contribute. Also note that the Remote - Tunnels extension has come out recently to try to avoid SSH routing related issues.

[perrylets]

Has there been any progress on this?

[oschulz]

Has there been any progress on this?

The "RemoteCommand trick" works quite well, I'm using it regularly together with a little Apptainer-Wrapper called cenv - here a little HowTo: https://github.com/oschulz/container-env/blob/main/README-VSCode.md

It would be really nice not to have to set remote.SSH.serverInstallPath for every system/container-image. An environment variable to override remote.SSH.serverInstallPath would be really helpful.

[perrylets]

The "RemoteCommand trick" works quite well, I'm using it regularly together with a little Apptainer-Wrapper called cenv - here a little HowTo: https://github.com/oschulz/container-env/blob/main/README-VSCode.md

I have 25 hosts and 6 images, so it would take a while to do it. I will try it later when I have more time, though.

Besides that, I will only need to run one container per machine (at the same time), maybe there's a solution that's easier for a case like mine.

Edit to clarify:

I have more than one remote I can access, but I will only ever use 1 image per remote.
I can use any image on any remote, but I can never use 2 images at the same time in the same remote. I can have n images (they can be different images or the same image) running in n separate remotes at the same time.