Spin container image does not seem to work

[rgl]

At https://github.com/rgl/spin-http-ts-example/tree/test-release I'm building an example spin http-ts application and pushing it with spin registry push to https://github.com/rgl/spin-http-ts-example/pkgs/container/spin-http-ts-example.

The application works fine when running locally (using spin up) and in fermyon cloud (using spin deploy), but I cannot make it work with containerd 1.7.12 and https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v2-spin-linux-x86_64.tar.gz.

containerd is configured as:

# containerd config default >/etc/containerd/config.toml
# cat >>/etc/containerd/config.toml <<EOF

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v2"
EOF

I've tried with:

ctr image pull \
    ghcr.io/rgl/spin-http-ts-example:0.1.0-rc5
ctr run \
    --rm \
    --runtime io.containerd.spin.v2 \
    ghcr.io/rgl/spin-http-ts-example:0.1.0-rc5 \
    ctr-spin-http-ts-example

But nothing seems to happen besides the shim being started:

# ps ax | grep shim
   6857 ?        S      0:00 /usr/local/bin/containerd-shim-spin-v2 -namespace default -id ctr-spin-http-ts-example -address /run/containerd/containerd.sock

Can you please help me getting it running in containerd?

[Mossaka]

Yeah to help us debugging this issue, could you please paste the relevant containerd log here?

[rgl]

In the meantime, I was able to successfully run it with crictl as described at https://github.com/rgl/spin-http-ts-example?tab=readme-ov-file#containerd-crictl-usage.

Though, I still cannot get it to run with ctr. I'm now trying to run in detached mode as described at https://github.com/rgl/spin-http-ts-example?tab=readme-ov-file#containerd-ctr-usage. Which, I include the relevant part here:

ctr run \
  --detach \
  --runtime io.containerd.spin.v2 \
  --net-host \
  ghcr.io/rgl/spin-http-ts-example:0.1.0 \
  ctr-spin-http-ts-example

But that ctr run --detach never goes into the background.

In the journal logs of the machine, when I run the above command, it only shows this, which does not seem to have anything relevant:

Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.544134475Z" level=info msg="server listen started"
Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.544180502Z" level=info msg="server started"
Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.544190769Z" level=info msg="Shim successfully started, waiting for exit signal..."
Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.547769174Z" level=info msg="found manifest with WASM OCI image format."
Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.570337404Z" level=info msg="cgroup manager V2 will be used"
Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.570473883Z" level=warn msg="Controller rdma is not yet implemented."
Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.570676898Z" level=warn msg="Controller misc is not yet implemented."
Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.571342576Z" level=warn msg="Controller rdma is not yet implemented."
Jan 22 17:59:27 linux containerd[3646]: time="2024-01-22T17:59:27.571361399Z" level=warn msg="Controller misc is not yet implemented."

I've tried another commands, maybe they can help you help me:

# ctr containers list
CONTAINER                                                           IMAGE                                     RUNTIME                  
ctr-spin-http-ts-example                                            ghcr.io/rgl/spin-http-ts-example:0.1.0    io.containerd.spin.v2    

# ctr tasks list
TASK                                                                PID     STATUS    
ctr-spin-http-ts-example                                            0       UNKNOWN

# ps axwu | grep spin
root        7530  0.0  1.3 1480216 27116 pts/3   Sl+  17:59   0:00 ctr run --detach --runtime io.containerd.spin.v2 --net-host ghcr.io/rgl/spin-http-ts-example:0.1.0 ctr-spin-http-ts-example
root        7556  0.0  0.5  74144 10384 ?        S    17:59   0:00 /usr/local/bin/containerd-shim-spin-v2 -namespace default -id ctr-spin-http-ts-example -address /run/containerd/containerd.sock

The container info:

# ctr container info ctr-spin-http-ts-example

Please see the output at https://gist.github.com/rgl/564a8156d0c22587c363c604c7f26513

I'm still clueless about this.

Please let me know if you want to see anything else.

Thank You!

PS When I do a ctr tasks list, the machine journal logs show:

Jan 22 18:12:24 linux containerd[3646]: time="2024-01-22T18:12:24.761848065Z" level=error msg="get state for ctr-spin-http-ts-example" error="context deadline exceeded: unknown"
Jan 22 18:12:24 linux containerd[3646]: time="2024-01-22T18:12:24.763763533Z" level=warning msg="unknown status" status=0

[jsturtevant]

What CNI are you using for ctr? Is it possible you have an IP conflict for the spin app?

[rgl]

What CNI are you using for ctr? Is it possible you have an IP conflict for the spin app?

PS: i'm playing with this at the https://github.com/rgl/containerd-playground-vagrant/tree/wasm branch. the last commit has the changes required to containerd to use spin shim.

Not sure. I'm installing https://github.com/containernetworking/plugins (see https://github.com/rgl/containerd-playground-vagrant/blob/main/provision-cni-plugins.sh).

Is there a way to list the current or used CNI?

Dunno if this helps:

# nerdctl network ls
NETWORK ID      NAME      FILE
17f29b073143    bridge    /etc/cni/net.d/nerdctl-bridge.conflist
                host      
                none  

Also, the virtual machine only has containerd running. So, all the IPs of that bridge are free. Also port 80 in the host is also free. Also notice that I'm using --net-host when launching the containerspin-shim.

BTW, The virtual machine where I'm trying this, is also having an unusual delayed shutdown time (there are 3 youki processes because I might have tried to launch the thing 3 times):

[jsturtevant]

Also notice that I'm using --net-host when launching the containerspin-shim.

So you are using the host network. To rule out and double check port issues could you see if anything is running on 80? sudo netstat -tunlp | grep 80. It's possible you have run something that failed to clean up properly...

[rgl]

there nothing listening at that port. nor there is any container running. please note that I'm running this inside a VM, and only containerd is running (and typical systemd daemons).

# ss -anp | grep :80

# ps axw | grep -i shim