Pull request jobs are flaking with k3s

[jsturtevant]

Wasmer has failed in e2es in several PR's recently:

#320
#319
#318

Looking at logs it doesn't give a ton of info, looks like the pods are stuck in pending state

fyi @0xE282B0 @dierbei

[jprendes]

I'm not sure it's related to wasmer, I think it's rather flakiness from k3s.

• #1191: wasmer
• #1190: wasmer
• #1189: wasmer
• #1186: wasmedge
• #1185: wasmer
• #1184: wasmedge
• #1182: wasmedge
• #1181: wasmer + wasmedge
• #1175: wasmedge

[jprendes]

I'm hoping #323 can give us further insight into the issue.

• It has already failed once with kind + wasmer, but passed with k3s + wasmer + ubuntu-22.04 in #1195
• It has failed once with k3s + wasmer + (ubuntu-20.04 + ubuntu-22.04) in #1197

[jsturtevant]

Failed with wasmtime on this round: https://github.com/containerd/runwasi/actions/runs/6228704779/job/16906088261?pr=323

[dierbei]

@jsturtevant @jprendes I'll take a look at that.

Right now according to my thoughts it is:

1. look at the Containerd logs in Pending state
2. if all are in Pending state, are there any conflicts (e.g. as in unit tests)

I'm going to go ahead and build a k3s environment and go ahead and test it.

[Mossaka]

Anyone tried reproducing it locally?

[dierbei]

Anyone tried reproducing it locally?

I've been a little busy the last couple of days, I'll give it a try as soon as I can.

[dierbei]

Anyone tried reproducing it locally?

Unfortunately, I tested make test-wasmer 13 times without any problems.

My OS is ubuntu 22.04.

I'm continuing to try.

[0xE282B0]

Hi, I noticed that the first shim with sidecar that comes up after installation is stuck. When deleting it, it starts without problems.
Often it is Wasmer because it is the first one that starts but observed it with Lunatic and WasmEdge as well.
Like in this test report: KWasm/kwasm-node-installer#43 (comment)

[dierbei]

@jprendes @jsturtevant @Mossaka @0xE282B0 I'm experiencing a Pending status, but right now I'm not quite sure what the problem is.

https://github.com/dierbei/runwasi/actions/runs/6248281600/job/16963026527#step:7:28

[0xE282B0]

Not sure if it is the same problem but in my case I get this error message on the linux container whit kubectl describe pod ...:

Error: failed to create containerd task: 
  failed to start shim: start failed: io.containerd.wasmtime.v1: 
    Other("failed to setup namespaces: 
      Other: could not open network namespace /proc/0/ns/net: No such file or directory (os error 2)")
: exit status 1: unknown

Then the sidecar container is in a restart loop.

[dierbei]

I took a closer look at the logs and realized that it seems to be because kubectl apply -f deploy.yaml is executing too fast.

This is because the --all-namespace fetch does not fetch the pods in the k8s kube-system namespace.

What should happen now is that the k8s kube-system pod is not started yet.

sudo bin/k3s kubectl get pods --all-namespaces
NAMESPACE   NAME                        READY   STATUS    RESTARTS   AGE
default     wasi-demo-79d9475fd-p8r7k   0/2     Pending   0          107s
default     wasi-demo-79d9475fd-p7sqz   0/2     Pending   0          107s
default     wasi-demo-79d9475fd-c848k   0/2     Pending   0          107s

[jprendes]

I took a closer look at the logs and realized that it seems to be because kubectl apply -f deploy.yaml is executing too fast.

I think you are right.
I've added some mitigation here, which runs k3s test for all runtimes:

• success: https://github.com/containerd/runwasi/actions/runs/6267907629/job/17021993601?pr=323
• success: https://github.com/containerd/runwasi/actions/runs/6267971481/job/17022065882?pr=323
• failed: https://github.com/containerd/runwasi/actions/runs/6268069976/job/17022318464?pr=323

That is compared to before, where I didn't manage to get a single clean run.

Now, if we check the failed run, it failed because the kube-system pods never came up after 1 minute, even before involving the shims, so it's not a problem with the shims.

[jsturtevant]

some additional logs and investigations here: #346 (comment)

[jsturtevant]

I've opened containerd/rust-extensions#210 since our logs don't have timing info in them

[jsturtevant]

besides #347 seeing the following

cp: cannot stat '/var/lib/rancher/k3s/agent/etc/containerd/config.toml': No such file or directory

[jprendes]

I think the best in that case, as wheel as when k3s's kube-system pods fall to star, is to uninstall k3s and install it again.

That looks like a corrupted k3s installation.

[jprendes]

This has been fixed by #353