Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to 1.14 breaks some docker container (os: debian) #1400

Closed
Skiepp opened this issue Jan 25, 2024 · 11 comments · Fixed by #1405
Closed

Upgrade to 1.14 breaks some docker container (os: debian) #1400

Skiepp opened this issue Jan 25, 2024 · 11 comments · Fixed by #1405

Comments

@Skiepp
Copy link

Skiepp commented Jan 25, 2024 •
edited
Loading

User Story
After upgrading crun to 1.14 some containers are not able to start.

  • OS: Debian 11
  • Docker Version: docker-ce:5:25.0.0-1~debian.11~bullseye

Steps to reproduce:

  • apt-get update -y crun
  • wait for containers to die

Workaround:

  • rollback to 1.13

Output
Output of docker start attached:

# docker start traefik
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: bpf create `0: (61) r2 = *(u32 *)(r1 +0)
1: (54) w2 &= 65535
2: (61) r3 = *(u32 *)(r1 +0)
3: (74) w3 >>= 16
4: (61) r4 = *(u32 *)(r1 +4)
5: (61) r5 = *(u32 *)(r1 +8)
6: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_w=inv2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
7: (55) if r4 != 0x5 goto pc+3
 R1=ctx(id=0,off=0,imm=0) R2_w=inv2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_w=inv5 R5_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
8: (55) if r5 != 0x2 goto pc+2

from 8 to 11: R1=ctx(id=0,off=0,imm=0) R2=inv2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
11: (55) if r2 != 0x2 goto pc+3
last_idx 11 first_idx 8
regs=4 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_w=inv5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=4 stack=0 marks
last_idx 7 first_idx 0
regs=4 stack=0 before 7: (55) if r4 != 0x5 goto pc+3
regs=4 stack=0 before 6: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 5: (61) r5 = *(u32 *)(r1 +8)
regs=4 stack=0 before 4: (61) r4 = *(u32 *)(r1 +4)
regs=4 stack=0 before 3: (74) w3 >>= 16
regs=4 stack=0 before 2: (61) r3 = *(u32 *)(r1 +0)
regs=4 stack=0 before 1: (54) w2 &= 65535
regs=4 stack=0 before 0: (61) r2 = *(u32 *)(r1 +0)
12: (55) if r4 != 0x88 goto pc+2
last_idx 12 first_idx 8
regs=10 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=10 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=10 stack=0 marks
last_idx 7 first_idx 0
regs=10 stack=0 before 7: (55) if r4 != 0x5 goto pc+3
regs=10 stack=0 before 6: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 5: (61) r5 = *(u32 *)(r1 +8)
regs=10 stack=0 before 4: (61) r4 = *(u32 *)(r1 +4)
15: (55) if r2 != 0x2 goto pc+4
last_idx 15 first_idx 8
regs=4 stack=0 before 12: (55) if r4 != 0x88 goto pc+2
regs=4 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=4 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
16: (55) if r4 != 0x5 goto pc+3
last_idx 16 first_idx 8
regs=10 stack=0 before 15: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 12: (55) if r4 != 0x88 goto pc+2
regs=10 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=10 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=10 stack=0 marks
17: (55) if r5 != 0x1 goto pc+2

from 17 to 20: R1=ctx(id=0,off=0,imm=0) R2=inv2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
20: (55) if r2 != 0x2 goto pc+4
last_idx 20 first_idx 20
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=4 stack=0 marks
last_idx 17 first_idx 8
regs=4 stack=0 before 17: (55) if r5 != 0x1 goto pc+2
regs=4 stack=0 before 16: (55) if r4 != 0x5 goto pc+3
regs=4 stack=0 before 15: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 12: (55) if r4 != 0x88 goto pc+2
regs=4 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=4 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
21: (55) if r4 != 0x1 goto pc+3
last_idx 21 first_idx 20
regs=10 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=10 stack=0 marks
last_idx 17 first_idx 8
regs=10 stack=0 before 17: (55) if r5 != 0x1 goto pc+2
regs=10 stack=0 before 16: (55) if r4 != 0x5 goto pc+3
regs=10 stack=0 before 15: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 12: (55) if r4 != 0x88 goto pc+2
regs=10 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=10 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=10 stack=0 marks
25: (55) if r2 != 0x2 goto pc+4
last_idx 25 first_idx 20
regs=4 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
26: (55) if r4 != 0x1 goto pc+3
last_idx 26 first_idx 20
regs=10 stack=0 before 25: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=10 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=10 stack=0 marks
30: (55) if r2 != 0x2 goto pc+4
last_idx 30 first_idx 20
regs=4 stack=0 before 26: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 25: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
31: (55) if r4 != 0x5 goto pc+3
last_idx 31 first_idx 20
regs=10 stack=0 before 30: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 26: (55) if r4 != 0x1 goto pc+3
regs=10 stack=0 before 25: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=10 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=10 stack=0 marks
32: (55) if r5 != 0x0 goto pc+2

from 32 to 35: R1=ctx(id=0,off=0,imm=0) R2=inv2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
35: (55) if r2 != 0x2 goto pc+4
last_idx 35 first_idx 35
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=4 stack=0 marks
last_idx 32 first_idx 20
regs=4 stack=0 before 32: (55) if r5 != 0x0 goto pc+2
regs=4 stack=0 before 31: (55) if r4 != 0x5 goto pc+3
regs=4 stack=0 before 30: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 26: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 25: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5_r=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
36: (55) if r4 != 0x1 goto pc+3
last_idx 36 first_idx 35
regs=10 stack=0 before 35: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967`: No space left on device: unknown
Error: failed to start containers: traefik


docker start traefik
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: bpf create `0: (61) r2 = *(u32 *)(r1 +0)
1: (54) w2 &= 65535
2: (61) r3 = *(u32 *)(r1 +0)
3: (74) w3 >>= 16
4: (61) r4 = *(u32 *)(r1 +4)
5: (61) r5 = *(u32 *)(r1 +8)
6: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_w=inv2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
7: (55) if r4 != 0x5 goto pc+3
 R1=ctx(id=0,off=0,imm=0) R2_w=inv2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_w=inv5 R5_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
8: (55) if r5 != 0x2 goto pc+2

from 8 to 11: R1=ctx(id=0,off=0,imm=0) R2=inv2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
11: (55) if r2 != 0x2 goto pc+3
last_idx 11 first_idx 8
regs=4 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_w=inv5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=4 stack=0 marks
last_idx 7 first_idx 0
regs=4 stack=0 before 7: (55) if r4 != 0x5 goto pc+3
regs=4 stack=0 before 6: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 5: (61) r5 = *(u32 *)(r1 +8)
regs=4 stack=0 before 4: (61) r4 = *(u32 *)(r1 +4)
regs=4 stack=0 before 3: (74) w3 >>= 16
regs=4 stack=0 before 2: (61) r3 = *(u32 *)(r1 +0)
regs=4 stack=0 before 1: (54) w2 &= 65535
regs=4 stack=0 before 0: (61) r2 = *(u32 *)(r1 +0)
12: (55) if r4 != 0x88 goto pc+2
last_idx 12 first_idx 8
regs=10 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=10 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=10 stack=0 marks
last_idx 7 first_idx 0
regs=10 stack=0 before 7: (55) if r4 != 0x5 goto pc+3
regs=10 stack=0 before 6: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 5: (61) r5 = *(u32 *)(r1 +8)
regs=10 stack=0 before 4: (61) r4 = *(u32 *)(r1 +4)
15: (55) if r2 != 0x2 goto pc+4
last_idx 15 first_idx 8
regs=4 stack=0 before 12: (55) if r4 != 0x88 goto pc+2
regs=4 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=4 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
16: (55) if r4 != 0x5 goto pc+3
last_idx 16 first_idx 8
regs=10 stack=0 before 15: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 12: (55) if r4 != 0x88 goto pc+2
regs=10 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=10 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=10 stack=0 marks
17: (55) if r5 != 0x1 goto pc+2

from 17 to 20: R1=ctx(id=0,off=0,imm=0) R2=inv2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
20: (55) if r2 != 0x2 goto pc+4
last_idx 20 first_idx 20
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=4 stack=0 marks
last_idx 17 first_idx 8
regs=4 stack=0 before 17: (55) if r5 != 0x1 goto pc+2
regs=4 stack=0 before 16: (55) if r4 != 0x5 goto pc+3
regs=4 stack=0 before 15: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 12: (55) if r4 != 0x88 goto pc+2
regs=4 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=4 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
21: (55) if r4 != 0x1 goto pc+3
last_idx 21 first_idx 20
regs=10 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=10 stack=0 marks
last_idx 17 first_idx 8
regs=10 stack=0 before 17: (55) if r5 != 0x1 goto pc+2
regs=10 stack=0 before 16: (55) if r4 != 0x5 goto pc+3
regs=10 stack=0 before 15: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 12: (55) if r4 != 0x88 goto pc+2
regs=10 stack=0 before 11: (55) if r2 != 0x2 goto pc+3
regs=10 stack=0 before 8: (55) if r5 != 0x2 goto pc+2
 R1=ctx(id=0,off=0,imm=0) R2_rw=invP2 R3_w=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_rw=invP5 R5_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=10 stack=0 marks
25: (55) if r2 != 0x2 goto pc+4
last_idx 25 first_idx 20
regs=4 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
26: (55) if r4 != 0x1 goto pc+3
last_idx 26 first_idx 20
regs=10 stack=0 before 25: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=10 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=10 stack=0 marks
30: (55) if r2 != 0x2 goto pc+4
last_idx 30 first_idx 20
regs=4 stack=0 before 26: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 25: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
31: (55) if r4 != 0x5 goto pc+3
last_idx 31 first_idx 20
regs=10 stack=0 before 30: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 26: (55) if r4 != 0x1 goto pc+3
regs=10 stack=0 before 25: (55) if r2 != 0x2 goto pc+4
regs=10 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=10 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=10 stack=0 marks
32: (55) if r5 != 0x0 goto pc+2

from 32 to 35: R1=ctx(id=0,off=0,imm=0) R2=inv2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
35: (55) if r2 != 0x2 goto pc+4
last_idx 35 first_idx 35
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4=inv5 R5=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent didn't have regs=4 stack=0 marks
last_idx 32 first_idx 20
regs=4 stack=0 before 32: (55) if r5 != 0x0 goto pc+2
regs=4 stack=0 before 31: (55) if r4 != 0x5 goto pc+3
regs=4 stack=0 before 30: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 26: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 25: (55) if r2 != 0x2 goto pc+4
regs=4 stack=0 before 21: (55) if r4 != 0x1 goto pc+3
regs=4 stack=0 before 20: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5_r=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0
parent already had regs=4 stack=0 marks
36: (55) if r4 != 0x1 goto pc+3
last_idx 36 first_idx 35
regs=10 stack=0 before 35: (55) if r2 != 0x2 goto pc+4
 R1=ctx(id=0,off=0,imm=0) R2_r=invP2 R3=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R4_r=invP5 R5=inv(id=0,umax_value=4294967`: No space left on device: unknown
Error: failed to start containers: traefik

(Quick note: No space left on device is not caused by missing space on the HDD)
@Skiepp Skiepp changed the title Upgrade to 1.14 breaks breaks some docker container (os: debian) Upgrade to 1.14 breaks some docker container (os: debian) Jan 25, 2024
@hswong3i
Copy link
Contributor

I also need to disable my https://build.opensuse.org/package/show/home:alvistack/containers-crun-1.14, and rolling back all of my Kubernetes nodes on Ubuntu 20.04/22.04 with crun 1.13.

If bpf related should due to 9306457

If apparmor related should due to 5078ce6

@giuseppe
Copy link
Member

giuseppe commented Jan 26, 2024 •
edited
Loading

I am not able to reproduce the issue (I've not tried to configure Docker), could you try if the following patch solves the problem for you?

diff --git a/src/libcrun/ebpf.c b/src/libcrun/ebpf.c
index 5534d47..cb0f8f2 100644
--- a/src/libcrun/ebpf.c
+++ b/src/libcrun/ebpf.c
@@ -483,12 +483,15 @@ libcrun_ebpf_load (struct bpf_program *program, int dirfd, const char *pin, libc
   fd = bpf (BPF_PROG_LOAD, &attr, sizeof (attr));
   if (fd < 0)
     {
-      const size_t log_size = 8192;
-      cleanup_free char *log = xmalloc (log_size);
-
       /* Prior to Linux 5.11, eBPF programs were accounted to the memlock
          prlimit.  Attempt to bump the limit, if possible.  */
       bump_memlock ();
+      fd = bpf (BPF_PROG_LOAD, &attr, sizeof (attr));
+    }
+  if (fd < 0)
+    {
+      const size_t log_size = 8192;
+      cleanup_free char *log = xmalloc (log_size);
 
       log[0] = '\0';
       attr.log_level = 1;

As a workaround, you can try forcing the memory lock ulimit to 4kb, if you are using systemd you can force LimitMEMLOCK=4096 or higher

@Skiepp
Copy link
Author

Skiepp commented Jan 26, 2024

Ciao Giuseppe, e grazie

Actually, in all vms affected by this issue the ulimit is set to unlimited, so I dont' think setting to 4kb can help.
I'll try to rebuild crun with your diff to see if it works.

@giuseppe
Copy link
Member

Actually, in all vms affected by this issue the ulimit is set to unlimited, so I dont' think setting to 4kb can help.

could you please check what is the limit for the Docker (or containerd) daemon?

You can grab it from /proc/$(PID DOCKER DAEMON)/limits

@Skiepp
Copy link
Author

Skiepp commented Jan 26, 2024

Sure!

root@norinf01:~# ps aux | grep dockerd
root       26033  2.3  0.8 4653564 136040 ?      Ssl  Jan25  22:26 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root      473511  0.0  0.0   6240   700 pts/0    S+   15:25   0:00 grep dockerd
root@norinf01:~# cat /proc/26033/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             unlimited            unlimited            processes
Max open files            524288               524288               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       63848                63848                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

@giuseppe
Copy link
Member

thanks, the locked memory ulimit seems fine. I can open a PR with my patch, if you've a possibility to try it out

@giuseppe
Copy link
Member

I've configured Docker on Debian 11 with crun, but I am not able to reproduce the issue. What kernel version are you using? How have you created the container?

@Skiepp
Copy link
Author

Skiepp commented Jan 27, 2024 •
edited
Loading

@giuseppe just a guess, could it be related to cgroupns which I set to host? It's the only customization which I have done on docker

Also, I wrote you an email if you wish to test something together, just a proposal ;)

@giuseppe
Copy link
Member

that should not really matter, but you never know...

Thanks for the proposal, yes that would help, I'd like to validate my patch in a configuration where we can see the regression

@Skiepp
Copy link
Author

Skiepp commented Jan 29, 2024

Hi @giuseppe, I tried to apply the diff you suggested, and the problem seems to have disappeared

Thanks a lot! Marco

giuseppe added a commit to giuseppe/crun that referenced this issue Jan 30, 2024
@giuseppe
ebpf: add fallback when bpf(2) fails
983b400 
commit 9306457 was too eager to
use the same fallback with a log, and that could fail for other
reasons (e.g. not enough buffer space).  Add a fallback without using
the log, so that on kernels older than 5.11 we have better chance of
success.

Closes: containers#1400

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/crun that referenced this issue Jan 30, 2024
@giuseppe
ebpf: add fallback when bpf(2) fails
f2ade60 
commit 9306457 was too eager to
use the same fallback with a log, and that could fail for other
reasons (e.g. not enough buffer space).  Add a fallback without using
the log, so that on kernels older than 5.11 we have a better chance of
success.

Closes: containers#1400

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe giuseppe mentioned this issue Jan 30, 2024
Merged
@giuseppe
Copy link
Member

thanks for onfirming it! Opened a PR: #1405

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ebpf: add fallback when bpf(2) fails giuseppe/crun
3 participants
@giuseppe @hswong3i @Skiepp