ci(buildkit): fix timeout for image job

[crazy-max]

No description provided.

[profnandaa]

I see it's still failing with timeout, we try 30 min or what could be the cause?

[crazy-max]

Ok got a proper stacktrace now: https://github.com/moby/buildkit/actions/runs/7231798843/job/19705540067?pr=4491#step:13:505

[crazy-max]

@tonistiigi Added ps aux per your suggestion and got: https://github.com/moby/buildkit/actions/runs/7233373573/job/19718123101?pr=4491#step:7:1109

PID   USER     TIME  COMMAND
    1 root      0:00 /sbin/docker-init -- buildkitd --debug
    7 root      0:30 buildkitd --debug
  163 root      0:00 buildctl dial-stdio
  442 root      0:00 buildkit-runc --log /var/lib/buildkit/runc-overlayfs/executor/runc-log.json --log-format json run --bundle /var/lib/buildkit/runc-overlayfs/executor/rtwgljgknwyvvi9kzrru3pbnb --keep rtwgljgknwyvvi9kzrru3pbnb
  502 root      0:00 /bin/sh -c   set -ex   xx-go build ${GOBUILDFLAGS} -gcflags="${GOGCFLAGS}" -ldflags "$(cat /tmp/.ldflags) -extldflags '-static'" -tags "osusergo netgo static_build seccomp ${BUILDKITD_TAGS}" -o /usr/bin/buildkitd ./cmd/buildkitd   xx-verify ${VERIFYFLAGS} /usr/bin/buildkitd   if [ "$(xx-info os)" = "linux" ]; then /usr/bin/buildkitd --version; fi 
37538 root      0:00 {buildkitd} /usr/bin/qemu-arm /usr/bin/buildkitd /usr/bin/buildkitd --version
37624 root      0:00 {buildkitd} /usr/bin/qemu-arm /usr/bin/buildkitd /usr/bin/buildkitd --version
42689 root      0:00 ps aux

So seems related to emulation when invoking buildkitd --version (related to #4357). I will add QEMU_STRACE=1 to have more logs.

[crazy-max]

Comparing QEMU logs in https://github.com/moby/buildkit/actions/runs/7238175771/job/19719814734?pr=4491#step:7:4174 that succeeds:

#29 54.03 3905 3905 futex(clock_gettime(0x000000c000076948,FUTEX_PRIVATE_FLAG|FUTEX_WAKE,1,NULL,NULL,0)CLOCK_MONOTONIC,0x000000c000087ee8) = 0 ({tv_sec = 206,tv_nsec = 447380543})
#29 54.03  = 1
#29 54.03 3905 clock_gettime(CLOCK_MONOTONIC,0x000000c000087ee8)3905  = 0futex( ({tv_sec = 206,tv_nsec = 447395170}0x00000000023ccd68, = 0
#29 54.03 FUTEX_PRIVATE_FLAG|FUTEX_WAIT,0,NULL,NULL,0)3905 clock_gettime(CLOCK_MONOTONIC,0x000000c00010fd68) = 0 ({tv_sec = 206,tv_nsec = 447412413})
#29 54.03 3905 futex(0x00000000023ccd68,FUTEX_PRIVATE_FLAG|FUTEX_WAKE,1,NULL,NULL,0) = 1
#29 54.03 3905 futex(0x000000c000076948, = FUTEX_PRIVATE_FLAG|FUTEX_WAIT0,0,
#29 54.03 )

With this one that fails: https://github.com/moby/buildkit/actions/runs/7238175771/job/19719964140?pr=4491#step:7:47491

50 440.6 3915 futex(0x000000c000286148,FUTEX_PRIVATE_FLAG|FUTEX_WAKE,1,NULL,NULL,0) = 1
#50 440.6 3915 clock_gettime(CLOCK_REALTIME,0x000000c000072608) =  = 00
#50 440.6  ({tv_sec = 1702819771,tv_nsec = 199764230})
#50 440.6 3915 3915 clock_gettime(clock_gettime(CLOCK_MONOTONICCLOCK_MONOTONIC,,0x000000c00046dd680x000000c000072608)) = 0 ( = 0{tv_sec = 546,tv_nsec = 846162178})
#50 440.6  ({tv_sec = 546,tv_nsec = 846162526})
#50 440.6 3915 epoll_pwait(4,824638363080,128,239948,0,0)3915 clock_gettime(CLOCK_MONOTONIC,0x000000c0000726f8) = 0 ({tv_sec = 546,tv_nsec = 846625303})
#50 440.6 3915 write(6,0x72673,1) = 1
#50 440.6  = 1
#50 440.6 3915 read(5,0x46d5a8,16) = 1
#50 440.6 3915 clock_gettime(CLOCK_MONOTONIC,0x000000c00046ddc0) = 0 ({tv_sec = 546,tv_nsec = 846804653})
#50 440.6 3915 clock_gettime(CLOCK_MONOTONIC,0x000000c00046dd68) = 0 ({tv_sec = 546,tv_nsec = 846818488})
#50 440.6 3915 epoll_pwait(4,824638363080,128,49999,0,0)3915 clock_gettime(CLOCK_MONOTONIC,0x000000c000207d68) = 0 ({tv_sec = 546,tv_nsec = 846844494})
#50 440.6 3915 futex(0x000000c0003ae548,FUTEX_PRIVATE_FLAG|FUTEX_WAIT,0,NULL,NULL,0) = 0
#50 440.6 3915 clock_gettime(CLOCK_MONOTONIC,0x000000c000087ee8) = 0 ({tv_sec = 546,tv_nsec = 855425396})
#50 440.6 3915 clock_gettime(CLOCK_MONOTONIC,0x000000c000087e68) = 0 ({tv_sec = 546,tv_nsec = 855438741})
#50 490.6 3915 futex(0x00000000023cd100,FUTEX_PRIVATE_FLAG|FUTEX_WAIT,0,0x000000c000087e68,NULL,0) = -1 errno=110 (Operation timed out)

An operation timed out and retries indefinitely?

[crazy-max]

This seems related to https://gitlab.com/qemu-project/qemu/-/issues/456 where PI (priority-inheritance) futexes are not supported in qemu. This is supported since QEMU 7.2.0 but atm we are using 7.0.0. Not sure why I can't repro consistently though.

Same happens in this project anonaddy/docker#245 (comment)

[crazy-max]

Still hangs with QEMU 8.0.4 😕 https://github.com/moby/buildkit/actions/runs/7246753647/job/19741177891?pr=4491#step:7:37760

#51 787.5 3755 futex(0x0221189c,FUTEX_PRIVATE_FLAG|FUTEX_WAIT,0,{tv_sec = 49,tv_nsec = 990587752},NULL,0) = -1 errno=110 (Operation timed out)

[tonistiigi]

@crazy-max I guess this should be draft, right?