Hang on "Initializing remote caches"

[Cynical-Optimist]

See original issue on GitLab
In GitLab by [Gitlab user @TheRealMichaelCatanzaro] on Oct 7, 2020, 18:50

Summary

$ bst build core.bst sdk.bst
[--:--:--][][] STATUS  Cache usage recomputed: 10B / infinity (0%)
[--:--:--][][] START   Build
[--:--:--][][] START   Loading elements
[00:00:04][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:02][][] SUCCESS Resolving elements
[--:--:--][][] START   Initializing remote caches

With buildstream-1.6.0-1.fc33, bst-external-0.21.0-1.fc33, and python3-3.9.0~rc2-1.fc33, this last step never completes.

Steps to reproduce

Clone gnome-build-meta, then try to bst build core.bst sdk.bst.

What is the current bug behavior?

Hang before anything starts to build.

What is the expected correct behavior?

It should build!

Relevant logs and/or screenshots

[Gitlab user @abderrahimk] asked for some strace. Here is the very end before it hangs performing a read():

openat(AT_FDCWD, "/dev/shm/mF8sKU", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
write(5, "\1\0\0\0\0\0\0\0\200\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 32, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0x7efc81be5000
link("/dev/shm/mF8sKU", "/dev/shm/sem.mp-e3jut377") = 0
fstat(5, {st_mode=S_IFREG|0600, st_size=32, ...}) = 0
unlink("/dev/shm/mF8sKU")               = 0
close(5)                                = 0
unlink("/dev/shm/sem.mp-e3jut377")      = 0
getpid()                                = 55091
getpid()                                = 55091
lstat("/dev/shm/7A80eV", 0x7ffd2832ba00) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/shm/7A80eV", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
write(5, "\377\377\377\177\0\0\0\0\200\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 32, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0x7efc81be4000
link("/dev/shm/7A80eV", "/dev/shm/sem.mp-7auqask5") = 0
fstat(5, {st_mode=S_IFREG|0600, st_size=32, ...}) = 0
unlink("/dev/shm/7A80eV")               = 0
close(5)                                = 0
unlink("/dev/shm/sem.mp-7auqask5")      = 0
getpid()                                = 55091
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
getpid()                                = 55091
stat("/usr/lib64/python3.9/multiprocessing", {st_mode=S_IFDIR|0755, st_size=576, ...}) = 0
stat("/usr/lib64/python3.9/multiprocessing/popen_fork.py", {st_mode=S_IFREG|0644, st_size=2377, ...}) = 0
stat("/usr/lib64/python3.9/multiprocessing/popen_fork.py", {st_mode=S_IFREG|0644, st_size=2377, ...}) = 0
openat(AT_FDCWD, "/usr/lib64/python3.9/multiprocessing/__pycache__/popen_fork.cpython-39.pyc", O_RDONLY|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0
ioctl(5, TCGETS, 0x7ffd2832b4a0)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(5, 0, SEEK_CUR)                   = 0
lseek(5, 0, SEEK_CUR)                   = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0
read(5, "a\r\r\n\0\0\0\0A\202b_I\t\0\0\343\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2513) = 2512
read(5, "", 1)                          = 0
close(5)                                = 0
pipe2([5, 6], O_CLOEXEC)                = 0
pipe2([7, 8], O_CLOEXEC)                = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7efc81549a10) = 55632
close(6)                                = 0
close(7)                                = 0
getpid()                                = 55091
rt_sigprocmask(SIG_SETMASK, [], [INT], 8) = 0
read(3,

Possible fixes

[Gitlab user @abderrahimk] suggests downgrading to python 3.8 might fix this. I'm about to try running buildstream in a F32 container to see.

Other relevant information

• BuildStream version affected: /milestone %BuildStream_v1.6

---

[Cynical-Optimist]

In GitLab by [Gitlab user @TheRealMichaelCatanzaro] on Oct 7, 2020, 18:59

suggests downgrading to python 3.8 might fix this. I'm about to try running buildstream in a F32 container to see.

Yeah it's building with no problems in my F32 container.

[Cynical-Optimist]

In GitLab by [Gitlab user @alatiera] on Oct 7, 2020, 19:19

With buildstream-1.6.0-1.fc33, bst-external-0.21.0-1.fc33, and python3-3.9.0~rc2-1.fc33, this last step never completes.

Btw, also f33, its reproducible with bst and bst-external isntalled from pip as well

[Cynical-Optimist]

In GitLab by [Gitlab user @abderrahimk] on Oct 7, 2020, 19:22

This is almost certainly python 3.9. BuildStream's use of forking and asyncio is not supported by upstream, and things end up being broken on every new point release.