-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
artifact test fail on first hydra command [error mounting "tmpfs" to rootfs at ".../.local": create mountpoint for .../.local mount: mkdirat .../.local: file exists: unknown.] #9645
Comments
could be issue with .local: we bind |
Only 2 artifact tests failed? What is special about artifacts tests? Do lots of them run simultaneously on a single builder? Also, internet says that machine restart solves the problem... |
Probably it's a docker bug, the question is how to avoid it, cause we won't be able to solve it directly (maybe only to report it) |
We know following:
So, I think we get such failures when we hit the RAM limit on the builders. Possible solutions:
@fruch , @soyacz I really believe above must help, let's avoid reverting the mentioned change. |
We can also limit the size of tmpfs usage We need it to mask things, I don't know how much we use it On Sunday I'll reduce the Jenkins excecuters to 1 per builder (it's now 4) |
I've change the builders to 1 executer, for the time being. and I'll open a PR to limit the tmpfs size |
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: scylladb#9645
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: scylladb#9645
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120) # Conflicts: # docker/env/hydra.sh
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120) # Conflicts: # docker/env/hydra.sh
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
i am still seeing this failure.
|
looks like I've missed the number of executors in the gcp builder configuration |
that means the change of tmpfs size, didn't really helped. |
since we are might be running multiple hydra commands on the same jenkins builder, and we are running into those kind of error: ``` docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/home/jenkins/.local": create mountpoint for /home/jenkins/.local mount: mkdirat /var/lib/docker/.../.local: file exists: unknown. ``` we are trying to limit the memory to try to avoid this issue Fixes: #9645 (cherry picked from commit 9a73120)
any more ideas on how to tackle this one ? before we revert the tmpfs change, and break UX a bit ? |
Where does it fail? |
it was reported in multiple place (no all of them reported here) last report I've seen from yesterday
but with no reference to the job |
May it come from the SCT branches where we didn't port the size limiter to? I looked the following artifact tests:
And bunch of others in the https://jenkins.scylladb.com/job/scylla-master/job/artifacts/ dir. And don't see there this kind of a failure. |
@fruch I walked through lots of artifacts CI test runs which happened after the bug fix and don't see any occurrence of this bug. I don't have evidence so far. |
|
Seems like dockerd isn't completely ready when we start calling it |
since we are running into cases docker is failing on this first stage, we would retry it 3 times with hope it would be enough to avoid those type of error Ref: scylladb#9645
since we are running into cases docker is failing on this first stage, we would retry it 3 times with hope it would be enough to avoid those type of error Ref: #9645
recently we run multiple times (2 so far), in a failure as such:
all the rest of the pipelien phase were running o.k. and passing
.local
remount we introduce in 102e63dThe text was updated successfully, but these errors were encountered: