Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0-glusterfs-fuse: writing to fuse device failed: No such file or directory #1986

Closed
jicki opened this issue Jan 11, 2021 · 20 comments
Closed
Labels
wontfix Managed by stale[bot]

Comments

@jicki
Copy link

jicki commented Jan 11, 2021

error

[2021-01-11 04:02:28.230946] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x17a)[0x7f5cf9da38ca] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x816a)[0x7f5cf78eb16a] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x8c3b)[0x7f5cf78ebc3b] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f5cf950b6db] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f5cf9234a3f] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-01-11 04:14:49.262457] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x17a)[0x7f5cf9da38ca] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x816a)[0x7f5cf78eb16a] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x827b)[0x7f5cf78eb27b] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f5cf950b6db] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f5cf9234a3f] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-01-11 04:28:53.825725] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x17a)[0x7f5cf9da38ca] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x816a)[0x7f5cf78eb16a] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x827b)[0x7f5cf78eb27b] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f5cf950b6db] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f5cf9234a3f] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-01-11 04:35:05.542832] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x17a)[0x7f5cf9da38ca] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x816a)[0x7f5cf78eb16a] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x827b)[0x7f5cf78eb27b] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f5cf950b6db] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f5cf9234a3f] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-01-11 06:11:00.520814] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x17a)[0x7f5cf9da38ca] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x816a)[0x7f5cf78eb16a] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x827b)[0x7f5cf78eb27b] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f5cf950b6db] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f5cf9234a3f] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-01-11 06:11:15.812215] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x17a)[0x7f5cf9da38ca] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x816a)[0x7f5cf78eb16a] (--> /usr/lib/x86_64-linux-gnu/glusterfs/7.5/xlator/mount/fuse.so(+0x827b)[0x7f5cf78eb27b] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f5cf950b6db] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f5cf9234a3f] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory

@perrynzhou
Copy link
Contributor

@jicki that seem's kernel module of fuse crash,you just can remount that.i want to know what's workload for your app?

@jicki
Copy link
Author

jicki commented Jan 18, 2021

@jicki that seem's kernel module of fuse crash,you just can remount that.i want to know what's workload for your app?

Does reinstalling mean upgrading the version? This error appears a lot in the log .

@shenwen2012
Copy link

@jicki you are Docker application in glusterfs ?

@mermetbt
Copy link

mermetbt commented Aug 3, 2021

I got the same problem:

[2021-08-03 17:49:04.506283] E [fuse-bridge.c:219:check_and_dump_fuse_W] (--> /lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x12d)[0x7f21b606973d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x8d12)[0x7f21b4634d12] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x9241)[0x7f21b4635241] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f21b5bfbfa3] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f21b58434cf] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-08-03 17:49:04.519448] E [fuse-bridge.c:219:check_and_dump_fuse_W] (--> /lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x12d)[0x7f21b606973d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x8d12)[0x7f21b4634d12] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x9241)[0x7f21b4635241] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f21b5bfbfa3] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f21b58434cf] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-08-03 17:49:04.529610] E [fuse-bridge.c:219:check_and_dump_fuse_W] (--> /lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x12d)[0x7f21b606973d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x8d12)[0x7f21b4634d12] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x9241)[0x7f21b4635241] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f21b5bfbfa3] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f21b58434cf] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-08-03 17:49:04.542524] E [fuse-bridge.c:219:check_and_dump_fuse_W] (--> /lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x12d)[0x7f21b606973d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x8d12)[0x7f21b4634d12] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x9241)[0x7f21b4635241] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f21b5bfbfa3] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f21b58434cf] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory

I'm using GlusterFS 5.5 on the hosts machine of a Kubernetes cluster.

I could read/write on the volume without problems at the same time and it works, but the services running in Docker have troubles to read/write from the volume.
The glusterfs volume is mounted on the k8s node and the docker containers are using this volume (I'm not running gluster inside docker).

@stale
Copy link

stale bot commented Mar 9, 2022

Thank you for your contributions.
Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity.
It will be closed in 2 weeks if no one responds with a comment here.

@stale stale bot added the wontfix Managed by stale[bot] label Mar 9, 2022
@shoufuzhang
Copy link

我遇到了同样的问题:

[2021-08-03 17:49:04.506283] E [fuse-bridge.c:219:check_and_dump_fuse_W] (--> /lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x12d)[0x7f21b606973d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x8d12)[0x7f21b4634d12] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x9241)[0x7f21b4635241] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f21b5bfbfa3] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f21b58434cf] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-08-03 17:49:04.519448] E [fuse-bridge.c:219:check_and_dump_fuse_W] (--> /lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x12d)[0x7f21b606973d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x8d12)[0x7f21b4634d12] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x9241)[0x7f21b4635241] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f21b5bfbfa3] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f21b58434cf] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-08-03 17:49:04.529610] E [fuse-bridge.c:219:check_and_dump_fuse_W] (--> /lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x12d)[0x7f21b606973d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x8d12)[0x7f21b4634d12] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x9241)[0x7f21b4635241] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f21b5bfbfa3] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f21b58434cf] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory
[2021-08-03 17:49:04.542524] E [fuse-bridge.c:219:check_and_dump_fuse_W] (--> /lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x12d)[0x7f21b606973d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x8d12)[0x7f21b4634d12] (--> /usr/lib/x86_64-linux-gnu/glusterfs/5.5/xlator/mount/fuse.so(+0x9241)[0x7f21b4635241] (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f21b5bfbfa3] (--> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f21b58434cf] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory

我在 Kubernetes 集群的主机上使用 GlusterFS 5.5。

我可以同时在卷上读取/写入而不会出现问题并且它可以工作,但是在 Docker 中运行的服务在从卷中读取/写入时遇到了麻烦。 glusterfs 卷安装在 k8s 节点上,docker 容器正在使用这个卷(我没有在 docker 中运行 gluster)。

请问您找到问题的原因了吗?我也正在遇到同样的问题

@shoufuzhang
Copy link

@mermetbt help

@shoufuzhang
Copy link

@mermetbt Did you find a solution?

@mermetbt
Copy link

Hello @shoufuzhang,

the problem is that glusterfs cannot support so many little changes on the filesystem at the speed required by a kubernetes cluster. I finally abandoned this solution and use glusterfs only for the my docker container registry.

You can add my wechat (mermetbt) if you want to exchange some feedback from my experience.

Regards,

@amarts
Copy link
Member

amarts commented Mar 22, 2022

@shoufuzhang @mermetbt Did you guys try github.com/kadalu/kadalu for your PVC requirements?

@shoufuzhang
Copy link

@shoufuzhang @mermetbt你们是否尝试过 github.com/kadalu/kadalu 来满足您的 PVC 要求?

Mounting gluster with kubernetes pv/pvc is fine, but mounting it on the host, then docker -v mounts it with this error

@mykaul
Copy link
Contributor

mykaul commented Mar 22, 2022

Gluster 5.5 is a very old release. A newer release should perform substantially better.

@shoufuzhang
Copy link

Gluster 5.5 是一个非常古老的版本。较新的版本应该表现得更好。

gluster version 6.9

@mykaul
Copy link
Contributor

mykaul commented Mar 22, 2022

Gluster 5.5 是一个非常古老的版本。较新的版本应该表现得更好。

gluster version 6.9

This is still very old, relatively. Please try latest 9.x or 10.x releases.

@amarts
Copy link
Member

amarts commented Mar 22, 2022

the issue seems to be the /proc/0/status file read issue which crashed glusterfs. Fixes are available only in 10.x or latest of 9.x series.

@shoufuzhang
Copy link

/proc/0/status ??

@amarts
Copy link
Member

amarts commented Mar 22, 2022

#2467 is my guess on this as the mount itself fails, and in the container ecosystem.

@perrynzhou
Copy link
Contributor

perrynzhou commented Mar 27, 2022 via email

@stale
Copy link

stale bot commented Oct 30, 2022

Thank you for your contributions.
Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity.
It will be closed in 2 weeks if no one responds with a comment here.

@stale stale bot added the wontfix Managed by stale[bot] label Oct 30, 2022
@stale
Copy link

stale bot commented Dec 24, 2022

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.

@stale stale bot closed this as completed Dec 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix Managed by stale[bot]
Projects
None yet
Development

No branches or pull requests

7 participants