[Bugs] [Bug 1608305] New: [glusterfs-3.6.9] Fuse-mount has been forced off

bugzilla at redhat.com bugzilla at redhat.com
Wed Jul 25 09:37:42 UTC 2018 

https://bugzilla.redhat.com/show_bug.cgi?id=1608305

            Bug ID: 1608305
           Summary: [glusterfs-3.6.9] Fuse-mount has been forced off
           Product: GlusterFS
           Version: 3.12
         Component: glusterd
          Assignee: bugs at gluster.org
          Reporter: knjeong at growthsoft.co.kr
                CC: bugs at gluster.org



Description of problem:


I was using glusterfs-3.6.9(community version) and suddenly fuse-mount was
disabled on the client with the error below. 
I have seen all of the gluster-logs and os-logs(/var/log/messages) on the
unmounted client and on the glusterfs server, and I'm guessing it is related to
memory, but I do not know the exact cause of the error.

What is the best way to determine the cause of the error in this situation?
Can you determine the exact cause with this information?
Help!



- /var/log/messages (unmounted client)

Jul 25 08:11:00 CLIENT gluster[5966]: pending frames:
Jul 25 08:11:00 CLIENT gluster[5966]: pending frames:
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(WRITE)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(LOOKUP)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: patchset:
git://git.gluster.com/glusterfs.git
Jul 25 08:11:00 CLIENT gluster[5966]: signal received: 6
Jul 25 08:11:00 CLIENT gluster[5966]: time of crash:
Jul 25 08:11:00 CLIENT gluster[5966]: 2018-07-24 23:11:00
Jul 25 08:11:00 CLIENT gluster[5966]: configuration details:
Jul 25 08:11:00 CLIENT gluster[5966]: argp 1
Jul 25 08:11:00 CLIENT gluster[5966]: backtrace 1
Jul 25 08:11:00 CLIENT gluster[5966]: dlfcn 1
Jul 25 08:11:00 CLIENT gluster[5966]: libpthread 1
Jul 25 08:11:00 CLIENT gluster[5966]: llistxattr 1
Jul 25 08:11:00 CLIENT gluster[5966]: setfsid 1
Jul 25 08:11:00 CLIENT gluster[5966]: spinlock 1
Jul 25 08:11:00 CLIENT gluster[5966]: epoll.h 1
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(1) op(LOOKUP)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: frame : type(0) op(0)
Jul 25 08:11:00 CLIENT gluster[5966]: patchset:
git://git.gluster.com/glusterfs.git
Jul 25 08:11:00 CLIENT gluster[5966]: signal received: 6
Jul 25 08:11:00 CLIENT gluster[5966]: time of crash:
Jul 25 08:11:00 CLIENT gluster[5966]: 2018-07-24 23:11:00
Jul 25 08:11:00 CLIENT gluster[5966]: configuration details:
Jul 25 08:11:00 CLIENT gluster[5966]: argp 1
Jul 25 08:11:00 CLIENT gluster[5966]: backtrace 1
Jul 25 08:11:00 CLIENT gluster[5966]: dlfcn 1
Jul 25 08:11:00 CLIENT gluster[5966]: libpthread 1
Jul 25 08:11:00 CLIENT gluster[5966]: llistxattr 1
Jul 25 08:11:00 CLIENT gluster[5966]: setfsid 1
Jul 25 08:11:00 CLIENT gluster[5966]: spinlock 1
Jul 25 08:11:00 CLIENT gluster[5966]: epoll.h 1
Jul 25 08:11:00 CLIENT gluster[5966]: xattr.h 1
Jul 25 08:11:00 CLIENT gluster[5966]: st_atim.tv_nsec 1
Jul 25 08:11:00 CLIENT gluster[5966]: package-string: glusterfs 3.7.1
Jul 25 08:11:00 CLIENT gluster[5966]: ---------
Jul 25 08:11:00 CLIENT kernel: audit_printk_skb: 75 callbacks suppressed
Jul 25 08:11:00 CLIENT kernel: type=1701 audit(1532473860.042:2595231): auid=0
uid=0 gid=0 ses=2 pid=5971 comm="glusterfs" reason="memory violation" sig=6
Jul 25 08:11:00 CLIENT gluster[5966]: xattr.h 1
Jul 25 08:11:00 CLIENT gluster[5966]: st_atim.tv_nsec 1
Jul 25 08:11:00 CLIENT gluster[5966]: package-string: glusterfs 3.7.1
Jul 25 08:11:00 CLIENT gluster[5966]: ---------
Jul 25 08:11:00 CLIENT kernel: audit_printk_skb: 75 callbacks suppressed
Jul 25 08:11:00 CLIENT kernel: type=1701 audit(1532473860.042:2595231): auid=0
uid=0 gid=0 ses=2 pid=5971 comm="glusterfs" reason="memory violation" sig=6



- glusterfs server log (brick-log / time -9h = same time)

[2018-07-24 23:11:00.580994] I [server.c:518:server_rpc_notify]
0-repl_dist_vol-server: disconnecting connection from
CLIENT-5964-2018/03/27-14:05:04:231328-VOL-client-6-0-1
[2018-07-24 23:11:00.581059] W [inodelk.c:404:pl_inodelk_log_cleanup]
0-VOL-server: releasing lock on 3614fad5-8de0-42f1-bd35-c13d9e451ca6 held by
{client=0x7eff9498d950, pid=0 lk-owner=9862425c0a7f0000}
[2018-07-24 23:11:00.581080] I [server-helpers.c:290:do_fd_cleanup]
0-VOL-server: fd cleanup on /SdBackup/101970/20180725T072155.avi
[2018-07-24 23:11:00.581111] I [client_t.c:417:gf_client_unref] 0-VOL-server:
Shutting down connection
CLIENT-5964-2018/03/27-14:05:04:231328-VOL-client-6-0-1
[2018-07-24 23:11:00.581045] I [server.c:518:server_rpc_notify] 0-VOL-server:
disconnecting connection from
CLIENT-5964-2018/03/27-14:05:04:231328-VOL-client-8-0-1
[2018-07-24 23:11:00.581121] I [client_t.c:417:gf_client_unref]
0-tidc_media_repl_dist_vol-server: Shutting down connection
CLIENT-5964-2018/03/27-14:05:04:231328-VOL-client-8-0-1



- memory (usual)

# free -h
              total        used        free      shared  buff/cache   available
Mem:            31G        2.6G        1.6G        203M         26G         27G
Swap:           15G        8.1G        7.6G



Version-Release number of selected component (if applicable): glusterfs-3.6.9
(community version)

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.

More information about the Bugs mailing list