[Bugs] [Bug 1693184] New: A brick process(glusterfsd) died with 'memory violation'

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1693184

            Bug ID: 1693184
           Summary: A brick process(glusterfsd) died with 'memory
                    violation'
           Product: GlusterFS
           Version: experimental
          Hardware: x86_64
                OS: Linux
            Status: NEW
         Component: replicate
          Assignee: bugs at gluster.org
          Reporter: knjeong at growthsoft.co.kr
                CC: bugs at gluster.org
  Target Milestone: ---
    Classification: Community



Description of problem:


I'm using a volume with two replicas of the 3.6.9 version of GlusterFS.
The volume on which the issue occurs is not very active and at one point a
process dies suddenly.
This issue has also caused core dumps, and what we found at the time of the
problem is as follows:



- /var/log/messages (Brick log is also the same)

Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: pending frames:
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: pending frames:
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: patchset:
git://git.gluster.com/glusterfs.git
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: signal received: 6
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: time of crash:
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: 2019-03-24 09:15:40
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: configuration details:
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: argp 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: backtrace 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: dlfcn 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: libpthread 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: llistxattr 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: setfsid 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: spinlock 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: epoll.h 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: xattr.h 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: st_atim.tv_nsec 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: package-string: glusterfs
3.6.9
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: ---------
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: patchset:
git://git.gluster.com/glusterfs.git
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: signal received: 6
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: time of crash:
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: 2019-03-24 09:15:40
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: configuration details:
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: argp 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: backtrace 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: dlfcn 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: libpthread 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: llistxattr 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: setfsid 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: spinlock 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: epoll.h 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: xattr.h 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: st_atim.tv_nsec 1
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: package-string: glusterfs
3.6.9
Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: ---------
Mar 24 18:15:40 P-NAS8 kernel: audit_printk_skb: 57 callbacks suppressed
Mar 24 18:15:40 P-NAS8 kernel: type=1701 audit(1553418940.165:27816716):
auid=1002 uid=0 gid=0 ses=3174727 pid=127312 comm="glusterfsd" reason="memory
violation" sig=6
Mar 24 18:15:40 P-NAS8 systemd-logind: Removed session 3174727.
Mar 24 18:15:40 P-NAS8 kernel: audit_printk_skb: 57 callbacks suppressed
Mar 24 18:15:40 P-NAS8 kernel: type=1701 audit(1553418940.165:27816716):
auid=1002 uid=0 gid=0 ses=3174727 pid=127312 comm="glusterfsd" reason="memory
violation" sig=6



- CoreDump

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfsd -s p-tview-nas8 --volfile-id
repl_dist_vol'.
Program terminated with signal 6, Aborted.
#0  0x00007fb6da9895f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glusterfs-3.6.9-1.el7.x86_64
(gdb) bt
#0  0x00007fb6da9895f7 in raise () from /lib64/libc.so.6
#1  0x00007fb6da98ace8 in abort () from /lib64/libc.so.6
#2  0x00007fb6da9c9317 in __libc_message () from /lib64/libc.so.6
#3  0x00007fb6da9d1023 in _int_free () from /lib64/libc.so.6
#4  0x00007fb6db968d29 in dict_destroy () from /lib64/libglusterfs.so.0
#5  0x00007fb6db99776d in call_stub_destroy () from /lib64/libglusterfs.so.0
#6  0x00007fb6ca286333 in iot_worker () from
/usr/lib64/glusterfs/3.6.9/xlator/performance/io-threads.so
#7  0x00007fb6db103dc5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fb6daa4a28d in clone () from /lib64/libc.so.6



- free
              total        used        free      shared  buff/cache   available
Mem:           31G         21G         1.1G        4.2G        9.0G        5.1G
Swap:          15G         173M        15G



Unfortunately, I didn't know the exact cause here.
Is there any other good way to determine the cause?
I look forward to your help.


Version-Release number of selected component (if applicable): glusterfs-3.6.9
(community version)


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.