[Bugs] [Bug 1364329] New: Glusterd crashes upon receiving SIGUSR1

Fri Aug 5 05:50:39 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1364329

            Bug ID: 1364329
           Summary: Glusterd crashes upon receiving SIGUSR1
           Product: GlusterFS
           Version: 3.8.2
         Component: glusterd
          Keywords: Triaged
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: amukherj at redhat.com
                CC: amukherj at redhat.com, bugs at gluster.org,
                    p.malishev at gmail.com
        Depends On: 1359131, 1359190

+++ This bug was initially created as a clone of Bug #1359190 +++

+++ This bug was initially created as a clone of Bug #1359131 +++

Description of problem:
I try to generate glusterdump file with SIGUSR1 to find out maximum op-version
for the glusterd on this host.
Glusterd crashes upon receiving SIGUSR1

Version-Release number of selected component (if applicable):
# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"

# dpkg -l | grep gluster
ii  glusterfs-client                      3.8.1-ubuntu1~trusty1              
amd64        clustered file-system (client package)
ii  glusterfs-common                      3.8.1-ubuntu1~trusty1              
amd64        GlusterFS common libraries and translator modules
ii  glusterfs-server                      3.8.1-ubuntu1~trusty1              
amd64        clustered file-system (server package)

How reproducible:
Glusterd crashes each time I send SIGUSR1

Steps to Reproduce:
1. rm -f /var/run/gluster/glusterdump.*
2. service glusterfs-server start
3. kill -SIGUSR1 $(pidof glusterd)
4. ps -ef | grep glusterd ; ls /var/run/gluster | grep glusterdump

Actual results:
No glusterd in the ps output.
No glusterdump file created.

glusterd crashes with the following lines in the log:
[2016-07-22 11:22:50.923915] I [MSGID: 106493]
[glusterd-rpc-ops.c:691:__glusterd_friend_update_cbk] 0-management: Received
ACC from uuid: c6654b8f-3ca8-424f-8d0f-7c0b7d8108cc
pending frames:
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2016-07-22 11:23:09
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.1
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f21f4b87b02]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f21f4b9104d]
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f21f3f81cb0]
/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x1d03)[0x7f21f3f96943]
/lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0x95)[0x7f21f4054165]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section_fd+0x69)[0x7f21f4bb18a9]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section+0xa5)[0x7f21f4bb1ab5]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x4ce3e)[0x7f21f4bb1e3e]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_xlator_info+0x58)[0x7f21f4bb2658]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_info+0x500)[0x7f21f4bb2d00]
/usr/sbin/glusterd(glusterfs_sigwaiter+0xce)[0x55ffe99d2bae]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f21f4318184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f21f404537d]
---------

Expected results:
Glusterd creates glusterdump file in /var/run/gluster

Additional info:

--- Additional comment from Atin Mukherjee on 2016-07-22 08:47:08 EDT ---

Does this happen everytime? I've seen it earlier as well but couldn't
reproduce.

--- Additional comment from Atin Mukherjee on 2016-07-22 08:48:06 EDT ---

coredump file?

--- Additional comment from Vijay Bellur on 2016-07-22 09:10:07 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump
crash) posted (#1) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-22 09:16:55 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump
crash) posted (#2) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-24 23:56:40 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump
crash) posted (#3) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-26 14:10:38 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump
crash) posted (#4) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-29 00:57:17 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump
crash) posted (#5) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-01 07:23:11 EDT ---

REVIEW: http://review.gluster.org/14987 (libglusterfs: fix glusterd statedump
crash) posted (#6) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-04 11:44:13 EDT ---

COMMIT: http://review.gluster.org/14987 committed in master by Jeff Darcy
(jdarcy at redhat.com) 
------
commit 049c91565dddb622b8902ccfeb36c0d414c609e1
Author: Atin Mukherjee <amukherj at redhat.com>
Date:   Tue May 31 16:44:48 2016 +0530

    libglusterfs: fix glusterd statedump crash

    commit 3c04a91 removed setting typeStr to NULL if num_allocs is set to 0,
this
    has caused this regression. Code has been put back like earlier and to
avoid
    statedump printing all the NULL values check is modified to see skip the
records
    if num_allocs is 0 instead of total_allocs

    Change-Id: Ib8bcc2fba908e88cf52b641c3f6bcba74f5e667c
    BUG: 1359190
    Signed-off-by: Atin Mukherjee <amukherj at redhat.com>
    Reviewed-on: http://review.gluster.org/14987
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: N Balachandran <nbalacha at redhat.com>
    Reviewed-by: Prashanth Pai <ppai at redhat.com>
    Reviewed-by: Jeff Darcy <jdarcy at redhat.com>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1359131
[Bug 1359131] Glusterd crashes upon receiving SIGUSR1
https://bugzilla.redhat.com/show_bug.cgi?id=1359190
[Bug 1359190] Glusterd crashes upon receiving SIGUSR1
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.