[Bugs] [Bug 1202290] [epoll+Snapshot] : Snapd crashed while trying to list snaps under .snaps folder

bugzilla at redhat.com bugzilla at redhat.com
Mon Mar 16 10:20:49 UTC 2015


https://bugzilla.redhat.com/show_bug.cgi?id=1202290

Poornima G <pgurusid at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED



--- Comment #1 from Poornima G <pgurusid at redhat.com> ---
Description of problem:
=======================
While trying to list the snapshots under .snaps folder, snapd crashed with
client.event-threads and server.event-threads set to 4 and 5 respectively.


Version-Release number of selected component (if applicable):
=============================================================
glusterfs 3.6.0.51

How reproducible:
=================
tried once


Steps to Reproduce:
==================
1.Create a 6x2 dist-rep volume and start it 
enable quota on the volume
set client.event-threads to 4 
set server.event-threads to 5 

2.Fuse and NFS mount the volume and create some IO on it 
for i in {1..20} ; do cp -rvf /etc fetc.$i ; done
for i in {1..20} ; do cp -rvf /etc n1etc.$i ; done

3.Create 256 snapshots in a loop 

4.Activate 60 snapshots in loop 

5.Enable USS on the volume

6.After 256 snapshot creation is completed, activate all the remaining
snapshots 

>From fuse mount, cd to .snaps and list the snapshots -> successful
>From NFS mount, cd to .snaps and list the snapshots -> No such file/directory 
>From fuse mount, cd to .snaps and list snapshots again -> failed with
"Transport endpoint is not connected"


Actual results:
==============
While listing snapshots under .snaps snapd crashed 


Expected results:
================
All snapshots should be listed under .snaps successfully


Additional info:
================
[root at inception core]# gluster v i 

Volume Name: vol0
Type: Distributed-Replicate
Volume ID: d1ac6dec-a438-4c9f-9b0a-671396088e40
Status: Started
Snap Volume: no
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: inception.lab.eng.blr.redhat.com:/rhs/brick1/b1
Brick2: rhs-arch-srv2.lab.eng.blr.redhat.com:/rhs/brick1/b1
Brick3: rhs-arch-srv3.lab.eng.blr.redhat.com:/rhs/brick1/b1
Brick4: rhs-arch-srv4.lab.eng.blr.redhat.com:/rhs/brick1/b1
Brick5: inception.lab.eng.blr.redhat.com:/rhs/brick2/b2
Brick6: rhs-arch-srv2.lab.eng.blr.redhat.com:/rhs/brick2/b2
Brick7: rhs-arch-srv3.lab.eng.blr.redhat.com:/rhs/brick2/b2
Brick8: rhs-arch-srv4.lab.eng.blr.redhat.com:/rhs/brick2/b2
Brick9: inception.lab.eng.blr.redhat.com:/rhs/brick3/b3
Brick10: rhs-arch-srv2.lab.eng.blr.redhat.com:/rhs/brick3/b3
Brick11: rhs-arch-srv3.lab.eng.blr.redhat.com:/rhs/brick3/b3
Brick12: rhs-arch-srv4.lab.eng.blr.redhat.com:/rhs/brick3/b3
Options Reconfigured:
features.uss: enable
features.barrier: disable
client.event-threads: 4
server.event-threads: 5
features.quota: on
performance.readdir-ahead: on
auto-delete: disable
snap-max-soft-limit: 90
snap-max-hard-limit: 256

bt  :
===
om_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-11.el6.x86_64
libselinux-2.0.94-5.8.el6.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64
openssl-1.0.1e-30.el6_6.5.x86_64 readline-6.0-4.el6.x86_64
zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x0000003f236093a0 in ?? ()
#1  0x00000033e4425060 in gf_log_set_log_buf_size (buf_size=0) at logging.c:256
#2  0x00000033e44251ff in gf_log_disable_suppression_before_exit
(ctx=0x22b3010) at logging.c:427
#3  0x00000033e443bac5 in gf_print_trace (signum=11, ctx=0x22b3010) at
common-utils.c:493
#4  0x0000003f232326a0 in ?? ()
#5  0x0000000000000000 in ?? ()


[root at inception core]# gluster v status vol0
Status of volume: vol0
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick inception.lab.eng.blr.redhat.com:/rhs
/brick1/b1                                  49155     0          Y       24032
Brick rhs-arch-srv2.lab.eng.blr.redhat.com:
/rhs/brick1/b1                              49155     0          Y       1709 
Brick rhs-arch-srv3.lab.eng.blr.redhat.com:
/rhs/brick1/b1                              49155     0          Y       1212 
Brick rhs-arch-srv4.lab.eng.blr.redhat.com:
/rhs/brick1/b1                              49155     0          Y       27968
Brick inception.lab.eng.blr.redhat.com:/rhs
/brick2/b2                                  49156     0          Y       24045
Brick rhs-arch-srv2.lab.eng.blr.redhat.com:
/rhs/brick2/b2                              49156     0          Y       1722 
Brick rhs-arch-srv3.lab.eng.blr.redhat.com:
/rhs/brick2/b2                              49156     0          Y       1226 
Brick rhs-arch-srv4.lab.eng.blr.redhat.com:
/rhs/brick2/b2                              49156     0          Y       27981
Brick inception.lab.eng.blr.redhat.com:/rhs
/brick3/b3                                  49157     0          Y       24058
Brick rhs-arch-srv2.lab.eng.blr.redhat.com:
/rhs/brick3/b3                              49157     0          Y       1735 
Brick rhs-arch-srv3.lab.eng.blr.redhat.com:
/rhs/brick3/b3                              49157     0          Y       1239 
Brick rhs-arch-srv4.lab.eng.blr.redhat.com:
/rhs/brick3/b3                              49157     0          Y       27994
Snapshot Daemon on localhost                N/A       N/A        N       31079
NFS Server on localhost                     2049      0          Y       31087
Self-heal Daemon on localhost               N/A       N/A        Y       24079
Quota Daemon on localhost                   N/A       N/A        Y       24119
Snapshot Daemon on rhs-arch-srv2.lab.eng.bl
r.redhat.com                                49845     0          Y       24540
NFS Server on rhs-arch-srv2.lab.eng.blr.red
hat.com                                     2049      0          Y       24548
Self-heal Daemon on rhs-arch-srv2.lab.eng.b
lr.redhat.com                               N/A       N/A        Y       1756 
Quota Daemon on rhs-arch-srv2.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       1776 
Snapshot Daemon on rhs-arch-srv4.lab.eng.bl
r.redhat.com                                49845     0          Y       27158
NFS Server on rhs-arch-srv4.lab.eng.blr.red
hat.com                                     2049      0          Y       27171
Self-heal Daemon on rhs-arch-srv4.lab.eng.b
lr.redhat.com                               N/A       N/A        Y       28015
Quota Daemon on rhs-arch-srv4.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       28034
Snapshot Daemon on rhs-arch-srv3.lab.eng.bl
r.redhat.com                                49845     0          Y       31320
NFS Server on rhs-arch-srv3.lab.eng.blr.red
hat.com                                     2049      0          Y       31330
Self-heal Daemon on rhs-arch-srv3.lab.eng.b
lr.redhat.com                               N/A       N/A        Y       1260 
Quota Daemon on rhs-arch-srv3.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       1279 

Task Status of Volume vol0
------------------------------------------------------------------------------
There are no active volume tasks

Following is the bt from crash:

#0  __pthread_mutex_lock (mutex=0x320) at pthread_mutex_lock.c:50
#1  0x00000033e4425060 in gf_log_set_log_buf_size (buf_size=0) at logging.c:256
#2  0x00000033e44251ff in gf_log_disable_suppression_before_exit
(ctx=0x22b3010) at logging.c:427
#3  0x00000033e443bac5 in gf_print_trace (signum=11, ctx=0x22b3010) at
common-utils.c:493
#4  <signal handler called>
#5  0x00000033e444f731 in __gf_free (free_ptr=0x7f911ef33c50) at mem-pool.c:231
#6  0x00000033e443da02 in gf_timer_proc (ctx=0x7f911ef35630) at timer.c:207
#7  0x0000003f236079d1 in start_thread (arg=0x7f8eb197b700) at
pthread_create.c:301
#8  0x0000003f232e88fd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115

In the test case multiple snapshots were created and then activated. And after
activation snapshots were accessed using USS. while accessing these snapshot
the crash is seen.

code wise this crash is happening during the timer thread destruction. Timer
thread is destroyed as part of glfs_fini.

Normally glfs_fini is called when snapshots are deactivated or deleted. But in
this case no snapshots were deleted or deactivated. In this case glfs_fini is
called due to failure in glfs_init.

For some reason the snapshot brick is not in started state leading to failure
in glfs_init. We could not figure out the exact cause of this since the brick
and snapshot logs were missing from the sos-report.

But anyway when glfs_init fails we call glfs_fini to clean up the resources
allocated. In the timer thread current THIS is overwritten and never restored,
leading to wrong value of THIS which causes segmentation fault in __gf_free
function.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=HKXwBy22vL&a=cc_unsubscribe


More information about the Bugs mailing list