[Bugs] [Bug 1379662] New: IO hang on ganesha mount during remove brick operation.

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1379662

            Bug ID: 1379662
           Summary: IO hang on ganesha mount during remove brick
                    operation.
           Product: Red Hat Gluster Storage
           Version: 3.2
         Component: nfs-ganesha
          Keywords: Triaged
          Severity: high
          Priority: high
          Assignee: kkeithle at redhat.com
          Reporter: sraj at redhat.com
        QA Contact: sraj at redhat.com
                CC: bugs at gluster.org, jthottan at redhat.com,
                    kkeithle at redhat.com, mvignesh at redhat.com,
                    ndevos at redhat.com, rhs-bugs at redhat.com,
                    skoduri at redhat.com, storage-qa-internal at redhat.com
        Depends On: 1365626



+++ This bug was initially created as a clone of Bug #1365626 +++

Description of problem:

IO hang on ganesha mount during remove brick operation

Version-Release number of selected component (if applicable):

[root at dhcp43-133 ~]# rpm -qa|grep glusterfs
glusterfs-libs-3.8.1-0.4.git56fcf39.el7rhgs.x86_64
glusterfs-fuse-3.8.1-0.4.git56fcf39.el7rhgs.x86_64
glusterfs-3.8.1-0.4.git56fcf39.el7rhgs.x86_64
glusterfs-api-3.8.1-0.4.git56fcf39.el7rhgs.x86_64
glusterfs-cli-3.8.1-0.4.git56fcf39.el7rhgs.x86_64
glusterfs-ganesha-3.8.1-0.4.git56fcf39.el7rhgs.x86_64
glusterfs-client-xlators-3.8.1-0.4.git56fcf39.el7rhgs.x86_64
glusterfs-server-3.8.1-0.4.git56fcf39.el7rhgs.x86_64
glusterfs-geo-replication-3.8.1-0.4.git56fcf39.el7rhgs.x86_64

[root at dhcp43-133 ~]# rpm -qa|grep ganesha
nfs-ganesha-gluster-2.4-0.dev.26.el7rhgs.x86_64
nfs-ganesha-2.4-0.dev.26.el7rhgs.x86_64
glusterfs-ganesha-3.8.1-0.4.git56fcf39.el7rhgs.x86_64

How reproducible:

Once

Steps to Reproduce:
1.Create a 6x2 dist-rep volume and enable ganesha on the volume.

2.Do a subdir v4 mount on the client

mount -t nfs -o vers=4 10.70.40.192:/newvolume/subdir /mnt1470753422.46

3.Start creating nested dir and files

for i in {1..30}; do mkdir /mnt1470753422.46/a$i;  for j in {1..50}; do mkdir
/mnt1470753422.46/a$i/b$j; for k in {1..50}; do touch
/mnt1470753422.46/a$i/b$j/c$k; done done done

4.Start the remove brick operation:

gluster volume remove-brick newvolume replica 2 
dhcp43-133.lab.eng.blr.redhat.com:/bricks/brick1/newvolume_brick0
dhcp41-206.lab.eng.blr.redhat.com:/bricks/brick1/newvolume_brick1 start

5. Once the remove brick operation is complete, commit the brick removal

gluster volume  remove-brick newvolume replica 2 
dhcp43-133.lab.eng.blr.redhat.com:/bricks/brick1/newvolume_brick0
dhcp41-206.lab.eng.blr.redhat.com:/bricks/brick1/newvolume_brick1 commit 

6. Observe that the IO hangs on the client and following messages are seen in
/var/log/ganesha.log

[root at dhcp46-206 ~]# ps -ef|grep mkdir
root      9288  9283  0 20:00 ?        00:00:02 bash -c cd /root && for i in
{1..30}; do mkdir /mnt1470753422.46/a$i;  for j in {1..50}; do mkdir
/mnt1470753422.46/a$i/b$j; for k in {1..50}; do touch
/mnt1470753422.46/a$i/b$j/c$k; done done done

09/08/2016 19:29:53 : epoch 57a9cca6 : dhcp43-133.lab.eng.blr.redhat.com :
ganesha.nfsd-26092[dbus_heartbeat] posix2fsal_error :FSAL :CRIT :Mapping
107(default) to ERR_FSAL_SERVERFAULT
09/08/2016 19:29:53 : epoch 57a9cca6 : dhcp43-133.lab.eng.blr.redhat.com :
ganesha.nfsd-26092[dbus_heartbeat] glusterfs_close_my_fd :FSAL :CRIT :Error :
close returns with Transport endpoint is not connected
09/08/2016 19:29:53 : epoch 57a9cca6 : dhcp43-133.lab.eng.blr.redhat.com :
ganesha.nfsd-26092[dbus_heartbeat] mdcache_lru_clean :INODE LRU :CRIT :Error
closing file in cleanup: Undefined server error
09/08/2016 19:29:53 : epoch 57a9cca6 : dhcp43-133.lab.eng.blr.redhat.com :
ganesha.nfsd-26092[dbus_heartbeat] posix2fsal_error :FSAL :CRIT :Mapping
107(default) to ERR_FSAL_SERVERFAULT
09/08/2016 19:29:53 : epoch 57a9cca6 : dhcp43-133.lab.eng.blr.redhat.com :
ganesha.nfsd-26092[dbus_heartbeat] glusterfs_close_my_fd :FSAL :CRIT :Error :
close returns with Transport endpoint is not connected
09/08/2016 19:29:53 : epoch 57a9cca6 : dhcp43-133.lab.eng.blr.redhat.com :
ganesha.nfsd-26092[dbus_heartbeat] mdcache_lru_clean :INODE LRU :CRIT :Error
closing file in cleanup: Undefined server error

Actual results:

IO hang on ganesha mount during remove brick operation.

Expected results:

Additional info:

sosreport and logs will be attached

--- Additional comment from Shashank Raj on 2016-08-09 13:37:01 EDT ---

sosreport and logs can be found under
http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1365626

--- Additional comment from Niels de Vos on 2016-09-12 01:39:46 EDT ---

All 3.8.x bugs are now reported against version 3.8 (without .x). For more
information, see
http://www.gluster.org/pipermail/gluster-devel/2016-September/050859.html


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1365626
[Bug 1365626] IO hang on ganesha mount during remove brick operation.
-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=uGWU2ejWmk&a=cc_unsubscribe