[Bugs] [Bug 1141940] New: Mount -t glusterfs never completes and all file-system commands hang

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1141940

            Bug ID: 1141940
           Summary: Mount -t glusterfs never completes and all file-system
                    commands hang
           Product: GlusterFS
           Version: 3.4.2
         Component: fuse
          Severity: high
          Assignee: gluster-bugs at redhat.com
          Reporter: chalcogen_eg_oxygen at yahoo.com
                CC: bugs at gluster.org



Description of problem:

I have been experimenting with server replacement procedures in a
twin-replicated system say, between server1 and server2. I do the following:
- shutdown on server2.
- On server1, remove-brick of the bricks on server2 for all replicated volumes. 
- On server1, detach peer server2.

After this, if you 
- replace server2 with server3, 
- make bricks on server 3, 
- add them in replica 2 mode to the (now) distributed volumes on server1, 

and then attempt to mount the volumes, you might find that the mount.glusterfs
program is stalled. Also, any file-system operation on the mount hangs, and
cannot be interrupted (but kill -9 does terminate it). 

Here's an example:

root     21891 21890  0 16:47 ?        00:00:00 /bin/mount -t glusterfs
server3:testvol /mnt
root     21892 21891  0 16:47 ?        00:00:00 /bin/sh /sbin/mount.glusterfs
server3:testvol /mnt -o rw
root     21931 21892  0 16:47 ?        00:00:00 /bin/sh /sbin/mount.glusterfs
server3:testvol /mnt -o rw
root     21932 21931  0 16:47 ?        00:00:00 stat -c %i /mnt

I checked the wchan for 21932 (stat) and found that it is 'fuse_get_req'. The
funny thing here is that if I try to set any volume option, the problem
immediately corrects itself. For example, I tried to set the following 2
options for debug: server.statedump-path and diagnostics.client-log-level and
each time this caused the problem to get resolved. 

Version-Release number of selected component (if applicable):
glusterfs 3.4.2 over linux kernel 2.6.34.

How reproducible:
Intermittent.

Steps to Reproduce:
1. Take two servers, server1 and server2. Probe each other. 
2. Create bricks server1:/bricks/testvol and server2:/bricks/testvol
3. gluster volume create testvol replica 2 server1:/bricks/testvol and
server2:/bricks/testvol
4. gluster volume start testvol
5. On both servers, mount -t glusterfs server<n>:testvol /mnt
6. Turn off server2.
7. gluster volume remove-brick testvol replica 1 server2:/bricks/testvol
8. gluster peer detach server2.
9. Introduce server3.
10. On server3, gluster peer probe server1.
11. Create server3:/bricks/testvol
12. On server3, gluster volume add-brick testvol replica 2
server3:/bricks/testvol 
13 On server 3, mount -t glusterfs server3:testvol /mnt


Actual results:

At this point, the mount command might hang indefinitely just so:

root     21891 21890  0 16:47 ?        00:00:00 /bin/mount -t glusterfs
server3:testvol /mnt
root     21892 21891  0 16:47 ?        00:00:00 /bin/sh /sbin/mount.glusterfs
server3:testvol /mnt -o rw
root     21931 21892  0 16:47 ?        00:00:00 /bin/sh /sbin/mount.glusterfs
server3:testvol /mnt -o rw
root     21932 21931  0 16:47 ?        00:00:00 stat -c %i /mnt

Any command on the file-system such as ls/stat/df will also hang (and so will
the strace of these commands). If you set a volume option on testvol using the
'gluster volume set ...' command, the problem disappears immediately.


Expected results:
Mount should complete successfully.

Additional info:

If you have two volumes like testvol and testvol2, then if the issue is
observed on testvol, it will be also observed for testvol2. However, resolving
it for testvol using volume set commands will not cause it to automatically go
away for testvol2.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=4WzYSPJsN4&a=cc_unsubscribe