[Gluster-users] fuse 3.3.1 fails/crashes on flush on Distributed-Striped-Replicate volume

Wed Jul 10 18:38:51 UTC 2013

I had the same problem with striped-replicated. 

https://bugzilla.redhat.com/show_bug.cgi?id=861423

Best, 
Michael 

-----Original Message-----
From: Benedikt Fraunhofer [mailto:benedikt.fraunhofer.l.gluster.fxy-3zz-w8t at traced.net] 
Sent: Monday, July 08, 2013 3:43 AM
To: Gluster-users at gluster.org
Subject: [Gluster-users] fuse 3.3.1 fails/crashes on flush on Distributed-Striped-Replicate volume

Hello List,

before filing a bug I wanted to check with the community if this is a known issue.

I'm running gluster 3.3.1 from the semiosis PPA on ubuntu 12.04.
I created a 2x2x2 volume:

gluster> volume info

Volume Name: gv0-s2-r2
Type: Distributed-Striped-Replicate
Volume ID: 8e1f1e86-63c0-4c27-9c90-fef92bdd0ca1
Status: Started
Number of Bricks: 2 x 2 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 172.31.32.33:/gluster/brick0
Brick2: 172.31.32.34:/gluster/brick0
Brick3: 172.31.32.37:/gluster/brick0
Brick4: 172.31.32.38:/gluster/brick0
Brick5: 172.31.32.39:/gluster/brick0
Brick6: 172.31.32.40:/gluster/brick0
Brick7: 172.31.32.42:/gluster/brick0
Brick8: 172.31.32.47:/gluster/brick0

which mounts fine, but it started to smell fishy when vim wasn't able to write its swapfile, but I ignored that.
I hacked my stress-script together which makes the fuse-mount instantly unuseable. The script just creates random file- and directory-names and tries to write data to these:
Essentially it does something like:
 mkdir /mnt/a/C/C2/C2s/
 touch /mnt/a/C/C2/C2s/C2syqS30tsA5sctL2C8j22zd04VWsZDj3fZSWCK8Uo
 head -c 12837 /dev/zero >
/mnt/a/C/C2/C2s/C2syqS30tsA5sctL2C8j22zd04VWsZDj3fZSWCK8Uo

which results in:
 stress.sh: line 28:
/mnt/a/y/yK/yK2/yK2ujHHPRJYukCaqFhYkAN5MVDJxoAmTu89nM5auLA: Software caused connection abort

the following calls result in:
 mkdir: cannot create directory `/mnt/a/l': Transport endpoint is not connected because the mount is gone. I thought I hit the ext4 64bit-hash bug, reformatted to xfs, but same game.

note that this only happens when done quickly; if you wait between directory creation, touch and write it works:

root at gluster-71:/mnt/a# mkdir -p a/b/c/d root at gluster-71:/mnt/a# touch a/b/c/d/e root at gluster-71:/mnt/a# head -c 32 /dev/zero > a/b/c/d/e root at gluster-71:/mnt/a# mkdir -p b/c/d/; touch b/c/d/e; head -c 32 /dev/zero >b/c/d/e
-bash: b/c/d/e: Software caused connection abort

logging says:

[2013-07-05 15:24:05.764677] I [fuse-bridge.c:3376:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.19
[2013-07-05 15:24:05.765679] I
[afr-common.c:1965:afr_set_root_inode_on_first_lookup]
0-gv0-s2-r2-replicate-0: added root inode
[2013-07-05 15:24:05.765747] I
[afr-common.c:1965:afr_set_root_inode_on_first_lookup]
0-gv0-s2-r2-replicate-1: added root inode
[2013-07-05 15:24:05.767220] I
[afr-common.c:1965:afr_set_root_inode_on_first_lookup]
0-gv0-s2-r2-replicate-3: added root inode
[2013-07-05 15:24:05.767304] I
[afr-common.c:1965:afr_set_root_inode_on_first_lookup]
0-gv0-s2-r2-replicate-2: added root inode
[2013-07-05 15:24:09.709367] I [dict.c:317:dict_get]
(-->/usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_create_unwind+0x13c)
[0x7fd39d7262ac]
(-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_create_cbk+0x5fb)
[0x7fd39d507bab]
(-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_ctx_handle+0x90)
[0x7fd39d510150]))) 0-dict: !this ||
key=trusted.gv0-s2-r2-stripe-1.stripe-size
[2013-07-05 15:24:09.709446] E
[stripe-helpers.c:268:stripe_ctx_handle] 0-gv0-s2-r2-stripe-1: Failed to get stripe-size
[2013-07-05 15:24:09.714424] W [fuse-bridge.c:968:fuse_err_cbk]
0-glusterfs-fuse: 17: FLUSH() ERR => -1 (Invalid argument)
[2013-07-05 15:24:57.247208] I [dict.c:317:dict_get]
(-->/usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_create_unwind+0x13c)
[0x7fd39d7262ac]
(-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_create_cbk+0x5fb)
[0x7fd39d507bab]
(-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_ctx_handle+0x90)
[0x7fd39d510150]))) 0-dict: !this ||
key=trusted.gv0-s2-r2-stripe-0.stripe-size
[2013-07-05 15:24:57.247436] E
[stripe-helpers.c:268:stripe_ctx_handle] 0-gv0-s2-r2-stripe-0: Failed to get stripe-size pending frames:
frame : type(1) op(TRUNCATE)
frame : type(1) op(OPEN)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)

patchset: git://git.gluster.com/glusterfs.git
signal received: 8
time of crash: 2013-07-05 15:24:57
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.3.1
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0)[0x7fd3a1d224a0]
/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_truncate+0x28c)[0x7fd39d50287c]
/usr/lib/glusterfs/3.3.1/xlator/cluster/distribute.so(dht_truncate+0x172)[0x7fd39d2dd8c2]
/usr/lib/glusterfs/3.3.1/xlator/performance/write-behind.so(wb_truncate+0x49e)[0x7fd39d0a769e]
/usr/lib/glusterfs/3.3.1/xlator/performance/read-ahead.so(ra_truncate+0x1b2)[0x7fd39ce96302]
/usr/lib/glusterfs/3.3.1/xlator/performance/io-cache.so(ioc_truncate+0x14c)[0x7fd39cc84aec]
/usr/lib/libglusterfs.so.0(default_truncate+0x124)[0x7fd3a271c584]
/usr/lib/glusterfs/3.3.1/xlator/performance/md-cache.so(mdc_truncate+0x13a)[0x7fd39c8625fa]
/usr/lib/glusterfs/3.3.1/xlator/debug/io-stats.so(io_stats_truncate+0x14f)[0x7fd39c64bb3f]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x17b7f)[0x7fd3a0001b7f]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bd6)[0x7fd39fff0bd6]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6ae2)[0x7fd39fff0ae2]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bce)[0x7fd39fff0bce]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_continue+0x41)[0x7fd39fff0851]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_inode+0x25)[0x7fd39fff0975]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6b05)[0x7fd39fff0b05]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bae)[0x7fd39fff0bae]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_and_resume+0x28)[0x7fd39fff1128]
/usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x1af74)[0x7fd3a0004f74]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fd3a20b2e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd3a1ddfccd]

I tried to read all the bugs which include "Failed to get stripe-size"
but they don't include anything fuse-related; anyhow it reads like
https://bugzilla.redhat.com/show_bug.cgi?id=861423
which
http://review.gluster.org/3904
should be solving?

Thx in advance

 Benedikt.
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users