[Gluster-users] glusterfs-3.2.x after rebalance, read failed

[John Mark Walker]

Moving to gluster-users.

lierihanmei <lierihanmei at 163.com> wrote:

HI all.
I have got a problem. after remove-bricks and rebalanced, when i read some file ,it gets io errors.


version:    glusterfs 3.2.7
OS      :    Centos6


steps:
1, gluster volume create str stripe 2 ip:/data1 ip:/data2 ip:/data3 ip:/data4 ip:/data5 ip:/data6
2. gluster volume start str
3. mount -t glusterfs ip:/str /mnt/str
4. write some file into /mnt/str ( then from /mnt/str,  all files are OK, can be read and write)
5. gluster volume remove-brick str ip:/data3 ip:/data4(all files are OK too)
6, gluster rebalance str start
7, after rebalance complete(some files are OK, some can not access, get IO ERROR)
and i test glusterfs 3.2.5, 3.2.6, it get the same error.


after rebalance , the data are all in ip:/data5 and ip:/data6,   but in ip:/data1 and ip:/data2only link file like "---------T 1 root root 0 Mar 26 14:07 /data/str1/file11"  retains.


files which can not get all have link files.
the link file's attr:
[root at centos6-template dht]# getfattr -m . -e hex -d /data/str1/file11
getfattr: Removing leading '/' from absolute path names
# file: data/str1/file11
trusted.gfid=0xef8c64fa4516424a80bbcc65ad988780
trusted.glusterfs.dht.linkto=0x7374722d7374726970652d3100
trusted.str-stripe-0.stripe-count=0x3200
trusted.str-stripe-0.stripe-index=0x3000
trusted.str-stripe-0.stripe-size=0x31333130373200
the data file's attr:
[root at centos6-template dht]# getfattr -m . -e hex -d /data/str6/file11
getfattr: Removing leading '/' from absolute path names
# file: data/str6/file11
trusted.gfid=0xef8c64fa4516424a80bbcc65ad988780
trusted.str-stripe-2.stripe-count=0x3200
trusted.str-stripe-2.stripe-index=0x3100
trusted.str-stripe-2.stripe-size=0x31333130373200


files that can access:
[root at centos6-template dht]# getfattr -m . -e hex -d /data/str6/file6
getfattr: Removing leading '/' from absolute path names
# file: data/str6/file6
trusted.gfid=0xf1d6a5ec4f054926a65a1114ed3ed619
trusted.glusterfs.dht.linkto=0x7374722d7374726970652d3000
trusted.str-stripe-1.stripe-count=0x3200
trusted.str-stripe-1.stripe-index=0x3100
trusted.str-stripe-1.stripe-size=0x31333130373200
but from mountpoint , the attr is:
[root at centos6-template dht]# getfattr -m . -e hex -d file6
# file: file6
trusted.str-stripe-0.stripe-count=0x3200
trusted.str-stripe-0.stripe-index=0x3000
trusted.str-stripe-0.stripe-size=0x31333130373200


log of glusterfs:
nsport (str-dht2-client-2)
[2013-03-26 11:56:30.957186] T [rpc-clnt.c:1224:rpc_clnt_record] 0-str-dht2-client-3: Auth Info: pid: 0, uid: 0, gid: 0, owner: 0
[2013-03-26 11:56:30.957202] T [rpc-clnt.c:1125:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 344, payload: 216, rpc hdr: 128
[2013-03-26 11:56:30.957233] T [rpc-clnt.c:1429:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x11x Program: GlusterFS 3.1, ProgVers: 310, Proc: 27) to rpc-transport (str-dht2-client-3)
[2013-03-26 11:56:30.957479] T [rpc-clnt.c:638:rpc_clnt_reply_init] 0-str-dht2-client-2: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.1, ProgVers: 310, Proc: 27) from rpc-transport (str-dht2-client-2)
[2013-03-26 11:56:30.957536] T [rpc-clnt.c:638:rpc_clnt_reply_init] 0-str-dht2-client-3: received rpc message (RPC XID: 0x11x Program: GlusterFS 3.1, ProgVers: 310, Proc: 27) from rpc-transport (str-dht2-client-3)
[2013-03-26 11:56:30.957563] D [stripe.c:2673:stripe_open_lookup_cbk] 0-str-dht2-stripe-1: /file11: stripe info need to be healed
[2013-03-26 11:56:30.957576] E [stripe.c:2691:stripe_open_lookup_cbk] 0-str-dht2-stripe-1: stripe size not set
[2013-03-26 11:56:30.957591] D [dht-common.c:2527:dht_fd_cbk] 0-str-dht2-dht: subvolume str-dht2-stripe-1 returned -1 (Input/output error)
[2013-03-26 11:56:30.957618] W [quick-read.c:1640:qr_fstat_helper] 0-str-dht2-quick-read: open failed on path (/file11) (Input/output error), unwinding fstat call
[2013-03-26 11:56:30.957643] W [fuse-bridge.c:516:fuse_attr_cbk] 0-glusterfs-fuse: 7: FSTAT() /file11 => -1 (Input/output error)
[2013-03-26 11:56:30.957869] T [fuse-bridge.c:2086:fuse_flush] 0-glusterfs-fuse: 8: FLUSH 0x7f1f9eb69024
[2013-03-26 11:56:30.957956] T [fuse-bridge.c:994:fuse_err_cbk] 0-glusterfs-fuse: 8: FLUSH() ERR => 0
[2013-03-26 11:56:30.957998] T [fuse-bridge.c:2110:fuse_release] 0-glusterfs-fuse: 9: RELEASE 0x7f1f9eb69024
[2013-03-26 11:56:33.809728] T [rpc-clnt.c:1224:rpc_clnt_record] 0-str-dht2-client-0: Auth Info: pid: 9105, uid: 0, gid: 0, owner: 9105
[2013-03-26 11:56:33.809759] T [rpc-clnt.c:1125:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 284, payload: 156, rpc hdr: 128
[2013-03-26 11:56:33.809794] T [rpc-clnt.c:1429:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.1, ProgVers: 310, Proc: 27) to rpc-transport (str-dht2-client-0)
[2013-03-26 11:56:33.809824] T [rpc-clnt.c:1224:rpc_clnt_record] 0-str-dht2-client-1: Auth Info: pid: 9105, uid: 0, gid: 0, owner: 9105
[2013-03-26 11:56:33.809840] T [rpc-clnt.c:1125:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 284, payload: 156, rpc hdr: 128
[2013-03-26 11:56:33.809871] T [rpc-clnt.c:1429:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x12x Program: GlusterFS 3.1, ProgVers: 310, Proc: 27) to rpc-transport (str-dht2-client-1)
[2013-03-26 11:56:33.809901] T [rpc-clnt.c:1224:rpc_clnt_record] 0-str-dht2-client-2: Auth Info: pid: 9105, uid: 0, gid: 0, owner: 9105