[Gluster-users] Assertion failed: ec_get_inode_size

Ashish Pandey aspandey at redhat.com
Thu Apr 14 17:07:33 UTC 2016


Hi Serkan, 

Could you also provide us the statedump of all the brick processes and clients? 



Commands to generate statedumps for brick processes/nfs server/quotad 

For bricks: gluster volume statedump <volname> 

For nfs server: gluster volume statedump <volname> nfs 
We can find the directory where statedump files are created using 'gluster --print-statedumpdir' 
Also, the mount logs would help us to debug the issue. 

Ashish 

----- Original Message -----

From: "Serkan Çoban" <cobanserkan at gmail.com> 
To: "Gluster Users" <gluster-users at gluster.org>, "Gluster Devel" <gluster-devel at gluster.org> 
Sent: Thursday, April 14, 2016 6:27:10 PM 
Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size 

Here is the related brick log: 

/var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E 
[inodelk.c:309:__inode_unlock_lock] 0-v0-locks: Matching lock not 
found for unlock 0-9223372036854775807, by 94d29e885e7f0000 on 
0x7f037413b990 
/var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E 
[MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk] 
0-v0-server: 712984: INODELK 
/workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605 
(1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument) [Invalid 
argument] 

On Thu, Apr 14, 2016 at 3:25 PM, Serkan Çoban <cobanserkan at gmail.com> wrote: 
> Hi, 
> 
> During read/write tests to a 78x(16+4) distributed disperse volume 
> from 50 clients, One clients hangs on read/write with the following 
> logs: 
> 
> [2016-04-14 11:11:04.728580] W [MSGID: 122056] 
> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata 
> in answers of 'LOOKUP' 
> [2016-04-14 11:11:04.728624] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on 
> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=DFFFF, 
> bad=20000) 
> [2016-04-14 11:11:04.736689] I [MSGID: 122058] 
> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2: 
> name heal successful on FFFFF 
> [2016-04-14 11:29:26.718036] W [MSGID: 122056] 
> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata 
> in answers of 'LOOKUP' 
> [2016-04-14 11:29:26.718121] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on 
> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=EFFFF, 
> bad=10000) 
> [2016-04-14 11:29:42.501760] I [MSGID: 122058] 
> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2: 
> name heal successful on FFFFF 
> [2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat] 
> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91) 
> [0x7f5ec9f942b1] 
> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57) 
> [0x7f5ec9f94497] 
> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4) 
> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop, 
> fop->locks[0].lock->loc.inode, &cbk->iatt[0].ia_size) 
> [2016-04-14 11:31:25.722372] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722411] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722450] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722477] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722503] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722577] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722605] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722742] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722794] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722818] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722840] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722883] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722906] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722958] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.722983] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.723037] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-56: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.723045] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-53: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.725044] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-59: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.741338] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-58: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.746602] E [MSGID: 114031] 
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-57: remote 
> operation failed [Invalid argument] 
> [2016-04-14 11:31:25.746629] W [MSGID: 122015] 
> [ec-common.c:1675:ec_unlocked] 0-v0-disperse-2: entry/inode unlocking 
> failed (FSTAT) [Invalid argument] 
> [2016-04-14 11:31:25.746687] E [ec-common.c:1639:ec_lock_unfreeze] 
> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae) 
> [0x7f5ec9fa009e] 
> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40) 
> [0x7f5ec9f97fc0] 
> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0x91) 
> [0x7f5ec9f975d1] ) 0-: Assertion failed: lock->refs == lock->inserted 
> [2016-04-14 11:31:25.746719] E [ec-common.c:1641:ec_lock_unfreeze] 
> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae) 
> [0x7f5ec9fa009e] 
> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40) 
> [0x7f5ec9f97fc0] 
> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0xf2) 
> [0x7f5ec9f97632] ) 0-: Assertion failed: list_empty(&lock->waiting) && 
> list_empty(&lock->owners) 
> [2016-04-14 11:40:54.139418] W [MSGID: 122056] 
> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-0: Mismatching xdata 
> in answers of 'LOOKUP' 
> [2016-04-14 11:40:54.139554] W [MSGID: 122053] 
> [ec-common.c:116:ec_check_status] 0-v0-disperse-0: Operation failed on 
> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=2FFFF, 
> bad=D0000) 
> [2016-04-14 11:40:54.143400] I [MSGID: 122058] 
> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-0: /workdir/raw_output2: 
> name heal successful on FFFFF 
> The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] 
> 0-v0-disperse-0: Mismatching xdata in answers of 'LOOKUP'" repeated 2 
> times between [2016-04-14 11:40:54.139418] and [2016-04-14 
> 11:40:54.139547] 
> [2016-04-14 11:42:31.471317] W [MSGID: 122056] 
> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-45: Mismatching 
> xdata in answers of 'LOOKUP' 
> The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] 
> 0-v0-disperse-45: Mismatching xdata in answers of 'LOOKUP'" repeated 8 
> times between [2016-04-14 11:42:31.471317] and [2016-04-14 
> 11:42:31.471557] 
> 
> Gluster version is 3.7.10 
> Any thoughts? 
> 
> Serkan 
_______________________________________________ 
Gluster-users mailing list 
Gluster-users at gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160414/51527367/attachment.html>


More information about the Gluster-users mailing list