[Gluster-users] Assertion failed: ec_get_inode_size

Ashish Pandey aspandey at redhat.com
Fri Apr 15 08:53:40 UTC 2016 


I think this is the statesump of only one brick. 
We would required statedump from all the bricks and client process in case of fuse or nfs process if it is mounted through nfs. 

---- 
Ashish 

----- Original Message -----

From: "Serkan Çoban" <cobanserkan at gmail.com> 
To: "Ashish Pandey" <aspandey at redhat.com> 
Cc: "Gluster Users" <gluster-users at gluster.org>, "Gluster Devel" <gluster-devel at gluster.org> 
Sent: Friday, April 15, 2016 2:11:57 PM 
Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size 

Sorry for typo, brick state dump file. 

On Fri, Apr 15, 2016 at 11:41 AM, Serkan Çoban <cobanserkan at gmail.com> wrote: 
> Hi I reproduce the problem, brick log file is in below link: 
> https://www.dropbox.com/s/iy09j7mm2hrsf03/bricks-02.5677.dump.1460705370.gz?dl=0 
> 
> 
> On Thu, Apr 14, 2016 at 8:07 PM, Ashish Pandey <aspandey at redhat.com> wrote: 
>> Hi Serkan, 
>> 
>> Could you also provide us the statedump of all the brick processes and 
>> clients? 
>> 
>> Commands to generate statedumps for brick processes/nfs server/quotad 
>> 
>> For bricks: gluster volume statedump <volname> 
>> 
>> For nfs server: gluster volume statedump <volname> nfs 
>> 
>> 
>> We can find the directory where statedump files are created using 'gluster 
>> --print-statedumpdir' 
>> Also, the mount logs would help us to debug the issue. 
>> 
>> Ashish 
>> 
>> ________________________________ 
>> From: "Serkan Çoban" <cobanserkan at gmail.com> 
>> To: "Gluster Users" <gluster-users at gluster.org>, "Gluster Devel" 
>> <gluster-devel at gluster.org> 
>> Sent: Thursday, April 14, 2016 6:27:10 PM 
>> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size 
>> 
>> 
>> Here is the related brick log: 
>> 
>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E 
>> [inodelk.c:309:__inode_unlock_lock] 0-v0-locks: Matching lock not 
>> found for unlock 0-9223372036854775807, by 94d29e885e7f0000 on 
>> 0x7f037413b990 
>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E 
>> [MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk] 
>> 0-v0-server: 712984: INODELK 
>> /workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605 
>> (1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument) [Invalid 
>> argument] 
>> 
>> On Thu, Apr 14, 2016 at 3:25 PM, Serkan Çoban <cobanserkan at gmail.com> wrote: 
>>> Hi, 
>>> 
>>> During read/write tests to a 78x(16+4) distributed disperse volume 
>>> from 50 clients, One clients hangs on read/write with the following 
>>> logs: 
>>> 
>>> [2016-04-14 11:11:04.728580] W [MSGID: 122056] 
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata 
>>> in answers of 'LOOKUP' 
>>> [2016-04-14 11:11:04.728624] W [MSGID: 122053] 
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on 
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=DFFFF, 
>>> bad=20000) 
>>> [2016-04-14 11:11:04.736689] I [MSGID: 122058] 
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2: 
>>> name heal successful on FFFFF 
>>> [2016-04-14 11:29:26.718036] W [MSGID: 122056] 
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata 
>>> in answers of 'LOOKUP' 
>>> [2016-04-14 11:29:26.718121] W [MSGID: 122053] 
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on 
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=EFFFF, 
>>> bad=10000) 
>>> [2016-04-14 11:29:42.501760] I [MSGID: 122058] 
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2: 
>>> name heal successful on FFFFF 
>>> [2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat] 
>>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91) 
>>> [0x7f5ec9f942b1] 
>>> 
>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57) 
>>> [0x7f5ec9f94497] 
>>> 
>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4) 
>>> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop, 
>>> fop->locks[0].lock->loc.inode, &cbk->iatt[0].ia_size) 
>>> [2016-04-14 11:31:25.722372] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722411] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722450] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722477] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722503] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722577] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722605] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722742] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722794] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722818] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722840] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722883] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722906] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722958] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.722983] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.723037] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-56: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.723045] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-53: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.725044] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-59: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.741338] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-58: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.746602] E [MSGID: 114031] 
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-57: remote 
>>> operation failed [Invalid argument] 
>>> [2016-04-14 11:31:25.746629] W [MSGID: 122015] 
>>> [ec-common.c:1675:ec_unlocked] 0-v0-disperse-2: entry/inode unlocking 
>>> failed (FSTAT) [Invalid argument] 
>>> [2016-04-14 11:31:25.746687] E [ec-common.c:1639:ec_lock_unfreeze] 
>>> 
>>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae) 
>>> [0x7f5ec9fa009e] 
>>> 
>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40) 
>>> [0x7f5ec9f97fc0] 
>>> 
>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0x91) 
>>> [0x7f5ec9f975d1] ) 0-: Assertion failed: lock->refs == lock->inserted 
>>> [2016-04-14 11:31:25.746719] E [ec-common.c:1641:ec_lock_unfreeze] 
>>> 
>>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae) 
>>> [0x7f5ec9fa009e] 
>>> 
>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40) 
>>> [0x7f5ec9f97fc0] 
>>> 
>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0xf2) 
>>> [0x7f5ec9f97632] ) 0-: Assertion failed: list_empty(&lock->waiting) && 
>>> list_empty(&lock->owners) 
>>> [2016-04-14 11:40:54.139418] W [MSGID: 122056] 
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-0: Mismatching xdata 
>>> in answers of 'LOOKUP' 
>>> [2016-04-14 11:40:54.139554] W [MSGID: 122053] 
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-0: Operation failed on 
>>> some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=2FFFF, 
>>> bad=D0000) 
>>> [2016-04-14 11:40:54.143400] I [MSGID: 122058] 
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-0: /workdir/raw_output2: 
>>> name heal successful on FFFFF 
>>> The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] 
>>> 0-v0-disperse-0: Mismatching xdata in answers of 'LOOKUP'" repeated 2 
>>> times between [2016-04-14 11:40:54.139418] and [2016-04-14 
>>> 11:40:54.139547] 
>>> [2016-04-14 11:42:31.471317] W [MSGID: 122056] 
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-45: Mismatching 
>>> xdata in answers of 'LOOKUP' 
>>> The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] 
>>> 0-v0-disperse-45: Mismatching xdata in answers of 'LOOKUP'" repeated 8 
>>> times between [2016-04-14 11:42:31.471317] and [2016-04-14 
>>> 11:42:31.471557] 
>>> 
>>> Gluster version is 3.7.10 
>>> Any thoughts? 
>>> 
>>> Serkan 
>> _______________________________________________ 
>> Gluster-users mailing list 
>> Gluster-users at gluster.org 
>> http://www.gluster.org/mailman/listinfo/gluster-users 
>> 
_______________________________________________ 
Gluster-users mailing list 
Gluster-users at gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160415/ca42ba83/attachment.html>

More information about the Gluster-users mailing list