[Gluster-users] Assertion failed: ec_get_inode_size

[Serkan Çoban]

Hi,

During read/write tests to a 78x(16+4) distributed disperse volume
from 50 clients, One clients hangs on read/write with the following
logs:

[2016-04-14 11:11:04.728580] W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata
in answers of 'LOOKUP'
[2016-04-14 11:11:04.728624] W [MSGID: 122053]
[ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on
some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=DFFFF,
bad=20000)
[2016-04-14 11:11:04.736689] I [MSGID: 122058]
[ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2:
name heal successful on FFFFF
[2016-04-14 11:29:26.718036] W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata
in answers of 'LOOKUP'
[2016-04-14 11:29:26.718121] W [MSGID: 122053]
[ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on
some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=EFFFF,
bad=10000)
[2016-04-14 11:29:42.501760] I [MSGID: 122058]
[ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2:
name heal successful on FFFFF
[2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat]
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91)
[0x7f5ec9f942b1]
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57)
[0x7f5ec9f94497]
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4)
[0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop,
fop->locks[0].lock->loc.inode, &cbk->iatt[0].ia_size)
[2016-04-14 11:31:25.722372] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722411] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722450] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722477] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722503] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722577] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722605] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722742] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722794] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722818] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722840] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722883] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722906] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722958] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722983] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.723037] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-56: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.723045] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-53: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.725044] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-59: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.741338] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-58: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.746602] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-57: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.746629] W [MSGID: 122015]
[ec-common.c:1675:ec_unlocked] 0-v0-disperse-2: entry/inode unlocking
failed (FSTAT) [Invalid argument]
[2016-04-14 11:31:25.746687] E [ec-common.c:1639:ec_lock_unfreeze]
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)
[0x7f5ec9fa009e]
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40)
[0x7f5ec9f97fc0]
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0x91)
[0x7f5ec9f975d1] ) 0-: Assertion failed: lock->refs == lock->inserted
[2016-04-14 11:31:25.746719] E [ec-common.c:1641:ec_lock_unfreeze]
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)
[0x7f5ec9fa009e]
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_unlocked+0x40)
[0x7f5ec9f97fc0]
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_lock_unfreeze+0xf2)
[0x7f5ec9f97632] ) 0-: Assertion failed: list_empty(&lock->waiting) &&
list_empty(&lock->owners)
[2016-04-14 11:40:54.139418] W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check] 0-v0-disperse-0: Mismatching xdata
in answers of 'LOOKUP'
[2016-04-14 11:40:54.139554] W [MSGID: 122053]
[ec-common.c:116:ec_check_status] 0-v0-disperse-0: Operation failed on
some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=2FFFF,
bad=D0000)
[2016-04-14 11:40:54.143400] I [MSGID: 122058]
[ec-heal.c:2340:ec_heal_do] 0-v0-disperse-0: /workdir/raw_output2:
name heal successful on FFFFF
The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check]
0-v0-disperse-0: Mismatching xdata in answers of 'LOOKUP'" repeated 2
times between [2016-04-14 11:40:54.139418] and [2016-04-14
11:40:54.139547]
[2016-04-14 11:42:31.471317] W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check] 0-v0-disperse-45: Mismatching
xdata in answers of 'LOOKUP'
The message "W [MSGID: 122056] [ec-combine.c:866:ec_combine_check]
0-v0-disperse-45: Mismatching xdata in answers of 'LOOKUP'" repeated 8
times between [2016-04-14 11:42:31.471317] and [2016-04-14
11:42:31.471557]

Gluster version is 3.7.10
Any thoughts?

Serkan