[Gluster-users] Need some help on Mismatching xdata / Failed combine iatt / Too many fd

[Ashish Pandey]

Hi Chen,

Could you please send us following logs- 
1 - brick logs - under /var/log/messages/brick/
2 - mount logs

Also some information like what kind of IO was happening (read,write, unlink, rename on different mount) to understand this issue in a better way.

---
Ashish

----- Original Message -----
From: "陈陈" <chenchen at smartquerier.com>
To: gluster-users at gluster.org
Sent: Friday, March 25, 2016 8:59:04 AM
Subject: [Gluster-users] Need some help on Mismatching xdata / Failed combine iatt / Too many fd

Hi Everyone,

I have a "2 x (4 + 2) = 12 Distributed-Disperse" volume. After upgraded 
to 3.7.8 I noticed the volume is frequently out of service. The 
glustershd.log is flooded by:

[ec-combine.c:866:ec_combine_check] 0-mainvol-disperse-1: Mismatching 
xdata in answers of 'LOOKUP'"
[ec-common.c:116:ec_check_status] 0-mainvol-disperse-1: Operation failed 
on some subvolumes (up=3F, mask=3F, remaining=0, good=1E, bad=21)
[ec-common.c:71:ec_heal_report] 0-mainvol-disperse-1: Heal failed 
[Invalid argument]
[ec-combine.c:206:ec_iatt_combine] 0-mainvol-disperse-0: Failed to 
combine iatt (inode: xxx, links: 1-1, uid: 1000-1000, gid: 1000-1000, 
rdev: 0-0, size: xxx-xxx, mode: 100600-100600)

in normal working state, and sometimes 1000+ lines of:

[client-rpc-fops.c:466:client3_3_open_cbk] 0-mainvol-client-7: remote 
operation failed. Path: <gfid:xxxx> (xxxx) [Too many open files]

and the brick went offline. "top open" showed "Max open fds: 899195".

Can anyone suggest me what happened, and what should I do? I was trying 
to deal with the terrible IOPS problem but things got even worse.

Each Server has 2 x E5-2630v3 (32threads/server), 32GB RAM. Additional 
infos are in the attachements. Many thanks.

Sincerely yours,
Chen

-- 
Chen Chen
Shanghai SmartQuerier Biotechnology Co., Ltd.
Add: Room 410, 781 Cai Lun Road, China (Shanghai) Pilot Free Trade Zone
         Shanghai 201203, P. R. China
Mob: +86 15221885893
Email: chenchen at smartquerier.com
Web: www.smartquerier.com

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users