<div dir="ltr">Actually we just discovered that the heal info command was returning different things when executed on the different nodes of our 3-replica setup.<div>When we execute it on node2 we did not see the split brain reported "/" but if I execute it on node0 and node1 I am seeing:</div><div><br><div>x@gfs-vm001:~$ sudo gluster volume heal gv0 info | tee heal-info</div><div>Brick gfs-vm000:/gluster/brick/brick0</div><div><gfid:81289110-867b-42ff-ba3b-1373a187032b></div><div>/ - Is in split-brain</div><div><br></div><div>Status: Connected</div><div>Number of entries: 2</div><div><br></div><div>Brick gfs-vm001:/gluster/brick/brick0</div><div>/ - Is in split-brain</div><div><br></div><div><gfid:81289110-867b-42ff-ba3b-1373a187032b></div><div>Status: Connected</div><div>Number of entries: 2</div><div><br></div><div>Brick gfs-vm002:/gluster/brick/brick0</div><div>/ - Is in split-brain</div><div><br></div><div>Status: Connected</div><div>Number of entries: 1</div></div><div><br></div><div><br></div><div>I ran getfattr -d -m . -e hex /gluster/brick/brick0 on all three nodes and I am seeing node2 has slightly different attr:</div><div>node0:</div><div><div>sudo getfattr -d -m . -e hex /gluster/brick/brick0</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: gluster/brick/brick0</div><div>trusted.afr.gv0-client-2=0x000000000000000100000000</div><div>trusted.gfid=0x00000000000000000000000000000001</div><div>trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2</div></div><div><br></div><div>node1:</div><div><div>sudo getfattr -d -m . -e hex /gluster/brick/brick0</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: gluster/brick/brick0</div><div>trusted.afr.gv0-client-2=0x000000000000000100000000</div><div>trusted.gfid=0x00000000000000000000000000000001</div><div>trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2</div></div><div><br></div><div>node2:</div><div><div>sudo getfattr -d -m . -e hex /gluster/brick/brick0</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: gluster/brick/brick0</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.afr.gv0-client-0=0x000000000000000200000000</div><div>trusted.afr.gv0-client-1=0x000000000000000200000000</div><div>trusted.afr.gv0-client-2=0x000000000000000000000000</div><div>trusted.gfid=0x00000000000000000000000000000001</div><div>trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2</div></div><div><br></div><div>Where do I go from here? Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 3, 2018 at 11:54 AM, Anh Vo <span dir="ltr"><<a href="mailto:vtqanh@gmail.com" target="_blank">vtqanh@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I am trying to mount nfs to gluster volume and got mount.nfs failure. Looking at nfs.log I am seeing these entries<div><br></div><div>Heal info does not show the mentioned gfid (
<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">00000000-0000-0000-0000-<wbr>000000000001</span> ) being in split-brain. <br><div><div><br></div><div>[2018-07-03 18:16:27.694953] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_<wbr>common_res] 0-nfs-nfsv3: / => (XID: c3ac3cc5, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error))</div><div>[2018-07-03 18:16:28.204685] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_<wbr>common_res] 0-nfs-nfsv3: / => (XID: c4ac3cc5, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error))</div><div>The message "E [MSGID: 108008] [afr-read-txn.c:90:afr_read_<wbr>txn_refresh_done] 0-gv0-replicate-0: Failing STAT on gfid 00000000-0000-0000-0000-<wbr>000000000001: split-brain observed. [Input/output error]" repeated 2 times between [2018-07-03 18:16:27.694903] and [2018-07-03 18:17:02.310689]</div><div>[2018-07-03 18:17:02.310722] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_<wbr>common_res] 0-nfs-nfsv3: / => (XID: 2a6f2526, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error))</div><div>[2018-07-03 18:17:02.628990] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_<wbr>txn_refresh_done] 0-gv0-replicate-0: Failing STAT on gfid 00000000-0000-0000-0000-<wbr>000000000001: split-brain observed. [Input/output error]</div><div>[2018-07-03 18:17:02.629023] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_<wbr>common_res] 0-nfs-nfsv3: / => (XID: 2b6f2526, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error))</div><div>[2018-07-03 18:17:00.398601] I [MSGID: 108031] [afr-common.c:2458:afr_local_<wbr>discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-2</div><div>[2018-07-03 18:17:01.666671] W [MSGID: 108027] [afr-common.c:2821:afr_<wbr>discover_done] 0-gv0-replicate-0: no read subvols for /</div><div>[2018-07-03 18:51:43.509385] W [MSGID: 108027] [afr-common.c:2821:afr_<wbr>discover_done] 0-gv0-replicate-0: no read subvols for /</div><div>[2018-07-03 18:51:43.936826] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_<wbr>txn_refresh_done] 0-gv0-replicate-0: Failing STAT on gfid 00000000-0000-0000-0000-<wbr>000000000001: split-brain observed. [Input/output error]</div><div>[2018-07-03 18:51:43.936868] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_<wbr>common_res] 0-nfs-nfsv3: / => (XID: 19b1731e, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error))</div><div>[2018-07-03 18:51:44.278901] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_<wbr>common_res] 0-nfs-nfsv3: / => (XID: 1ab1731e, FSINFO: NFS: 5(I/O error), POSIX: 5(Input/output error))</div></div><div><br></div></div></div>
</blockquote></div><br></div>