<div dir="ltr">Hey Milos,<div><br></div><div>I see that gfid got healed for those directories from the getfattr output and the glfsheal log also has messages corresponding to deleting the entries on one brick as part of healing which then got recreated on the brick with the correct gfid. Can you run the "guster volume heal <volname>" & "gluster volume heal <volname> info" command and paste the output here?</div><div>If you still see entries pending heal, give the latest glustershd.log files from both the nodes along with the getfattr output of the files which are listed in the heal info output.</div><div><br></div><div>Regards,</div><div>Karthik</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 21, 2019 at 6:03 PM Milos Cuculovic <<a href="mailto:cuculovic@mdpi.com">cuculovic@mdpi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Sure:<div><br></div><div>brick1:</div><div><div>————————————————————————————————————————————————————————————</div><div>————————————————————————————————————————————————————————————</div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0xe358ff34504241d387efe1e76eb28bb0</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/5543982fab4b56060aa09f667a8ae617</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/5543982fab4b56060aa09f667a8ae617</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x33585a577c4a4c55b39b9abc07eacff1</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/a8b7f31775eebc8d1867e7f9de7b6eaf</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/a8b7f31775eebc8d1867e7f9de7b6eaf</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x99cdfe0773d74d7ab29592ce40cf47f3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/c1d3f3c2d7ae90e891e671e2f20d5d4b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/c1d3f3c2d7ae90e891e671e2f20d5d4b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x8a96a80ddf8a4163ac00bb12369389df</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/e5934699809a3b6dcfc5945f408b978b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/e5934699809a3b6dcfc5945f408b978b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x04d087a96cc6421ca7bc356fc20d7753</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0xdee76d06204d4bd9b7652eb7a79be1a5</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div></div><div>————————————————————————————————————————————————————————————</div><div><span style="font-family:Menlo;font-size:11px">sudo stat /data/data-cluster/dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59</span></div></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 40809094709 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:06:26.994047597 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:28.294689870 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.077654239 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/5543982fab4b56060aa09f667a8ae617</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/5543982fab4b56060aa09f667a8ae617'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 49399908865 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:07:20.342140927 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:28.318690015 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.133654344 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/a8b7f31775eebc8d1867e7f9de7b6eaf</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/a8b7f31775eebc8d1867e7f9de7b6eaf'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 53706303549 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:06:55.414097315 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:28.362690281 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.141654359 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/c1d3f3c2d7ae90e891e671e2f20d5d4b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/c1d3f3c2d7ae90e891e671e2f20d5d4b'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 57990935591 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:07:08.558120309 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:14.226604869 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.189654448 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/e5934699809a3b6dcfc5945f408b978b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/e5934699809a3b6dcfc5945f408b978b'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 62291339781 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:06:02.070003998 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:28.458690861 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.281654621 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 66574223479 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:28:10.826584325 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:10.834584374 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-20 14:06:07.937449353 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">root@storage3:/var/log/glusterfs# </span></div><div><div>————————————————————————————————————————————————————————————</div><div>————————————————————————————————————————————————————————————</div></div><div><br></div><div>brick2:<br><div>————————————————————————————————————————————————————————————</div><div>————————————————————————————————————————————————————————————</div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.dirty=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.storage2-client-0=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0xdee76d06204d4bd9b7652eb7a79be1a5</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/5543982fab4b56060aa09f667a8ae617</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/5543982fab4b56060aa09f667a8ae617</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.storage2-client-0=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x33585a577c4a4c55b39b9abc07eacff1</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/a8b7f31775eebc8d1867e7f9de7b6eaf</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/a8b7f31775eebc8d1867e7f9de7b6eaf</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.storage2-client-0=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x99cdfe0773d74d7ab29592ce40cf47f3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/c1d3f3c2d7ae90e891e671e2f20d5d4b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/c1d3f3c2d7ae90e891e671e2f20d5d4b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.storage2-client-0=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x8a96a80ddf8a4163ac00bb12369389df</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/e5934699809a3b6dcfc5945f408b978b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/e5934699809a3b6dcfc5945f408b978b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.storage2-client-0=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x04d087a96cc6421ca7bc356fc20d7753</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.dirty=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.storage2-client-0=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0xdee76d06204d4bd9b7652eb7a79be1a5</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">————————————————————————————————————————————————————————————</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 42232631305 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:06:26.994047597 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:28.294689870 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.078748131 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/5543982fab4b56060aa09f667a8ae617</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/5543982fab4b56060aa09f667a8ae617'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 78589109305 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:07:20.342140927 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:28.318690015 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.134748477 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/a8b7f31775eebc8d1867e7f9de7b6eaf</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/a8b7f31775eebc8d1867e7f9de7b6eaf'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 54972096517 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:06:55.414097315 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:28.362690281 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.162748650 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/c1d3f3c2d7ae90e891e671e2f20d5d4b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/c1d3f3c2d7ae90e891e671e2f20d5d4b'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 40821259275 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:07:08.558120309 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:14.226604869 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.194748848 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/e5934699809a3b6dcfc5945f408b978b</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/e5934699809a3b6dcfc5945f408b978b'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 15876654 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:06:02.070003998 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:28.458690861 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 13:01:03.282749392 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo stat /data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive/e7cdc94f60d390812a5f9754885e119e'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 33 <span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 0 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 49408944650 Links: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2019-03-20 11:28:10.826584325 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-20 11:28:10.834584374 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-20 14:06:07.940849268 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div><div><span style="font-variant-ligatures:no-common-ligatures">————————————————————————————————————————————————————————————</span></div><div><span style="font-variant-ligatures:no-common-ligatures">————————————————————————————————————————————————————————————</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"></span></div></span></div></div></div></div></div><div style="overflow-wrap: break-word;"><div><div><div><div><span style="font-variant-ligatures:no-common-ligatures"><div><span style="font-variant-ligatures:no-common-ligatures">The file is from brick 2 that I upgraded and started the heal on.</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div></span></div><div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br class="gmail-m_6575129410122648059Apple-interchange-newline">- Kindest regards,<br><br>Milos Cuculovic<br>IT Manager<br><br>---<br>MDPI AG<br>Postfach, CH-4020 Basel, Switzerland<br>Office: St. Alban-Anlage 66, 4052 Basel, Switzerland<br>Tel. +41 61 683 77 35<br>Fax +41 61 302 89 18<br><a href="mailto:cuculovic@mdpi.com" target="_blank">Email: cuculovic@mdpi.com</a><br>Skype: milos.cuculovic.mdpi<br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Disclaimer: The information and files contained in this message are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error, please notify me and delete this message from your system. You may not copy this message in its entirety or in part, or disclose its contents to anyone.</div>
</div>
<div><br><blockquote type="cite"><div>On 21 Mar 2019, at 13:05, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:</div><br class="gmail-m_6575129410122648059Apple-interchange-newline"><div><div dir="ltr">Can you give me the stat & getfattr output of all those 6 entries from both the bricks and the glfsheal-<volname>.log file from the node where you run this command?<div>Meanwhile can you also try running this with the source-brick option?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 21, 2019 at 5:22 PM Milos Cuculovic <<a href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Thank you Karthik,<div><br></div><div>I have run this for all files (see example below) and it says the file is not in split-brain:</div><div><br></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo gluster volume heal storage2 split-brain latest-mtime /dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Healing /dms/final_archive/41be9ff5ec05c4b1c989c6053e709e59 failed: File not in split-brain.</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Volume heal failed.</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br class="gmail-m_6575129410122648059gmail-m_-426258795379207644Apple-interchange-newline">- Kindest regards,<br><br>Milos Cuculovic<br>IT Manager<br><br>---<br>MDPI AG<br>Postfach, CH-4020 Basel, Switzerland<br>Office: St. Alban-Anlage 66, 4052 Basel, Switzerland<br>Tel. +41 61 683 77 35<br>Fax +41 61 302 89 18<br><a href="mailto:cuculovic@mdpi.com" target="_blank">Email: cuculovic@mdpi.com</a><br>Skype: milos.cuculovic.mdpi<br></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Disclaimer: The information and files contained in this message are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error, please notify me and delete this message from your system. You may not copy this message in its entirety or in part, or disclose its contents to anyone.</div>
</div>
<div><br><blockquote type="cite"><div>On 21 Mar 2019, at 12:36, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:</div><br class="gmail-m_6575129410122648059gmail-m_-426258795379207644Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Milos,<div><br></div><div>Thanks for the logs and the getfattr output.</div><div>From the logs I can see that there are 6 entries under the directory "/data/data-cluster/dms/final_archive" named</div><div>41be9ff5ec05c4b1c989c6053e709e59</div><div><div>5543982fab4b56060aa09f667a8ae617</div><div>a8b7f31775eebc8d1867e7f9de7b6eaf</div><div>c1d3f3c2d7ae90e891e671e2f20d5d4b</div><div>e5934699809a3b6dcfc5945f408b978b</div><div>e7cdc94f60d390812a5f9754885e119e</div></div><div><div>which are having gfid mismatch, so the heal is failing on this directory.</div><div><br></div></div><div>You can use the CLI option to resolve these files from gfid mismatch. You can use any of the 3 methods available:</div><div>1. bigger-file</div><div><span style="color:rgb(55,71,79);font-family:"Roboto Mono","Courier New",Courier,monospace;font-size:14.96px;background-color:rgba(236,236,236,0.5)">gluster volume heal <VOLNAME> split-brain bigger-file <FILE></span><br></div><div><span style="color:rgb(55,71,79);font-family:"Roboto Mono","Courier New",Courier,monospace;font-size:14.96px;background-color:rgba(236,236,236,0.5)"><br></span></div>2. latest-mtime<div><span style="color:rgb(55,71,79);font-family:"Roboto Mono","Courier New",Courier,monospace;font-size:14.96px;background-color:rgba(236,236,236,0.5)">gluster volume heal <VOLNAME> split-brain latest-mtime <FILE></span><span style="color:rgb(55,71,79);font-family:"Roboto Mono","Courier New",Courier,monospace;font-size:14.96px;background-color:rgba(236,236,236,0.5)"><br></span></div><div><span style="color:rgb(55,71,79);font-family:"Roboto Mono","Courier New",Courier,monospace;font-size:14.96px;background-color:rgba(236,236,236,0.5)"><br></span></div>3. source-brick<div><span style="color:rgb(55,71,79);font-family:"Roboto Mono","Courier New",Courier,monospace;font-size:14.96px;background-color:rgba(236,236,236,0.5)">gluster volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME> <FILE></span><span style="color:rgb(55,71,79);font-family:"Roboto Mono","Courier New",Courier,monospace;font-size:14.96px;background-color:rgba(236,236,236,0.5)"><br></span></div><div><span style="color:rgb(55,71,79);font-family:"Roboto Mono","Courier New",Courier,monospace;font-size:14.96px;background-color:rgba(236,236,236,0.5)"><br></span></div>where <FILE> must be absolute path w.r.t. the volume, starting with '/'.</div><div dir="ltr">If all those entries are directories then go for either latest-mtime/source-brick option.</div><div>After you resolve all these gfid-mismatches, run the "gluster volume heal <volname>" command. Then check the heal info and let me know the result.</div><div><br></div><div>Regards,</div><div>Karthik</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 21, 2019 at 4:27 PM Milos Cuculovic <<a href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Sure, thank you for following up.<div></div></div><div><div></div></div><div><div></div><div><br></div><div>About the commands, here is what I see:</div><div><br></div><div>brick1:</div><div>—————————————————————————————————————</div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo gluster volume heal storage2 info</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage3:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:256ca960-1601-4f0d-9b08-905c6fd52326> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Possibly undergoing heal</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage4:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Possibly undergoing heal</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 2</span></div></div><div><span style="font-variant-ligatures:no-common-ligatures">—————————————————————————————————————</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.dirty=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.storage2-client-1=0x000000000000000000000010</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x16c6a1e2b3fe4851972b998980097a87</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div></span></div><div>—————————————————————————————————————</div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">stat /data/data-cluster/dms/final_archive</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 3497984 <span class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 8768 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 26427748396 Links: 72123</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2018-10-09 04:22:40.514629044 +0200</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-21 11:55:37.382278863 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 11:55:37.382278863 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div></div><div>—————————————————————————————————————</div><div>—————————————————————————————————————</div><div><br></div><div>brick2:</div><div><div>—————————————————————————————————————</div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo gluster volume heal storage2 info</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage3:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:256ca960-1601-4f0d-9b08-905c6fd52326> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Possibly undergoing heal</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage4:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Possibly undergoing heal</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 2</span></div></div><div><span style="font-variant-ligatures:no-common-ligatures">—————————————————————————————————————</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo getfattr -d -m . -e hex /data/data-cluster/dms/final_archive</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">getfattr: Removing leading '/' from absolute path names</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"># file: data/data-cluster/dms/final_archive</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.dirty=0x000000000000000000000000</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.storage2-client-0=0x000000000000000000000001</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0x16c6a1e2b3fe4851972b998980097a87</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht.mds=0x00000000</span></div><div><span style="font-variant-ligatures:no-common-ligatures">—————————————————————————————————————</span></div></span></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">stat /data/data-cluster/dms/final_archive</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> File: '/data/data-cluster/dms/final_archive'</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Size: 3497984 <span class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956Apple-tab-span" style="white-space:pre-wrap"> </span>Blocks: 8760 IO Block: 4096 directory</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Device: 807h/2055d<span class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956Apple-tab-span" style="white-space:pre-wrap"> </span>Inode: 13563551265 Links: 72124</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: (0755/drwxr-xr-x) Uid: ( 33/www-data) Gid: ( 33/www-data)</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Access: 2018-10-09 04:22:40.514629044 +0200</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Modify: 2019-03-21 11:55:46.382565124 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Change: 2019-03-21 11:55:46.382565124 +0100</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> Birth: -</span></div></div><div><span style="font-variant-ligatures:no-common-ligatures">—————————————————————————————————————</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>Hope this helps.</div><div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956Apple-interchange-newline">- Kindest regards,<br><br>Milos Cuculovic<br>IT Manager<br><br>---<br>MDPI AG<br>Postfach, CH-4020 Basel, Switzerland<br>Office: St. Alban-Anlage 66, 4052 Basel, Switzerland<br>Tel. +41 61 683 77 35<br>Fax +41 61 302 89 18<br><a href="mailto:cuculovic@mdpi.com" target="_blank">Email: cuculovic@mdpi.com</a><br>Skype: milos.cuculovic.mdpi<br></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Disclaimer: The information and files contained in this message are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error, please notify me and delete this message from your system. You may not copy this message in its entirety or in part, or disclose its contents to anyone.</div>
</div>
<div><br><blockquote type="cite"><div>On 21 Mar 2019, at 11:43, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:</div><br class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956Apple-interchange-newline"><div><div dir="ltr">Can you attach the "glustershd.log" file which will be present under "/var/log/glusterfs/" from both the nodes and the "stat" & "getfattr -d -m . -e hex <file-path-on-brick>" output of all the entries listed in the heal info output from both the bricks?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 21, 2019 at 3:54 PM Milos Cuculovic <<a href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Thanks Karthik!<div><br></div><div>I was trying to find some resolution methods from [2] but unfortunately none worked (I can explain what I tried if needed).</div><div><br></div><div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I guess the volume you are talking about is of type replica-2 (1x2).</div></div></div></div></div></blockquote>That’s correct, aware of the arbiter solution but still didn’t took time to implement.</div><div><br></div><div>From the info results I posted, how to know in which situation I am. No files are mentioned in spit brain, only directories. One brick has 3 entries and one two entries.<div><br class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956gmail-m_1351967363676197946webkit-block-placeholder"></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo gluster volume heal storage2 info</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">[sudo] password for sshadmin: </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage3:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:256ca960-1601-4f0d-9b08-905c6fd52326> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Possibly undergoing heal</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage4:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Possibly undergoing heal</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 2</span></div></div><div><br></div><div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">- Kindest regards,<br><br>Milos Cuculovic<br>IT Manager<br><br>---<br>MDPI AG<br>Postfach, CH-4020 Basel, Switzerland<br>Office: St. Alban-Anlage 66, 4052 Basel, Switzerland<br>Tel. +41 61 683 77 35<br>Fax +41 61 302 89 18<br><a href="mailto:cuculovic@mdpi.com" target="_blank">Email: cuculovic@mdpi.com</a><br>Skype: milos.cuculovic.mdpi<br></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Disclaimer: The information and files contained in this message are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error, please notify me and delete this message from your system. You may not copy this message in its entirety or in part, or disclose its contents to anyone.</div>
</div>
<div><br><blockquote type="cite"><div>On 21 Mar 2019, at 10:27, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:</div><br class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956gmail-m_1351967363676197946Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>Note: I guess the volume you are talking about is of type replica-2 (1x2). Usually replica 2 volumes are prone to split-brain. If you can consider converting them to arbiter or replica-3, they will handle most of the cases which can lead to slit-brains. For more information see [1].<br></div><div><br></div><div>Resolving the split-brain: [2] talks about how to interpret the heal info output and different ways to resolve them using the CLI/manually/using the favorite-child-policy.</div><div>If you are having entry split brain, and is a gfid split-brain (file/dir having different gfids on the replica bricks) then you can use the CLI option to resolve them. If a directory is in gfid split-brain in a distributed-replicate volume and you are using the source-brick option please make sure you use the brick of this subvolume, which has the same gfid as that of the other distribute subvolume(s) where you have the correct gfid, as the source.</div><div>If you are having a type mismatch then follow the steps in [3] to resolve the split-brain.</div><div><br></div><div>[1] <a href="https://docs.gluster.org/en/v3/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/" target="_blank">https://docs.gluster.org/en/v3/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/</a></div><div>[2] <a href="https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/" target="_blank">https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/</a></div><div>[3] <a href="https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/#dir-split-brain" target="_blank">https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/#dir-split-brain</a></div><div><br></div><div>HTH,</div><div>Karthik</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 21, 2019 at 1:45 PM Milos Cuculovic <<a href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>I was now able to catch the split brain log:<div><br></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo gluster volume heal storage2 info</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage3:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:256ca960-1601-4f0d-9b08-905c6fd52326> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Is in split-brain</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage4:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Is in split-brain</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 2</span></div><div><br></div><div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Milos</div></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div><blockquote type="cite"><div>On 21 Mar 2019, at 09:07, Milos Cuculovic <<a href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>> wrote:</div><br class="gmail-m_6575129410122648059gmail-m_-426258795379207644gmail-m_8539117702775147956gmail-m_1351967363676197946gmail-m_-1511781683842647198Apple-interchange-newline"><div><div>Since 24h, after upgrading from 4.0 to 4.1.7 one of the servers, the heal shows this:<br><br>sudo gluster volume heal storage2 info<br>Brick storage3:/data/data-cluster<br><gfid:256ca960-1601-4f0d-9b08-905c6fd52326> <br><gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> <br>/dms/final_archive - Possibly undergoing heal<br><br>Status: Connected<br>Number of entries: 3<br><br>Brick storage4:/data/data-cluster<br><gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> <br>/dms/final_archive - Possibly undergoing heal<br><br>Status: Connected<br>Number of entries: 2<br><br>The same files stay there. From time to time the status of the /dms/final_archive is in split brain at the following command shows:<br><br>sudo gluster volume heal storage2 info split-brain<br>Brick storage3:/data/data-cluster<br>/dms/final_archive<br>Status: Connected<br>Number of entries in split-brain: 1<br><br>Brick storage4:/data/data-cluster<br>/dms/final_archive<br>Status: Connected<br>Number of entries in split-brain: 1<br><br>How to know the file who is in split brain? The files in /dms/final_archive are not very important, fine to remove (ideally resolve the split brain) for the ones that differ.<br><br>I can only see the directory and GFID. Any idea on how to resolve this situation as I would like to continue with the upgrade on the 2nd server, and for this the heal needs to be done with 0 entries in sudo gluster volume heal storage2 info<br><br>Thank you in advance, Milos.</div></div></blockquote></div><br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div>