<p dir="ltr">Usually when this happens I run '/find /fuse/mount/point -exec stat {} \;' from a client (using gluster with oVirt).<br>
Yet, my scale is multiple times smaller and I don't know how this will affect you (except it will trigger a heal).</p>
<p dir="ltr">So the round-robin of the DNS clarifies the mystery .In such case, maybe FUSE client is not the problem.Still it is worth trying a VM with the new gluster version to mount the cluster.</p>
<p dir="ltr">From the profile (took a short glance over it from my phone), not all bricks are spending much of their time in LOOKUP.<br>
Maybe your data is not evenly distributed? Is that ever possible ?<br>
Sadly you can't rebalance untill all those heals are pending.(Maybe I'm wrong)</p>
<p dir="ltr">Have you checked the speed of 'ls /my/brick/subdir1/' on each brick ?</p>
<p dir="ltr">Sadly, I'm just a gluster user, so take everything with a grain of salt.</p>
<p dir="ltr">Best Regards,<br>
Strahil Nikolov</p>
<div class="quote">On Apr 21, 2019 18:03, Patrick Rennie <patrickmrennie@gmail.com> wrote:<br type='attribution'><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I just tried to check my "gluster volume heal gvAA01 statistics" and it doesn't seem like a full heal was still in progress, just an index, I have started the full heal again and am trying to monitor it with "gluster volume heal gvAA01 info" which just shows me thousands of gfid file identifiers scrolling past. <div>What is the best way to check the status of a heal and track the files healed and progress to completion? </div><div><br /></div><div>Thank you,</div><div>- Patrick</div></div><br /><div class="elided-text"><div dir="ltr">On Sun, Apr 21, 2019 at 10:28 PM Patrick Rennie <<a href="mailto:patrickmrennie@gmail.com">patrickmrennie@gmail.com</a>> wrote:<br /></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex"><div dir="ltr">I think just worked out why NFS lookups are sometimes slow and sometimes fast as the hostname uses round robin DNS lookups, if I change to a specific host, 01-B, it's always quick, and if I change to the other brick host, 02-B, it's always slow. <div>Maybe that will help to narrow this down? </div></div><br /><div class="elided-text"><div dir="ltr">On Sun, Apr 21, 2019 at 10:24 PM Patrick Rennie <<a href="mailto:patrickmrennie@gmail.com">patrickmrennie@gmail.com</a>> wrote:<br /></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Hi Strahil, </div><div><br /></div><div>Thank you for your reply and your suggestions. I'm not sure which logs would be most relevant to be checking to diagnose this issue, we have the brick logs, the cluster mount logs, the shd logs or something else? I have posted a few that I have seen repeated a few times already. I will continue to post anything further that I see. </div><div>I am working on migrating data to some new storage, so this will slowly free up space, although this is a production cluster and new data is being uploaded every day, sometimes faster than I can migrate it off. I have several other similar clusters and none of them have the same problem, one the others is actually at 98-99% right now (big problem, I know) but still performs perfectly fine compared to this cluster, I am not sure low space is the root cause here. </div><div><br /></div><div>I currently have 13 VMs accessing this cluster, I have checked each one and all of them use one of the two options below to mount the cluster in fstab</div><div><br /></div><div>HOSTNAME:/gvAA01 /mountpoint glusterfs defaults,_netdev,rw,log-level=WARNING,direct-io-mode=disable,use-readdirp=no 0 0</div><div>HOSTNAME:/gvAA01 /mountpoint glusterfs defaults,_netdev,rw,log-level=WARNING,direct-io-mode=disable</div><div><br /></div><div>I also have a few other VMs which use NFS to access the cluster, and these machines appear to be significantly quicker, initially I get a similar delay with NFS but if I cancel the first "ls" and try it again I get < 1 sec lookups, this can take over 10 minutes by FUSE/gluster client, but the same trick of cancelling and trying again doesn't work for FUSE/gluster. Sometimes the NFS queries have no delay at all, so this is a bit strange to me. </div><div>HOSTNAME:/gvAA01 /mountpoint/ nfs defaults,_netdev,vers=3,async,noatime 0 0</div><div><br /></div><div>Example:</div><div>user@VM:~$ time ls /cluster/folder</div><div>^C</div><div><br /></div><div>real 9m49.383s</div><div>user 0m0.001s</div><div>sys 0m0.010s</div><div><br /></div><div>user@VM:~$ time ls /cluster/folder</div><div><results></div><div><br /></div><div>real 0m0.069s</div><div>user 0m0.001s</div><div>sys 0m0.007s</div><div><br /></div><div>---</div><div><br /></div><div>I have checked the profiling as you suggested, I let it run for around a minute, then cancelled it and saved the profile info. </div><div><br /></div><div>root@HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 start</div><div>Starting volume profile on gvAA01 has been successful</div><div>root@HOSTNAME:/var/log/glusterfs# time ls /cluster/folder</div><div>^C</div><div><br /></div><div>real 1m1.660s</div><div>user 0m0.000s</div><div>sys 0m0.002s</div><div><br /></div><div>root@HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 info >> ~/profile.txt</div><div>root@HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 stop</div><div><br /></div><div>I will attach the results to this email as it's o</div></div></div></blockquote></div></blockquote></div></blockquote></div>