<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>No, it is not necessary that the first brick would be local one.<br></div><div><br></div><div>I really don't think starting from local node will make a difference.<br></div><div>The major time spent is not in getting list of entries from .gluster/indices/xattrop folder.</div><div>The LOCK->XATTR_CHECK->UNLOCK is the cycle which takes most of the time which is not going to change even if it is from local brick.<br></div><div><br></div><div>---<br></div><div>Ashish<br></div><div><br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Strahil" <hunter86_bg@yahoo.com><br><b>To: </b>"Ashish" <aspandey@redhat.com>, "Gluster" <gluster-devel@gluster.org>, "Gluster" <gluster-users@gluster.org><br><b>Sent: </b>Wednesday, March 6, 2019 10:21:26 PM<br><b>Subject: </b>Re: [Gluster-users] Gluster : Improvements on "heal info" command<br><div><br></div><p dir="ltr">Hi ,</p><p dir="ltr">This sounds nice. I would like to ask if the order is starting from the local node's bricks first ? (I am talking about --brick=one)</p><p dir="ltr">Best Regards,<br> Strahil Nikolov</p><div class="quote">On Mar 5, 2019 10:51, Ashish Pandey <aspandey@redhat.com> wrote:<br><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div><div style="font-family:'times new roman' , 'new york' , 'times' , serif;font-size:12pt;color:#000000" data-mce-style="font-family: 'times new roman' , 'new york' , 'times' , serif; font-size: 12pt; color: #000000;"><div>Hi All,<br></div><div><br></div><div>We have observed and heard from gluster users about the long time "heal info" command takes.<br></div><div>Even when we all want to know if a gluster volume is healthy or not, it takes time to list down all the files from all the bricks after which we can be <br></div><div>sure if the volume is healthy or not.<br></div><div>Here, we have come up with some options for "heal info" command which provide report quickly and reliably.<br></div><div><br></div><div><strong>gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all]</strong></div><div>--------<br></div><div><br></div><div>Problem: "gluster v heal <volname> info" command picks each subvolume and checks the .glusterfs/indices/xattrop folder of every brick of that subvolume to find out if there is any entry<br>which needs to be healed. It picks the entry and takes a lock on that entry to check xattrs to find out if that entry actually needs heal or not.<br></div><div>This <strong>LOCK->CHECK-XATTR->UNLOCK</strong> cycle takes lot of time for each file.<br></div><div><br></div><div>Let's consider two most often seen cases for which we use "heal info" and try to understand the improvements.<br></div><div><br><strong>Case -1 :</strong> Consider 4+2 EC volume and all the bricks on 6 different nodes.</div><div>A brick of the volume is down and client has written 10000 files on one of the mount point of this volume. Entries for these 10K files will be created on ".glusterfs/indices/xattrop"</div><div>on all the rest of 5 bricks. Now, brick is UP and when we use "heal info" command for this volume, it goes to all the bricks and picks these 10K file entries and<br>goes through <strong>LOCK->CHECK-XATTR->UNLOCK</strong> cycle for all the files. This happens for all the bricks, that means, we check 50K files and perform the LOCK->CHECK-XATTR->UNLOCK cycle 50K times,</div><div>while only 10K entries were sufficient to check. It is a very time consuming operation. If IO"s are happening one some of the new files, we check these files also which will add the time.<br></div><div>Here, all we wanted to know if our volume has been healed and healthy.<br></div><div><br></div><div><strong>Solution :</strong> Whenever a brick goes down and comes up and when we use "heal info" command, our *main intention* is to find out if the volume is *healthy* or *unhealthy*. A volume is unhealthy even if one<br>file is not healthy. So, we should scan bricks one by one and as soon as we find that one brick is having some entries which require to be healed, we can come out and list the files and say the volume is not<br>healthy. No need to scan rest of the bricks. That's where "--brick=[one,all]" option has been introduced.<br></div><div><br></div><div>"gluster v heal vol info --brick=[one,all]"<br>"one" - It will scan the brick sequentially and as soon as it will find any unhealthy entries, it will list it out and stop scanning other bricks.<br>"all" - It will act just like current behavior and provide all the files from all the bricks. If we do not provide this option, default (current) behavior will be applicable.<br><div><br></div><strong>Case -2 :</strong> Consider 24 X (4+2) EC volume. Let's say one brick from *only one* of the sub volume has been replaced and a heal has been triggered.</div><div>To know if the volume is in healthy state, we go to each brick of *each and every sub volume* and check if there are any entries in ".glusterfs/indices/xattrop" folder which need heal or not.</div><div>If we know which sub volume participated in brick replacement, we just need to check health of that sub volume and not query/check other sub volumes.<br></div><div><br></div><div>If several clients are writing number of files on this volume, an entry for each of these files will be created in .glusterfs/indices/xattrop and "heal info'<br></div><div>command will go through LOCK->CHECK-XATTR->UNLOCK cycle to find out if these entries need heal or not which takes lot of time.</div><div>In addition to this a client will also see performance drop as it will have to release and take lock again.</div><div><br></div><div><br></div><div><strong>Solution:</strong> Provide an option to mention number of sub volume for which we want to check heal info.<br><div><br></div>"gluster v heal vol info --subvol=<no of subvolume> "<br>Here, --subvol will be given number of the subvolume we want to check.<br>Example:<br>"gluster v heal vol info --subvol=1 "<br><div><br></div><div><br></div><div>===================================<br></div><div>Performance Data - <br></div><div>A quick performance test done on standalone system.<br></div><div><br></div><div>Type: Distributed-Disperse<br>Volume ID: ea40eb13-d42c-431c-9c89-0153e834e67e<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 2 x (4 + 2) = 12<br>Transport-type: tcp<br>Bricks:<br>Brick1: apandey:/home/apandey/bricks/gluster/vol-1<br>Brick2: apandey:/home/apandey/bricks/gluster/vol-2<br>Brick3: apandey:/home/apandey/bricks/gluster/vol-3<br>Brick4: apandey:/home/apandey/bricks/gluster/vol-4<br>Brick5: apandey:/home/apandey/bricks/gluster/vol-5<br>Brick6: apandey:/home/apandey/bricks/gluster/vol-6<br>Brick7: apandey:/home/apandey/bricks/gluster/new-1<br>Brick8: apandey:/home/apandey/bricks/gluster/new-2<br>Brick9: apandey:/home/apandey/bricks/gluster/new-3<br>Brick10: apandey:/home/apandey/bricks/gluster/new-4<br>Brick11: apandey:/home/apandey/bricks/gluster/new-5<br>Brick12: apandey:/home/apandey/bricks/gluster/new-6<br><div><br></div></div><div>Just disabled the shd to get the data -<br></div><div><br></div><div>Killed one brick each from two subvolumes and wrote 2000 files on mount point.<br></div>[root@apandey vol]# for i in {1..2000};do echo abc >> file-$i; done<br><div><br></div>Start the volume using force option and get the heal info. Following is the data -<br><div><br></div></div><div>[root@apandey glusterfs]# time gluster v heal vol info --brick=one >> /dev/null <<<<<<<< This will scan brick one by one and come out as soon as we find volume is unhealthy.<br><div><br></div>real 0m8.316s<br>user 0m2.241s<br>sys 0m1.278s<br>[root@apandey glusterfs]# <br><div><br></div>[root@apandey glusterfs]# time gluster v heal vol info >> /dev/null <<<<<<<< This is current behavior.<br><div><br></div>real 0m26.097s<br>user 0m10.868s<br>sys 0m6.198s<br>[root@apandey glusterfs]# <br>===================================</div><div><br></div><div>I would like your comments/suggestions on this improvements.<br></div><div>Specially, would like to hear on the new syntax of the command -<br></div><div><br></div><div><strong>gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all]</strong></div><div><strong><br></strong></div><div>Note that if we do not provide new options, command will behave just like it does right now.<strong><br></strong></div><div>Also, this improvement is valid for AFR and EC.<br></div><div><strong><br></strong></div><div>---<br></div><div>Ashish<br></div><div><strong><br></strong></div><div><strong><br></strong></div><div><strong><br></strong></div><div><strong><br></strong></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div></blockquote></div><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>https://lists.gluster.org/mailman/listinfo/gluster-users</div><div><br></div></div></body></html>