<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 15, 2017 at 12:47 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>priv->pending_key[] array stores pending-xattr names. It is populated at the time of a<br>graph initialisation (see init() in afr.c and thereafter its call to afr_pending_xattrs_init()).<br></div><div><br></div><div>For example if I have a newly created 2x3 distributed-replicate volume called "dis-rep", my<br></div><div>priv->pending_key[] array would look like the following:<br><br></div><div>priv->pending_key[0] = "trusted.afr.dis-rep-client-0"<br>priv->pending_key[1] = "trusted.afr.dis-rep-client-1"<br>priv->pending_key[2] = "trusted.afr.dis-rep-client-2"<br><br>for the replicate-0 subvolume of dht.<br><br>And for replicate-1, it would be<br><br>priv->pending_key[0] = "trusted.afr.dis-rep-client-3"<br>priv->pending_key[1] = "trusted.afr.dis-rep-client-4"<br>priv->pending_key[2] = "trusted.afr.dis-rep-client-5"<br><br></div><div>HTH,<br>Krutika<br></div><div><br></div></div></blockquote><div><br></div><div> I think afr_pending_xattrs_init() was added in Ver 3.9 or later. I was using 3.7.15 . Will update and check.</div><div><br></div><div>Best regards</div><div>Jk</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 14, 2017 at 4:58 PM, jayakrishnan mm <span dir="ltr"><<a href="mailto:jayakrishnan.mm@gmail.com" target="_blank">jayakrishnan.mm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="auto"><div><div class="gmail-m_-2830605324105598968h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Feb 14, 2017 6:58 PM, "Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail-m_-2830605324105598968m_-595613241428006867quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div class="gmail-m_-2830605324105598968m_-595613241428006867quoted-text">On Tue, Feb 14, 2017 at 1:01 PM, jayakrishnan mm <span dir="ltr"><<a href="mailto:jayakrishnan.mm@gmail.com" target="_blank">jayakrishnan.mm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619HOEnZb"><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619h5"><div dir="ltr"><br><div class="gmail_quote"><br><br><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Feb 13, 2017 at 7:07 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Hi JK,<br></div><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490m_-1714078032564809454gmail-">On Mon, Feb 13, 2017 at 1:06 PM, jayakrishnan mm <span dir="ltr"><<a href="mailto:jayakrishnan.mm@gmail.com" target="_blank">jayakrishnan.mm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Krutika,<div>Could you pls. tell me what is the meaning of base file name ? I mean the part after xattrop-* what do this number signify ?</div><div><br></div><div>Ex: <span style="font-size:12.8px">xattrop-<b>a321b856-05b3-48d3<wbr>-</b></span><span style="font-size:12.8px"><b>a393-805be83c6b73 . </b>Is this gfid of some file ?</span></div></div></blockquote><div><br></div></span><div>No. <span style="font-size:12.8px"><b>a321b856-05b3-48d3-</b></span><span style="font-size:12.8px"><b>a393-805be8<wbr>3c6b73<span> </span></b>is a randomly UUID.<br>So in order to not consume one on-disk inode per index, index translator<br></span></div><div><span style="font-size:12.8px">creates this one base file called xattrop-xxxxxxxxx....xxx of size 0 bytes<br>where xxxxxxxx.....xxx is a randomly generated UUID and all indices that<br>need to be created to signify that certain gfids need heal will be hard-linked to<br>this base file. So the number of inodes consumed will remain 1, irrespective of<br></span></div><div><span style="font-size:12.8px">whether 10 files need heal or a 100 or 1000 or even a million.<br></span></div><div><span style="font-size:12.8px">You can do an `ls -li` under xattrop directory where a couple of files need heal<br></span></div><div><span style="font-size:12.8px">to see it for yourself.</span></div></div></div></div></blockquote><div><br></div></span><div>Understood. So all files will have gfid entries inside .glusterfs/xx/yy , where xx/yy is the initial part of gfid.</div><div>Only those need healing are kept under xattrop directory, and in order to save inodes, they are hard linked to a base file .</div><div>And self heal daemon can traverse this directory later for the purpose of healing.</div></div></div></div></div></div></div></div></blockquote><div><br></div></div><div>The hard links under .glusterfs/xx/yy are not to be confused with the indices under indices/xattrop. Even files that *need* heal will have<br></div><div>hard links to the original file containing user-written data that resides under its normal parent dir, under .glusterfs/xx/yy.<br></div><div>GFID being a unique property of every inode, the indices under xattrop are named after the gfid of individual files. <br></div><div class="gmail-m_-2830605324105598968m_-595613241428006867elided-text"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619HOEnZb"><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619h5"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>I have encountered some extra gfids </div><div><br></div><div><div><i>Volume Name: rep-vol</i></div><div><i>Type: Replicate</i></div><div><i>Volume ID: 8667044d-b75e-4fc0-9ae8-2d2b39<wbr>b8558f</i></div><div><i>Status: Started</i></div><div><i>Number of Bricks: 1 x 2 = 2</i></div><div><i>Transport-type: tcp</i></div><div><i>Bricks:</i></div><div><i>Brick1: 192.168.36.220:/home/user/glus<wbr>ter/rep-brick1</i></div><div><i>Brick2: 192.168.36.220:/home/user/glus<wbr>ter/rep-brick2</i></div><div><i>Options Reconfigured:</i></div><div><i>performance.readdir-ahead: on</i></div></div><div><i><br></i></div><div><b>I kill brick1 process, so that status is as below</b></div><div><i><br></i></div><div><div style="font-style:italic">Status of volume: rep-vol</div><div style="font-style:italic">Gluster process TCP Port RDMA Port Online Pid</div><div style="font-style:italic">------------------------------<wbr>------------------------------<wbr>------------------</div><div style="font-style:italic">Brick 192.168.36.220:/home/user/glus<wbr>ter/rep</div><div style="font-style:italic">-brick1 N/A N/A N N/A </div><div style="font-style:italic">Brick 192.168.36.220:/home/user/glus<wbr>ter/rep</div><div style="font-style:italic">-brick2 49221 0 Y 24925</div><div style="font-style:italic">NFS Server on localhost N/A N/A N N/A </div><div style="font-style:italic">Self-heal Daemon on localhost N/A N/A Y 24954</div><div style="font-style:italic"> </div><div style="font-style:italic">Task Status of Volume rep-vol</div><div style="font-style:italic">------------------------------<wbr>------------------------------<wbr>------------------</div><div style="font-style:italic">There are no active volume tasks</div><div style="font-style:italic"><br></div><div><b>Copy wish.txt to /mnt/gluster/rep (mount point)</b></div></div><div><br></div><div><div><i>root@dhcp-192-168-36-220:/home<wbr>/user/gluster/rep-brick2/.glus<wbr>terfs/indices/xattrop# ls -li</i></div><div><i>total 0</i></div><div><i>3670235 ---------- 3 root root 0 Feb 9 07:04 00000000-0000-0000-0000-000000<wbr>000001</i></div><div><i>3670235 ---------- 3 root root 0 Feb 9 07:04 a3e23755-4ec6-42d2-ac2c-ad4bd6<wbr>82cdbd</i></div><div><i>3670235 ---------- 3 root root 0 Feb 9 07:04 xattrop-8263faed-cba8-4738-919<wbr>7-93e4e7e103ff</i></div></div><div><i><br></i></div><div><b>As expected.</b></div><div><b><br></b></div><div><b>Now I create another file test.txt on the mount point.</b></div><div><b><br></b></div><div><div><i>root@dhcp-192-168-36-220:/home<wbr>/user/gluster/rep-brick2/.glus<wbr>terfs/indices/xattrop# ls -li</i></div><div><i>total 0</i></div><div><i>3670235 ---------- 7 root root 0 Feb 9 07:04 00000000-0000-0000-0000-000000<wbr>000001</i></div><div><i>3670235 ---------- 7 root root 0 Feb 9 07:04 <b>571ca3f1-5c1b-426d-990e-191aa6<wbr>2ea9c4</b></i></div><div><i>3670235 ---------- 7 root root 0 Feb 9 07:04 6b7f7823-b864-4f48-8a07-7c073f<wbr>8d2ef5</i></div><div><i>3670235 ---------- 7 root root 0 Feb 9 07:04 7b4e97fc-8734-4dba-a72a-a750a2<wbr>2abd2d</i></div><div><i>3670235 ---------- 7 root root 0 Feb 9 07:04 96587bb0-2ff8-4d97-8470-69bb48<wbr>be9fd2</i></div><div><i>3670235 ---------- 7 root root 0 Feb 9 07:04 <b>a3e23755-4ec6-42d2-ac2c-ad4bd6<wbr>82cdbd</b></i></div><div><i>3670235 ---------- 7 root root 0 Feb 9 07:04 xattrop-8263faed-cba8-4738-919<wbr>7-93e4e7e103ff</i></div><div><i><br></i></div><div><div style="font-style:italic">root@dhcp-192-168-36-220:/home<wbr>/user/gluster/rep-brick2/.glus<wbr>terfs/indices/xattrop# getfattr -d -e hex -m . ../../../wish.txt | grep gfid</div><div style="font-style:italic">trusted.gfid=<b>0xa3e237554ec642d<wbr>2ac2cad4bd682cdbd</b></div><div style="font-style:italic">root@dhcp-192-168-36-220:/home<wbr>/user/gluster/rep-brick2/.glus<wbr>terfs/indices/xattrop# getfattr -d -e hex -m . ../../../test.txt | grep gfid</div><div style="font-style:italic">trusted.gfid=<b>0x571ca3f15c1b426<wbr>d990e191aa62ea9c4</b></div><div style="font-style:italic"><br></div><div style="font-style:italic"><br></div><div><b>Why some extra gfids?</b></div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div></div><div>Did you try converting those extra gfids to path and figure out what entries those are?<br></div><div>If you haven't, could you please do that and get back?<font color="#888888"><br><br></font></div><font color="#888888"><div>-Krutika<br></div></font></div></div></div></blockquote></div></div></div></div></div><div dir="auto">I will check and let you know. Meanwhile, i would like to know what priv->pending_key refers to. I know priv->pending refers to failed nodes. </div><div dir="auto">Regards </div><div dir="auto">JK</div><div><div class="gmail-m_-2830605324105598968h5"><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail-m_-2830605324105598968m_-595613241428006867quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><font color="#888888"><div><br></div></font><div class="gmail-m_-2830605324105598968m_-595613241428006867elided-text"><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619HOEnZb"><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619h5"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div><i><br></i></div><div>Regards</div><div>JK</div><div style="font-weight:bold"><br></div></div><div><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span style="font-size:12.8px"><span class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490m_-1714078032564809454gmail-HOEnZb"><font color="#888888"><br></font></span></span></div><span class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490m_-1714078032564809454gmail-HOEnZb"><font color="#888888"><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">-Krutika<br><br></span> </div></font></span><div><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490m_-1714078032564809454gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Regards</span></div><div><span style="font-size:12.8px">JK</span></div><div><span style="font-size:12.8px"><br></span></div></div><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490m_-1714078032564809454gmail-m_8337965966752076631m_2595747837426484233gmail-HOEnZb"><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490m_-1714078032564809454gmail-m_8337965966752076631m_2595747837426484233gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 19, 2016 at 12:31 PM, jayakrishnan mm <span dir="ltr"><<a href="mailto:jayakrishnan.mm@gmail.com" target="_blank">jayakrishnan.mm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><p dir="ltr">OK. </p><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490m_-1714078032564809454gmail-m_8337965966752076631m_2595747837426484233gmail-m_-2901094072244191907HOEnZb"><div class="gmail-m_-2830605324105598968m_-595613241428006867m_-6442787832382969619m_686717336847313490m_-1714078032564809454gmail-m_8337965966752076631m_2595747837426484233gmail-m_-2901094072244191907h5">
<div class="gmail_quote">On Apr 19, 2016 11:25 AM, "Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The hardlink will be removed, yes, but the base file will stay.<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 19, 2016 at 8:31 AM, jayakrishnan mm <span dir="ltr"><<a href="mailto:jayakrishnan.mm@gmail.com" target="_blank">jayakrishnan.mm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Hi,<br></div>Is the hardlink not removed, after healing is done ?<br><br></div>--JK<br><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 18, 2016 at 6:22 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div>That's just a base file that all the gfid files are hard-linked to.<br></div>Since it is pointless to consume one inode for each gfid that needs a heal, we use a base file<br></div>with an identifiable name (xattrop-*) and then hard-link the actual gfid files representing pointers for heal<br></div>to this file.<br><br></div>-Krutika <br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Mon, Apr 18, 2016 at 11:00 AM, jayakrishnan mm <span dir="ltr"><<a href="mailto:jayakrishnan.mm@gmail.com" target="_blank">jayakrishnan.mm@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div><div dir="ltr"><div><div><div><div>Hi, <br>Self healing daemon refers to .glusterfs/indices/xattrop directory to see the files which are to be healed, and these dir should contain gfids of those files.<br><br></div>I see some other ids also which are prefixed with xattrop- , for example :<br><br>root@ad3:/data/ssd/dsi-ec8-bri<wbr>ck/.glusterfs/indices/xattrop# ll<br>total 8<br>drw------- 2 root root 4096 Apr 8 10:53 ./<br>drw------- 3 root root 4096 Apr 8 10:53 ../<br>---------- 1 root root 0 Apr 8 10:53 xattrop-a321b856-05b3-48d3-a39<wbr>3-805be83c6b73<br><br></div>What is the meaning ? <br><br><br></div>Best Regards<br></div>JK<br><div><div><div><br></div></div></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div></div></div><br></div></div>
</div><br></div>
</div></div></blockquote></div></div><br></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>