<div dir="ltr">Hi Ravi,<div>Thanks . I have created a simple 2 node replica volume.</div><div><br></div><div><div><i>root@dhcp-192-168-36-220:/home/user/gluster/rep-brick1# gluster v info rep-vol</i></div><div><i> </i></div><div><i>Volume Name: rep-vol</i></div><div><i>Type: Replicate</i></div><div><i>Volume ID: c9c9ef39-27e5-44d5-be69-82423c743304</i></div><div><i>Status: Started</i></div><div><i>Number of Bricks: 1 x 2 = 2</i></div><div><i>Transport-type: tcp</i></div><div><i>Bricks:</i></div><div><i>Brick1: 192.168.36.220:/home/user/gluster/rep-brick1</i></div><div><i>Brick2: 192.168.36.220:/home/user/gluster/rep-brick2</i></div><div><i>Options Reconfigured:</i></div><div><i>features.inode-quota: off</i></div><div><i>features.quota: off</i></div><div><i>performance.readdir-ahead: on</i></div><div><i><br></i></div><div> <b>Killed brick1 process.</b></div><div><br></div><div><div><i>root@dhcp-192-168-36-220:/home/user/gluster/rep-brick1# gluster v status rep-vol</i></div><div><i>Status of volume: rep-vol</i></div><div><i>Gluster process TCP Port RDMA Port Online Pid</i></div><div><i>------------------------------------------------------------------------------</i></div><div><i>Brick 192.168.36.220:/home/user/gluster/rep</i></div><div><i>-brick1 N/A N/A N N/A </i></div><div><i>Brick 192.168.36.220:/home/user/gluster/rep</i></div><div><i>-brick2 49211 0 Y 20157</i></div><div><i>NFS Server on localhost N/A N/A N N/A </i></div><div><i>Self-heal Daemon on localhost N/A N/A Y 20186</i></div><div><i> </i></div><div><i>Task Status of Volume rep-vol</i></div><div><i>------------------------------------------------------------------------------</i></div><div><i>There are no active volume task</i></div></div><div><br></div><div><b>And copying wish.txt to mount directory.</b></div><div><br></div><div><b>From brick2 ,</b></div><div><br></div><div><div><i>root@dhcp-192-168-36-220:/home/user/gluster/rep-brick2/.glusterfs# getfattr -d -e hex -m . ../wish.txt </i></div><div><i># file: ../wish.txt</i></div><div><i>trusted.afr.dirty=0x000000000000000000000000</i></div><div><i>trusted.afr.rep-vol-client-0=0x000000020000000100000000</i></div><div><i>trusted.bit-rot.version=0x0200000000000000589ab1410003e910</i></div><div><i>trusted.gfid=0xe9f3aafb3f844bca8922a00d48abc643</i></div></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra"><i>root@dhcp-192-168-36-220:/home/user/gluster/rep-brick2/.glusterfs/indices/xattrop# ll</i></div><div class="gmail_extra"><i>total 8</i></div><div class="gmail_extra"><i>drw------- 2 root root 4096 Feb 8 13:50 ./</i></div><div class="gmail_extra"><i>drw------- 4 root root 4096 Feb 8 13:48 ../</i></div><div class="gmail_extra"><i>---------- 4 root root 0 Feb 8 13:50 00000000-0000-0000-0000-000000000001</i></div><div class="gmail_extra"><i>---------- 4 root root 0 Feb 8 13:50 00000000-0000-0000-0000-000000000005</i></div><div class="gmail_extra"><i>---------- 4 root root 0 Feb 8 13:50 e9f3aafb-3f84-4bca-8922-a00d48abc643</i></div><div class="gmail_extra"><i>---------- 4 root root 0 Feb 8 13:50 xattrop-b3beb437-cea4-46eb-9eb4-8d83bfa7baa1</i></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div></div><div class="gmail_extra"> In the above, I can see the gfid of wish.txt (e9f3aafb-3f84-4bca-8922-a00d48abc643) , which need to be healed.</div><div class="gmail_extra">1. What are " <i>00000000-0000-0000-0000-000000000001</i>" and "<i>00000000-0000-0000-0000-000000000005</i> " ?</div><div class="gmail_extra">(I can understand <i>trusted.afr.rep-vol-client-0 as the changelog of brick1 as seen by brick2--- from <a href="https://github.com/gluster/glusterfs-specs/blob/master/done/Features/afr-v1.md">https://github.com/gluster/glusterfs-specs/blob/master/done/Features/afr-v1.md</a>)</i></div><div class="gmail_extra"><i><br></i></div><div class="gmail_extra">2. I know xattrop-* is a base file. How this is related to the files which require healing ? (Assuming more than one file to be healed).</div><div class="gmail_extra"> What does the numeric part on xattrop-* (<i> xattrop-<b>b3beb437-cea4-46eb-9eb4-8d83bfa7baa1</b>) </i>signify?</div><div class="gmail_extra"><br></div><div class="gmail_extra">3. After brick1 is brought to online, the file is healed. Now only xattrop-* remain under .glusterfs/indices/xattrop.</div><div class="gmail_extra"> But still there is gfid entry in .glusterfs/e9/f3 directory. Is this an expected behavior?</div><div class="gmail_extra"><br></div><div class="gmail_extra"><i><br></i></div><div class="gmail_extra"><br></div><div class="gmail_extra"><i><br></i></div><div class="gmail_extra"><i><br></i></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 7, 2017 at 8:21 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF"><div><div class="gmail-h5">
<div class="gmail-m_-2192529121827419570moz-cite-prefix">On 02/07/2017 01:32 PM, jayakrishnan mm
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Feb 6, 2017 at 6:05 PM,
Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF">
<div>
<div class="gmail-m_-2192529121827419570gmail-h5">
<div class="gmail-m_-2192529121827419570gmail-m_8081714311220739515moz-cite-prefix">On
02/06/2017 03:15 PM, jayakrishnan mm wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Feb 6, 2017
at 2:36 PM, jayakrishnan mm <span dir="ltr"><<a href="mailto:jayakrishnan.mm@gmail.com" target="_blank">jayakrishnan.mm@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div class="gmail-m_-2192529121827419570gmail-m_8081714311220739515gmail-h5">On
Fri, Feb 3, 2017 at 7:58 PM,
Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF"><span>
<div class="gmail-m_-2192529121827419570gmail-m_8081714311220739515gmail-m_6144383082695127397m_-9133827226248740748moz-cite-prefix">On
02/03/2017 09:14 AM,
jayakrishnan mm wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Feb 2, 2017
at 8:17 PM,
Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div bgcolor="#FFFFFF"><span class="gmail-m_-2192529121827419570gmail-m_8081714311220739515gmail-m_6144383082695127397m_-9133827226248740748gmail-">
<div class="gmail-m_-2192529121827419570gmail-m_8081714311220739515gmail-m_6144383082695127397m_-9133827226248740748gmail-m_-4204914587949192022moz-cite-prefix">On
02/02/2017
10:46 AM,
jayakrishnan
mm wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi</div>
<div><br>
</div>
<div>How do I
determine,
which part of
the code is
run on the
client, and
which part of
the code is
run on the
server
nodes by
merely looking
at the the
glusterfs
source code ?</div>
<div>I knew
there are
client side
and server
side
translators
which will run
on respective
platforms. I
am looking at
part of self
heal daemon
source
(ec/afr)
which will run
on the server
nodes and
the part which
run on the
clients.</div>
</div>
</blockquote>
<br>
</span> The
self-heal
daemon that
runs on the
server is also
a client
process in the
sense that it
has client
side xlators
like ec or afr
and
protocol/client
(see the shd
volfile
'glustershd-server.vol')
loaded and
talks to the
bricks like a
normal client
does.<br>
The difference
is that only
self-heal
related
'logic' get
executed on
the shd while
both self-heal
and I/O
related logic
get executed
from the
mount. The
self-heal
logic resides
mostly in
afr-self-heal*.[ch]
while I/O
related logic
is there in
the other
files.<br>
HTH,<br>
Ravi<br>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
</span> Hi JK,<span><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><span style="font-size:12.8px">Dear Ravi,</span>
<div style="font-size:12.8px">Thanks
for your kind
explanation.</div>
<div style="font-size:12.8px">So,
each server node
will have a
separate
self-heal
daemon(shd) up
and running ,
every time a
child_up event
occurs, and this
will be an
index healer. </div>
<div style="font-size:12.8px">And
each daemon
will spawn
"priv->child_count
" number of
threads on each
server node .
correct ?</div>
</div>
</div>
</div>
</blockquote>
</span> shd is always
running and yes those many
threads are spawned for
index heal when the process
starts.<span><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div style="font-size:12.8px">1.
When exactly a
full healer
spawns threads?
<br>
</div>
</div>
</div>
</div>
</blockquote>
</span> Whenever you run
`gluster volume heal volname
full`. See afr_xl_op().
There are some bugs in
launching full heal though.<span><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div style="font-size:12.8px">2.
When
can GF_EVENT_TRANSLATOR_OP
& GF_SHD_OP_HEAL_INDEX happen together (so that index healer spawns
thread) ?</div>
<div style="font-size:12.8px">
similarly when
can
GF_EVENT_TRANSLATOR_OP
& GF_SHD_OP_HEAL_FULL happen ? During replace-brick ?</div>
<div style="font-size:12.8px">Is
it possible that
index healer and
full healer
spawns threads
together (so
that total
number of
threads is
2*priv->child_count)?</div>
<div style="font-size:12.8px"><br>
</div>
</div>
</div>
</div>
</blockquote>
</span> index heal threads
wake up and run once every
10 minutes or whatever the
cluster.heal-timeout is.
They are also run when a
brick comes up like you
said, via afr_notify(). It
is also run when you
manually launch 'gluster
volume heal volname`. Again
see afr_xl_op().<span><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div style="font-size:12.8px">3.
In
/var/lib/glusterd/glustershd/g<wbr>lustershd-server.vol
, why
debug/io-stats
is chosen as
the top xlator ?</div>
<div style="font-size:12.8px"><br>
</div>
</div>
</div>
</div>
</blockquote>
</span> io-stats is
generally loaded as the top
most xlator in all graphs at
the appropriate place for
gathering profile-info, but
for shd, I'm not sure if it
has any specific use other
than acting as a placeholder
as a parent to all replica
xlators.<br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Hi Ravi,</div>
<div><br>
</div>
<div>The self heal daemon searches in
.glusterfs/indices/xattrop directory for
the files/dirs to be healed . Who is
updating this information , and on what
basis ? </div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
Please see <a class="gmail-m_-2192529121827419570gmail-m_8081714311220739515moz-txt-link-freetext" href="https://github.com/gluster/glusterfs-specs/blob/master/done/Features/afr-v1.md" target="_blank">https://github.com/gluster/glu<wbr>sterfs-specs/blob/master/done/<wbr>Features/afr-v1.md</a>,
it is a bit dated (relevant to AFR v1, which is in
glusterfs 3.5 and older I think) but the concepts are
similar. The entries are added/removed by the index
translator during the pre-op/post-op phases of the AFR
transaction .</div>
</blockquote>
<div><br>
</div>
<div>Hi Ravi,</div>
<div><br>
</div>
<div> Went thru' the document & source code. I see
there are options to enable/disable entry/data/metadata
change logs. If "data-change-log" is 1 (by default , it
is 1), this will enable data change log, which results
in __changelog_enabled() to return 1 and thereafter
call afr_changelog_pre_op() . Similar logic for post-op
also, which occurs just before unlock.</div>
<div>Is this responsible for creating/deleting entries
inside .glusterfs/indices/xattrop ?</div>
</div>
</div>
</div>
</blockquote></div></div>
Yes, index_xattrop() adds the entry during pre-op and removes it
during post-op if it was successful.<span class="gmail-"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> Currently I can't verify, since the mount point for
the rep volume hangs when data-change-log is set to 0.
(using glusterfs v 3.7.15). Ideally, the entries should
not appear (in the case of brick failure and a write
thereafter) if this option is set to '0', am I correct ?</div>
<div><br>
</div>
<div>Best Regards</div>
<div>JK</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="gmail-m_-2192529121827419570gmail-"><br>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Thanks Ravi, for the explanation.</div>
<div>Regards</div>
<div>JK </div>
<span>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div bgcolor="#FFFFFF"> <br>
Regards,<br>
Ravi<span><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div style="font-size:12.8px">Thanks</div>
<div><span style="font-size:12.8px">Best
regards</span> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Best
regards</div>
<div>JK</div>
</div>
<br>
<fieldset class="gmail-m_-2192529121827419570gmail-m_8081714311220739515gmail-m_6144383082695127397m_-9133827226248740748gmail-m_-4204914587949192022mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-devel mailing list
<a class="gmail-m_-2192529121827419570gmail-m_8081714311220739515gmail-m_6144383082695127397m_-9133827226248740748gmail-m_-4204914587949192022moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>
<a class="gmail-m_-2192529121827419570gmail-m_8081714311220739515gmail-m_6144383082695127397m_-9133827226248740748gmail-m_-4204914587949192022moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a></pre>
</blockquote>
<p>
</p>
</div>
</blockquote></div>
</div></div>
</blockquote><p>
</p></span></div></blockquote></span></div>
</div></div>
</blockquote></div>
</div></div>
</blockquote><p>
</p></span></div></blockquote></div>
</div></div>
</blockquote><p>
</p></span></div></blockquote></div><br></div></div></div>