<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 29, 2017 at 8:12 PM, Paolo Margara <span dir="ltr"><<a href="mailto:paolo.margara@polito.it" target="_blank">paolo.margara@polito.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class="">
<p>Il 29/06/2017 16:27, Pranith Kumar Karampuri ha scritto:<br>
</p>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jun 29, 2017 at 7:48 PM,
Paolo Margara <span dir="ltr"><<a href="mailto:paolo.margara@polito.it" target="_blank">paolo.margara@polito.it</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Pranith,</p>
<p>I'm using this guide
<a class="m_6127762250489253015gmail-m_-465708280187193584moz-txt-link-freetext" href="https://github.com/nixpanic/glusterdocs/blob/f6d48dc17f2cb6ee4680e372520ec3358641b2bc/Upgrade-Guide/upgrade_to_3.8.md" target="_blank">https://github.com/nixpanic/gl<wbr>usterdocs/blob/f6d48dc17f2cb6e<wbr>e4680e372520ec3358641b2bc/<wbr>Upgrade-Guide/upgrade_to_3.8.<wbr>md</a></p>
<p>Definitely my fault, but I think that is better to
specify somewhere that restarting the service is not
enough simply because in many other case, with other
services, is sufficient.</p>
</div>
</blockquote>
<div>The steps include the following command before
installing 3.8 as per the page (<a href="https://github.com/nixpanic/glusterdocs/blob/f6d48dc17f2cb6ee4680e372520ec3358641b2bc/Upgrade-Guide/upgrade_to_3.8.md#online-upgrade-procedure-for-servers" target="_blank">https://github.com/nixpanic/<wbr>glusterdocs/blob/<wbr>f6d48dc17f2cb6ee4680e372520ec3<wbr>358641b2bc/Upgrade-Guide/<wbr>upgrade_to_3.8.md#online-<wbr>upgrade-procedure-for-servers</a>)<br>
</div>
<div>So I guess we have it covered?<br>
</div>
</div>
</div>
</div>
</blockquote></span>
As I said it's my fault ;-)</div></blockquote><div><br></div><div>Ah! sorry. Thanks for your mail!<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><div><div class="h5"><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<ul>
<li>Stop all gluster services using the below command or
through your favorite way to stop them.<span class="m_6127762250489253015gmail-pl-c"></span></li>
<li><span class="m_6127762250489253015gmail-pl-c">killall glusterfs
glusterfsd glusterd</span><br>
</li>
</ul>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Now I'm restarting every brick process (and waiting
for the heal to complete), this is fixing my problem.</p>
<p>Many thanks for the help.<br>
</p>
<p><br>
</p>
<p>Greetings,</p>
<p> Paolo<br>
</p>
<div>
<div class="m_6127762250489253015gmail-h5"> <br>
<div class="m_6127762250489253015gmail-m_-465708280187193584moz-cite-prefix">Il
29/06/2017 13:03, Pranith Kumar Karampuri ha
scritto:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="m_6127762250489253015gmail-h5">
<div dir="ltr">
<div>Paolo,<br>
</div>
Which document did you follow for the
upgrade? We can fix the documentation if there
are any issues.<br>
</div>
</div>
</div>
<div class="gmail_extra">
<div>
<div class="m_6127762250489253015gmail-h5"><br>
<div class="gmail_quote">On Thu, Jun 29, 2017 at
2:07 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span>
<div class="m_6127762250489253015gmail-m_-465708280187193584m_3264995662940995335moz-cite-prefix">On
06/29/2017 01:08 PM, Paolo Margara
wrote:<br>
</div>
<blockquote type="cite">
<p>Hi all,</p>
<p>for the upgrade I followed this
procedure:</p>
<ul>
<li>put node in maintenance mode
(ensure no client are active)</li>
<li>yum versionlock delete
glusterfs*<br>
</li>
<li>service glusterd stop</li>
<li>yum update</li>
<li>systemctl daemon-reload <br>
</li>
<li>service glusterd start</li>
<li>yum versionlock add glusterfs*</li>
<li>gluster volume heal
vm-images-repo full</li>
<li>gluster volume heal
vm-images-repo info</li>
</ul>
<p>on each server every time I ran
'gluster --version' to confirm the
upgrade, at the end I ran 'gluster
volume set all cluster.op-version
30800'.</p>
<p>Today I've tried to manually kill a
brick process on a non critical
volume, after that into the log I
see:</p>
<p>[2017-06-29 07:03:50.074388] I
[MSGID: 100030]
[glusterfsd.c:2454:main]
0-/usr/sbin/glusterfsd: Started
running /usr/sbin/glusterfsd version
3.8.12 (args: /usr/sbin/glusterfsd
-s virtnode-0-1-gluster --volfile-id
iso-images-repo.virtnode-0-1-g<wbr>luster.data-glusterfs-brick1b-<wbr>iso-images-repo
-p /var/lib/glusterd/vols/iso-ima<wbr>ges-repo/run/virtnode-0-1-glus<wbr>ter-data-glusterfs-brick1b-iso<wbr>-images-repo.pid
-S /var/run/gluster/c779852c21e2a<wbr>91eaabbdda3b9127262.socket
--brick-name
/data/glusterfs/brick1b/iso-im<wbr>ages-repo
-l /var/log/glusterfs/bricks/data<wbr>-glusterfs-brick1b-iso-images-<wbr>repo.log
--xlator-option
*-posix.glusterd-uuid=e93ebee7<wbr>-5d95-4100-a9df-4a3e60134b73
--brick-port 49163 --xlator-option
iso-images-repo-server.listen-<wbr>port=49163)</p>
<p>I've checked after the restart and
indeed now the directory
'entry-changes' is created, but why
stopping the glusterd service has
not stopped also the brick
processes?</p>
</blockquote>
<br>
</span> Just stopping,upgrading and
restarting glusterd does not restart the
brick processes, You would need to kill
all gluster processes on the node before
upgrading. After upgrading, when you
restart glusterd, it will automatically
spawn the rest of the gluster processes on
that node.<span><br>
<br>
<blockquote type="cite">
<p>Now how can I recover from this
issue? Restarting all brick
processes is enough?</p>
</blockquote>
</span> Yes, but ensure there are no
pending heals like Pranith mentioned. <a class="m_6127762250489253015gmail-m_-465708280187193584m_3264995662940995335moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.7/" target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/Upgrade-Guide/upgra<wbr>de_to_3.7/</a>
lists the steps for upgrade to 3.7 but the
steps mentioned there are similar for any
rolling upgrade.<br>
<br>
-Ravi
<div>
<div class="m_6127762250489253015gmail-m_-465708280187193584h5"><br>
<blockquote type="cite">
<p><br>
</p>
<p>Greetings,</p>
<p> Paolo Margara<br>
</p>
<br>
<div class="m_6127762250489253015gmail-m_-465708280187193584m_3264995662940995335moz-cite-prefix">Il
28/06/2017 18:41, Pranith Kumar
Karampuri ha scritto:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jun 28, 2017 at 9:45
PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="m_6127762250489253015gmail-m_-465708280187193584m_3264995662940995335HOEnZb">
<div class="m_6127762250489253015gmail-m_-465708280187193584m_3264995662940995335h5">On
06/28/2017 06:52 PM,
Paolo Margara wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi list,<br>
<br>
yesterday I noted
the following lines
into the
glustershd.log log
file:<br>
<br>
[2017-06-28
11:53:05.000890] W
[MSGID: 108034]<br>
[afr-self-heald.c:479:afr_shd_<wbr>index_sweep]<br>
0-iso-images-repo-replicate-0: unable to get index-dir on<br>
iso-images-repo-client-0<br>
[2017-06-28
11:53:05.001146] W
[MSGID: 108034]<br>
[afr-self-heald.c:479:afr_shd_<wbr>index_sweep]
0-vm-images-repo-replicate-0:<br>
unable to get
index-dir on
vm-images-repo-client-0<br>
[2017-06-28
11:53:06.001141] W
[MSGID: 108034]<br>
[afr-self-heald.c:479:afr_shd_<wbr>index_sweep]
0-hosted-engine-replicate-0:<br>
unable to get
index-dir on
hosted-engine-client-0<br>
[2017-06-28
11:53:08.001094] W
[MSGID: 108034]<br>
[afr-self-heald.c:479:afr_shd_<wbr>index_sweep]
0-vm-images-repo-replicate-2:<br>
unable to get
index-dir on
vm-images-repo-client-6<br>
[2017-06-28
11:53:08.001170] W
[MSGID: 108034]<br>
[afr-self-heald.c:479:afr_shd_<wbr>index_sweep]
0-vm-images-repo-replicate-1:<br>
unable to get
index-dir on
vm-images-repo-client-3<br>
<br>
Digging into the
mailing list archive
I've found another
user with a<br>
similar issue (the
thread was
'[Gluster-users]
glustershd: unable
to get<br>
index-dir on
myvolume-client-0'),
the solution
suggested was to
verify<br>
if the
/<path-to-backend-brick>/.glus<wbr>terfs/indices
directory contains<br>
all these sub
directories:
'dirty',
'entry-changes' and
'xattrop' and if<br>
some of them does
not exists simply
create it with
mkdir.<br>
<br>
In my case the
'entry-changes'
directory is not
present on all the<br>
bricks and on all
the servers:<br>
<br>
/data/glusterfs/brick1a/hosted<wbr>-engine/.glusterfs/indices/:<br>
total 0<br>
drw------- 2 root
root 55 Jun 28 15:02
dirty<br>
drw------- 2 root
root 57 Jun 28 15:02
xattrop<br>
<br>
/data/glusterfs/brick1b/iso-im<wbr>ages-repo/.glusterfs/indices/:<br>
total 0<br>
drw------- 2 root
root 55 May 29 14:04
dirty<br>
drw------- 2 root
root 57 May 29 14:04
xattrop<br>
<br>
/data/glusterfs/brick2/vm-imag<wbr>es-repo/.glusterfs/indices/:<br>
total 0<br>
drw------- 2 root
root 112 Jun 28
15:02 dirty<br>
drw------- 2 root
root 66 Jun 28
15:02 xattrop<br>
<br>
/data/glusterfs/brick3/vm-imag<wbr>es-repo/.glusterfs/indices/:<br>
total 0<br>
drw------- 2 root
root 64 Jun 28 15:02
dirty<br>
drw------- 2 root
root 66 Jun 28 15:02
xattrop<br>
<br>
/data/glusterfs/brick4/vm-imag<wbr>es-repo/.glusterfs/indices/:<br>
total 0<br>
drw------- 2 root
root 112 Jun 28
15:02 dirty<br>
drw------- 2 root
root 66 Jun 28
15:02 xattrop<br>
<br>
I've recently
upgraded gluster
from 3.7.16 to
3.8.12 with the
rolling<br>
upgrade procedure
and I haven't noted
this issue prior of
the update, on<br>
another system
upgraded with the
same procedure I
haven't encountered<br>
this problem.<br>
<br>
Currently all VM
images appear to be
OK but prior to
create the<br>
'entry-changes' I
would like to ask if
this is still the
correct<br>
procedure to fix
this issue<br>
</blockquote>
<br>
</div>
</div>
Did you restart the bricks
after the upgrade? That
should have created the
entry-changes directory.
Can you kill the brick and
restart it and see if the
dir is created? Double
check from the brick logs
that you're indeed running
3.12: "Started running
/usr/local/sbin/glusterfsd
version 3.8.12" should
appear when the brick
starts.<br>
</blockquote>
<div><br>
</div>
<div>Please note that if you
are going the route of
killing and restarting,
you need to do it in the
same way you did rolling
upgrade. You need to wait
for heal to complete
before you kill the other
nodes. But before you do
this, it is better you
look at the logs or
confirm the steps you used
for doing upgrade.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
-Ravi
<div class="m_6127762250489253015gmail-m_-465708280187193584m_3264995662940995335HOEnZb">
<div class="m_6127762250489253015gmail-m_-465708280187193584m_3264995662940995335h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
and if this
problem could have
affected the<br>
heal operations
occurred meanwhile.<br>
<br>
Thanks.<br>
<br>
<br>
Greetings,<br>
<br>
Paolo Margara<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing
list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="m_6127762250489253015gmail-m_-465708280187193584m_3264995662940995335gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
-- <br>
</div>
</div>
<div class="m_6127762250489253015gmail-m_-465708280187193584gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="m_6127762250489253015gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>