<div dir="ltr">Hi Jan,<div><br></div><div>It is not recommended that you automate the script for 'volume start force'. </div><div>Bricks do not go offline just like that. There will be some genuine issue </div><div>which triggers this. Could you please attach the entire glusterd.logs and </div><div>the brick logs around the time so that someone would be able to look?</div><div><br></div><div>Just to make sure, please check if you have any network outage(using iperf</div><div>or some standard tool).</div><div><br></div><div>@Hari, i think you forgot to provide the bug link, please provide so that Jan</div><div>or someone can check if it is related.</div><div><br></div><div><br></div><div class="gmail_extra"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>--</div><div>Thanks & Regards,</div><div>Manikandan Selvaganesan.</div><div>(@Manikandan Selvaganesh on Web)</div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Fri, Jun 30, 2017 at 3:19 PM, Jan <span dir="ltr"><<a href="mailto:jan.h.zak@gmail.com" target="_blank">jan.h.zak@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Hari,<div><br></div><div>thank you for your support!</div><div><br></div><div><div>Did I try to check offline bricks multiple times?</div><div>Yes – I gave it enough time (at least 20 minutes) to recover but it stayed offline.</div><div><br></div><div>Version?</div><div>All nodes are 100% equal – I tried fresh installation several times during my testing, Every time it is CentOS Minimal install with all updates and without any additional software:</div><div><br></div><div>uname -r</div><div>3.10.0-514.21.2.el7.x86_64</div><div><br></div><div>yum list installed | egrep 'gluster|ganesha'</div><div>centos-release-gluster310.<wbr>noarch 1.0-1.el7.centos @extras </div><div>glusterfs.x86_64 3.10.2-1.el7 @centos-gluster310</div><div>glusterfs-api.x86_64 3.10.2-1.el7 @centos-gluster310</div><div>glusterfs-cli.x86_64 3.10.2-1.el7 @centos-gluster310</div><div>glusterfs-client-xlators.x86_<wbr>64 3.10.2-1.el7 @centos-gluster310</div><div>glusterfs-fuse.x86_64 3.10.2-1.el7 @centos-gluster310</div><div>glusterfs-ganesha.x86_64 3.10.2-1.el7 @centos-gluster310</div><div>glusterfs-libs.x86_64 3.10.2-1.el7 @centos-gluster310</div><div>glusterfs-server.x86_64 3.10.2-1.el7 @centos-gluster310</div><div>libntirpc.x86_64 1.4.3-1.el7 @centos-gluster310</div><div>nfs-ganesha.x86_64 2.4.5-1.el7 @centos-gluster310</div><div>nfs-ganesha-gluster.x86_64 2.4.5-1.el7 @centos-gluster310</div><div>userspace-rcu.x86_64 0.7.16-3.el7 @centos-gluster310</div><div><br></div><div>Grepping for the brick process?</div><div>I’ve just tried it again. Process doesn’t exist when brick is offline.</div><div><br></div><div>Force start command?</div><div>sudo gluster volume start MyVolume force</div><div><br></div><div>That works! Thank you.</div><div><br></div><div>If I have this issue too often then I can create simple script that greps all bricks on the local server and force start when it’s offline. I can schedule such script once after for example 5 minutes after boot.</div><div><br></div><div>But I’m not sure if it’s good idea to automate it. I’d be worried that I can force it up even when the node doesn’t “see” other nodes and cause split brain issue.</div></div><div><br></div><div>Thank you!</div><div><br></div><div>Kind regards,</div><div>Jan</div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 30, 2017 at 8:01 AM, Hari Gowtham <span dir="ltr"><<a href="mailto:hgowtham@redhat.com" target="_blank">hgowtham@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jan,<br>
<br>
comments inline.<br>
<span><br>
On Fri, Jun 30, 2017 at 1:31 AM, Jan <<a href="mailto:jan.h.zak@gmail.com" target="_blank">jan.h.zak@gmail.com</a>> wrote:<br>
> Hi all,<br>
><br>
> Gluster and Ganesha are amazing. Thank you for this great work!<br>
><br>
> I’m struggling with one issue and I think that you might be able to help me.<br>
><br>
> I spent some time by playing with Gluster and Ganesha and after I gain some<br>
> experience I decided that I should go into production but I’m still<br>
> struggling with one issue.<br>
><br>
> I have 3x node CentOS 7.3 with the most current Gluster and Ganesha from<br>
> centos-gluster310 repository (3.10.2-1.el7) with replicated bricks.<br>
><br>
> Servers have a lot of resources and they run in a subnet on a stable<br>
> network.<br>
><br>
> I didn’t have any issues when I tested a single brick. But now I’d like to<br>
> setup 17 replicated bricks and I realized that when I restart one of nodes<br>
> then the result looks like this:<br>
><br>
> sudo gluster volume status | grep ' N '<br>
><br>
> Brick glunode0:/st/brick3/dir N/A N/A N N/A<br>
> Brick glunode1:/st/brick2/dir N/A N/A N N/A<br>
><br>
<br>
</span>did you try it multiple times?<br>
<div><div class="m_-6066479447614399176h5"><br>
> Some bricks just don’t go online. Sometime it’s one brick, sometime tree and<br>
> it’s not same brick – it’s random issue.<br>
><br>
> I checked log on affected servers and this is an example:<br>
><br>
> sudo tail /var/log/glusterfs/bricks/st-b<wbr>rick3-0.log<br>
><br>
> [2017-06-29 17:59:48.651581] W [socket.c:593:__socket_rwv] 0-glusterfs:<br>
> readv on <a href="http://10.2.44.23:24007" rel="noreferrer" target="_blank">10.2.44.23:24007</a> failed (No data available)<br>
> [2017-06-29 17:59:48.651622] E [glusterfsd-mgmt.c:2114:mgmt_r<wbr>pc_notify]<br>
> 0-glusterfsd-mgmt: failed to connect with remote-host: glunode0 (No data<br>
> available)<br>
> [2017-06-29 17:59:48.651638] I [glusterfsd-mgmt.c:2133:mgmt_r<wbr>pc_notify]<br>
> 0-glusterfsd-mgmt: Exhausted all volfile servers<br>
> [2017-06-29 17:59:49.944103] W [glusterfsd.c:1332:cleanup_and<wbr>_exit]<br>
> (-->/lib64/libpthread.so.0(+0x<wbr>7dc5) [0x7f3158032dc5]<br>
> -->/usr/sbin/glusterfsd(gluste<wbr>rfs_sigwaiter+0xe5) [0x7f31596cbfd5]<br>
> -->/usr/sbin/glusterfsd(cleanu<wbr>p_and_exit+0x6b) [0x7f31596cbdfb] )<br>
> 0-:received signum (15), shutting down<br>
> [2017-06-29 17:59:50.397107] E [socket.c:3203:socket_connect] 0-glusterfs:<br>
> connection attempt on <a href="http://10.2.44.23:24007" rel="noreferrer" target="_blank">10.2.44.23:24007</a> failed, (Network is unreachable)<br>
> [2017-06-29 17:59:50.397138] I [socket.c:3507:socket_submit_r<wbr>equest]<br>
> 0-glusterfs: not connected (priv->connected = 0)<br>
> [2017-06-29 17:59:50.397162] W [rpc-clnt.c:1693:rpc_clnt_subm<wbr>it]<br>
> 0-glusterfs: failed to submit rpc-request (XID: 0x3 Program: Gluster<br>
> Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)<br>
><br>
> I think that important message is “Network is unreachable”.<br>
><br>
> Question<br>
> 1. Could you please tell me, is that normal when you have many bricks?<br>
> Networks is definitely stable and other servers use it without problem and<br>
> all servers run on a same pair of switches. My assumption is that in the<br>
> same time many bricks try to connect and that doesn’t work.<br>
<br>
</div></div>no. it shouldnt happen if there are multiple bricks.<br>
there was a bug related to this [1]<br>
to verify if that was the issue I need to know a few things.<br>
1) are all the node of the same version.<br>
2) did you check grepping for the brick process using the ps command?<br>
need to verify is the brick is still up and is not connected to glusterd alone.<br>
<span><br>
<br>
><br>
> 2. Is there an option to configure a brick to enable some kind of<br>
> autoreconnect or add some timeout?<br>
> gluster volume set brick123 option456 abc ??<br>
</span>If the brick process is not seen in the ps aux | grep glusterfsd<br>
The way to start a brick is to use the volume start force command.<br>
If brick is not started there is no point configuring it. and to start<br>
a brick we cant<br>
use the configure command.<br>
<span><br>
><br>
> 3. What it the recommend way to fix offline brick on the affected server? I<br>
> don’t want to use “gluster volume stop/start” since affected bricks are<br>
> online on other server and there is no reason to completely turn it off.<br>
</span>gluster volume start force will not bring down the bricks that are<br>
already up and<br>
running.<br>
<br>
><br>
> Thank you,<br>
> Jan<br>
><br>
> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<span class="m_-6066479447614399176HOEnZb"><font color="#888888"><br>
<br>
<br>
--<br>
Regards,<br>
Hari Gowtham.<br>
</font></span></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div></div>