<div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">Thanks. I might have tried to stop the volume before rebooting the glusterd. Also, it's fine now, I think volume is already started.</div><div dir="auto"><br></div><div dir="auto">Regards,</div><div dir="auto">Jeevan.</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 26, 2018, 6:14 PM Sanju Rakonde <<a href="mailto:srakonde@redhat.com">srakonde@redhat.com</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Jeevan,<div><br></div><div>You might be hitting <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1635820" target="_blank" rel="noreferrer">https://bugzilla.redhat.com/show_bug.cgi?id=1635820</a></div><div><br></div><div>Were any of the volumes in "Created" state, when the peer reject issue is seen?</div><div><br></div><div>Thanks,</div><div>Sanju</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 26, 2018 at 9:35 AM Jeevan Patnaik <<a href="mailto:g1patnaik@gmail.com" target="_blank" rel="noreferrer">g1patnaik@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Hi Atin,<div dir="auto"><br></div><div dir="auto">Thanks for the details. I think the issue is with few of the nodes which aren't serving any bricks in rejected state. When I remove them from pool and stop glusterfs in those nodes, everything seems normal. </div><div dir="auto"><br></div><div dir="auto">We keep those nodes as spares, but have glusterd runnin. coz in our configuration, servers are also clients and we are using gluster NFS without failover for mounts and to localize the impact if a node goes down, we use localhost as the nfs server on each node. </div><div dir="auto">I.e.,</div><div dir="auto">mount -t nfs localhost:/volume /mointpoint</div><div dir="auto"><br></div><div dir="auto">So, glusterfs should be running in these spare nodes. Now is this okay to keep those nodes in the pool? Will they go to rejected state again and cause transaction locks. Why aren't they in sync though they're part of the pool.</div><div dir="auto"><br></div><div dir="auto">Regards,</div><div dir="auto">Jeevan.</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 26, 2018, 8:22 AM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank" rel="noreferrer">amukherj@redhat.com</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 26, 2018 at 8:21 AM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" rel="noreferrer noreferrer" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Sun, Nov 25, 2018 at 8:40 PM Jeevan Patnaik <<a href="mailto:g1patnaik@gmail.com" rel="noreferrer noreferrer" target="_blank">g1patnaik@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">I am getting output Another transaction is in progress with few gluster volume commands including stop command. And with gluster volume status command, it's just hung and fails with timeout error.<br></div></div></blockquote><div><br></div><div>This is primarily because of not allowing glusterd to complete it's handshake with others when concurrent restart of glusterd services are done (as I could understand from your previous email in the list). With GlusterD (read as GD1) this is a current challenge w.r.t it's design where due to its N X N handshaking mechanism at the restart sequence to bring all the configuration data into inconsistent what we've seen is the overall recovery time of the cluster can take very long if N is on the higher side (in your case N = 72 which is certainly high) and hence the recommendation is not to restart the glusterd services concurrently and wait for the handshaking to complete.</div></div></div></blockquote><div><br></div><div>Forgot to mention that GlusterD2 ( <a href="https://github.com/gluster/glusterd2" rel="noreferrer noreferrer" target="_blank">https://github.com/gluster/glusterd2</a>) which is in development phase addresses this design problem.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto"></div><div dir="auto"><br></div><div dir="auto">So, I want to find out which transaction is hung and how to know this? I ran volume statedump command, but didn't wait till it's completed to check if it's hung or giving any resut, as it is also taking time.</div></div></blockquote><div><br></div><div>kill -SIGUSR1 $(pidof glusterd) should get you a glusterd statedump file in /var/run/gluster which can point to a backtrace dump at the bottom to understand which transaction is currently in progress. In case this transaction is queued up for more than 180 seconds (which is not usual) the unlock timer kicks out such locks.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto"><br></div><div dir="auto">Please help me with this.. I'm struggling with these gluster timeout errors :(</div><div dir="auto"><br></div><div dir="auto">Besides, I have also tuned </div><div dir="auto">transport.listen-backlog gluster to 200 and following kernel parameters to avoid syn overflow rejects:</div><div dir="auto">net.core.somaxconn = 1024</div><div dir="auto">net.ipv4.tcp_max_syn_backlog = 20480</div><div dir="auto"><br></div><div dir="auto">Regards,</div><div dir="auto">Jeevan.</div><div dir="auto"><br></div><div dir="auto"><br></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" rel="noreferrer noreferrer" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div>
</blockquote></div></div></div>
</blockquote></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank" rel="noreferrer">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-2720634175323315067m_439111513972335459gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks,<br></div>Sanju<br></div></div>
</blockquote></div>