<p dir="ltr">Start by adding gluster-devel and providing links to your logs.<br>
You can join the gluster community meeting and ask for help.</p>
<p dir="ltr">Best Regards,<br>
Strahil Nikolov</p>
<div class="quote">On Oct 28, 2019 20:07, Timothy Orme <torme@ancestry.com> wrote:<br type='attribution'><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
I had tried increasing the log level, but didn't find anything of note.</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
<br />
</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
However, after trying a number of different things over the weekend, it turned out that simply starting and stopping the volume seemed to have fixed this.</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
<br />
</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
It does then seem like a bug perhaps, or some confused state, given that it doesn't seem to be any issue with communication between nodes. I'm not really sure how to report it though, given that I don't have steps to reproduce, or much insight into what the
cause might be from logging.<br />
</div>
<div></div>
<hr style="display:inline-block;width:98%" />
<div dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Strahil <hunter86_bg@yahoo.com><br />
<b>Sent:</b> Sunday, October 27, 2019 10:19 AM<br />
<b>To:</b> Timothy Orme <torme@ancestry.com>; gluster-users <gluster-users@gluster.org><br />
<b>Subject:</b> [EXTERNAL] Re: Re: [Gluster-users] Remove Brick Rebalance Hangs With No Activity</font>
<div> </div>
</div>
<div>
<p dir="ltr">I guess you can increase loglevel ( check <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__access.redhat.com_documentation_en-2Dus_red-5Fhat-5Fgluster-5Fstorage_3_html_administration-5Fguide_configuring-5Fthe-5Flog-5Flevel&d=DwMGaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=d0SJB4ihnau-Oyws6GEzcipkV9DfxCuMbgdSRgXeuxM&m=9egPta2lJ02rLk5KJSl3qn2mEUT8wQMfwioLnHY6a-s&s=802zgGPyfRaCfnIIq-B8NSfcr7V2ie-0S4YLU2yPEpI&e=">
</a><a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level</a> )</p>
<p dir="ltr">Also, have you checked if new and old servers can communicate properly ?</p>
<p dir="ltr">Also consider a tcpdump (for a short time) on the problematic node can prove if communication is OK.</p>
<p dir="ltr">I would go with the logs first.</p>
<p dir="ltr">Best Regards,<br />
Strahil Nikolov</p>
<div>On Oct 26, 2019 20:25, Timothy Orme <torme@ancestry.com> wrote:<br />
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
Thats what I thought as well. All instances seem to be responding and alive according to the volume status. I also was able to run a `rebalance fix-layout` without any issues, so it seems that communication between the nodes is OK. I also tried replacing
the <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__10.158.10.1&d=DwMGaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=d0SJB4ihnau-Oyws6GEzcipkV9DfxCuMbgdSRgXeuxM&m=9egPta2lJ02rLk5KJSl3qn2mEUT8wQMfwioLnHY6a-s&s=A36clfcJL6IF2zIxCifcYh8D3F6cklL-hykkfB74Xos&e=">
</a><a href="http://10.158.10.1">10.158.10.1</a> brick with an entirely new server since that seemed to be the common one between in the logs. Self heal ran just fine in that replica set. However, it still is just hanging on the removal when I try and then remove those bricks.</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
<br />
</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
I might try and full rebalance as well, just to verify that it works.</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
<br />
</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
Only other thing I can think to note is that I'm using SSL for both client and server, and maybe thats obfuscating some more important error message, but it would still seem odd given that other communication between the nodes is just fine.</div>
<div style="font-family:'calibri' , 'arial' , 'helvetica' , sans-serif;font-size:12pt;color:rgb( 0 , 0 , 0 )">
<br />
</div>
<div></div></div></blockquote></div></div></div></blockquote></div>