<div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 20, 2019 at 9:52 AM Artem Russakovskii <<a href="mailto:archon810@gmail.com">archon810@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Can I roll back performance.write-behind: off and lru-limit=0 then? I'm<br>
waiting for the debug packages to be available for OpenSUSE, then I can<br>
help Amar with another debug session.<br>
<br></blockquote><div><br></div><div>Yes, the write-behind issue is now fixed. You can enable write-behind. Also remove lru-limit=0, so you can also utilize the benefit of garbage collection introduced in 5.4</div><div><br></div><div>Lets get to fixing the problem once the debuginfo packages are available.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
In the meantime, have you had time to set up 1x4 replicate testing? I was<br>
told you were only testing 1x3, and it's the 4th brick that may be causing<br>
the crash, which is consistent with this whole time only 1 of 4 bricks<br>
constantly crashing. The other 3 have been rock solid. I'm hoping you could<br>
find the issue without a debug session this way.<br>
<br></blockquote><div><br></div><div>That is my gut feeling still. Added a basic test case with 4 bricks, <a href="https://review.gluster.org/#/c/glusterfs/+/22328/">https://review.gluster.org/#/c/glusterfs/+/22328/</a>. But I think this particular issue is happening only on certain pattern of access for 1x4 setup. Lets get to the root of it once we have debuginfo packages for Suse builds.</div><div> </div><div>-Amar</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, Android Police <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK Mirror<br>
<<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical Robot LLC<br>
<a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> | +ArtemRussakovskii<br>
<<a href="https://plus.google.com/+ArtemRussakovskii" rel="noreferrer" target="_blank">https://plus.google.com/+ArtemRussakovskii</a>> | @ArtemR<br>
<<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
<br>
<br>
On Tue, Mar 19, 2019 at 8:27 PM Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
wrote:<br>
<br>
> Hi Artem,<br>
><br>
> I think you are running into a different crash. The ones reported which<br>
> were prevented by turning off write-behind are now fixed.<br>
> We will need to look into the one you are seeing to see why it is<br>
> happening.<br>
><br>
> Regards,<br>
> Nithya<br>
><br>
><br>
> On Tue, 19 Mar 2019 at 20:25, Artem Russakovskii <<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>><br>
> wrote:<br>
><br>
>> The flood is indeed fixed for us on 5.5. However, the crashes are not.<br>
>><br>
>> Sincerely,<br>
>> Artem<br>
>><br>
>> --<br>
>> Founder, Android Police <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK Mirror<br>
>> <<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical Robot LLC<br>
>> <a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> | +ArtemRussakovskii<br>
>> <<a href="https://plus.google.com/+ArtemRussakovskii" rel="noreferrer" target="_blank">https://plus.google.com/+ArtemRussakovskii</a>> | @ArtemR<br>
>> <<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
>><br>
>><br>
>> On Mon, Mar 18, 2019 at 5:41 AM Hu Bert <<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>> wrote:<br>
>><br>
>>> Hi Amar,<br>
>>><br>
>>> if you refer to this bug:<br>
>>> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1674225" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1674225</a> : in the test<br>
>>> setup i haven't seen those entries, while copying & deleting a few GBs<br>
>>> of data. For a final statement we have to wait until i updated our<br>
>>> live gluster servers - could take place on tuesday or wednesday.<br>
>>><br>
>>> Maybe other users can do an update to 5.4 as well and report back here.<br>
>>><br>
>>><br>
>>> Hubert<br>
>>><br>
>>><br>
>>><br>
>>> Am Mo., 18. März 2019 um 11:36 Uhr schrieb Amar Tumballi Suryanarayan<br>
>>> <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>>:<br>
>>> ><br>
>>> > Hi Hu Bert,<br>
>>> ><br>
>>> > Appreciate the feedback. Also are the other boiling issues related to<br>
>>> logs fixed now?<br>
>>> ><br>
>>> > -Amar<br>
>>> ><br>
>>> > On Mon, Mar 18, 2019 at 3:54 PM Hu Bert <<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>><br>
>>> wrote:<br>
>>> >><br>
>>> >> update: upgrade from 5.3 -> 5.5 in a replicate 3 test setup with 2<br>
>>> >> volumes done. In 'gluster peer status' the peers stay connected during<br>
>>> >> the upgrade, no 'peer rejected' messages. No cksum mismatches in the<br>
>>> >> logs. Looks good :-)<br>
>>> >><br>
>>> >> Am Mo., 18. März 2019 um 09:54 Uhr schrieb Hu Bert <<br>
>>> <a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>>:<br>
>>> >> ><br>
>>> >> > Good morning :-)<br>
>>> >> ><br>
>>> >> > for debian the packages are there:<br>
>>> >> ><br>
>>> <a href="https://download.gluster.org/pub/gluster/glusterfs/5/5.5/Debian/stretch/amd64/apt/pool/main/g/glusterfs/" rel="noreferrer" target="_blank">https://download.gluster.org/pub/gluster/glusterfs/5/5.5/Debian/stretch/amd64/apt/pool/main/g/glusterfs/</a><br>
>>> >> ><br>
>>> >> > I'll do an upgrade of a test installation 5.3 -> 5.5 and see if<br>
>>> there<br>
>>> >> > are some errors etc. and report back.<br>
>>> >> ><br>
>>> >> > btw: no release notes for 5.4 and 5.5 so far?<br>
>>> >> > <a href="https://docs.gluster.org/en/latest/release-notes/" rel="noreferrer" target="_blank">https://docs.gluster.org/en/latest/release-notes/</a> ?<br>
>>> >> ><br>
>>> >> > Am Fr., 15. März 2019 um 14:28 Uhr schrieb Shyam Ranganathan<br>
>>> >> > <<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>>:<br>
>>> >> > ><br>
>>> >> > > We created a 5.5 release tag, and it is under packaging now. It<br>
>>> should<br>
>>> >> > > be packaged and ready for testing early next week and should be<br>
>>> released<br>
>>> >> > > close to mid-week next week.<br>
>>> >> > ><br>
>>> >> > > Thanks,<br>
>>> >> > > Shyam<br>
>>> >> > > On 3/13/19 12:34 PM, Artem Russakovskii wrote:<br>
>>> >> > > > Wednesday now with no update :-/<br>
>>> >> > > ><br>
>>> >> > > > Sincerely,<br>
>>> >> > > > Artem<br>
>>> >> > > ><br>
>>> >> > > > --<br>
>>> >> > > > Founder, Android Police <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK<br>
>>> Mirror<br>
>>> >> > > > <<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical Robot LLC<br>
>>> >> > > > <a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> <<a href="http://beerpla.net/" rel="noreferrer" target="_blank">http://beerpla.net/</a>> | +ArtemRussakovskii<br>
>>> >> > > > <<a href="https://plus.google.com/+ArtemRussakovskii" rel="noreferrer" target="_blank">https://plus.google.com/+ArtemRussakovskii</a>> | @ArtemR<br>
>>> >> > > > <<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
>>> >> > > ><br>
>>> >> > > ><br>
>>> >> > > > On Tue, Mar 12, 2019 at 10:28 AM Artem Russakovskii <<br>
>>> <a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a><br>
>>> >> > > > <mailto:<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>>> wrote:<br>
>>> >> > > ><br>
>>> >> > > > Hi Amar,<br>
>>> >> > > ><br>
>>> >> > > > Any updates on this? I'm still not seeing it in OpenSUSE<br>
>>> build<br>
>>> >> > > > repos. Maybe later today?<br>
>>> >> > > ><br>
>>> >> > > > Thanks.<br>
>>> >> > > ><br>
>>> >> > > > Sincerely,<br>
>>> >> > > > Artem<br>
>>> >> > > ><br>
>>> >> > > > --<br>
>>> >> > > > Founder, Android Police <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>,<br>
>>> APK Mirror<br>
>>> >> > > > <<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical Robot LLC<br>
>>> >> > > > <a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> <<a href="http://beerpla.net/" rel="noreferrer" target="_blank">http://beerpla.net/</a>> | +ArtemRussakovskii<br>
>>> >> > > > <<a href="https://plus.google.com/+ArtemRussakovskii" rel="noreferrer" target="_blank">https://plus.google.com/+ArtemRussakovskii</a>> | @ArtemR<br>
>>> >> > > > <<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
>>> >> > > ><br>
>>> >> > > ><br>
>>> >> > > > On Wed, Mar 6, 2019 at 10:30 PM Amar Tumballi Suryanarayan<br>
>>> >> > > > <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a> <mailto:<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>>> wrote:<br>
>>> >> > > ><br>
>>> >> > > > We are talking days. Not weeks. Considering already it<br>
>>> is<br>
>>> >> > > > Thursday here. 1 more day for tagging, and packaging.<br>
>>> May be ok<br>
>>> >> > > > to expect it on Monday.<br>
>>> >> > > ><br>
>>> >> > > > -Amar<br>
>>> >> > > ><br>
>>> >> > > > On Thu, Mar 7, 2019 at 11:54 AM Artem Russakovskii<br>
>>> >> > > > <<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a> <mailto:<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>>><br>
>>> wrote:<br>
>>> >> > > ><br>
>>> >> > > > Is the next release going to be an imminent hotfix,<br>
>>> i.e.<br>
>>> >> > > > something like today/tomorrow, or are we talking<br>
>>> weeks?<br>
>>> >> > > ><br>
>>> >> > > > Sincerely,<br>
>>> >> > > > Artem<br>
>>> >> > > ><br>
>>> >> > > > --<br>
>>> >> > > > Founder, Android Police <<br>
>>> <a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK<br>
>>> >> > > > Mirror <<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical<br>
>>> Robot LLC<br>
>>> >> > > > <a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> <<a href="http://beerpla.net/" rel="noreferrer" target="_blank">http://beerpla.net/</a>> |<br>
>>> +ArtemRussakovskii<br>
>>> >> > > > <<a href="https://plus.google.com/+ArtemRussakovskii" rel="noreferrer" target="_blank">https://plus.google.com/+ArtemRussakovskii</a>> |<br>
>>> @ArtemR<br>
>>> >> > > > <<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
>>> >> > > ><br>
>>> >> > > ><br>
>>> >> > > > On Tue, Mar 5, 2019 at 11:09 AM Artem Russakovskii<br>
>>> >> > > > <<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a> <mailto:<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>>><br>
>>> wrote:<br>
>>> >> > > ><br>
>>> >> > > > Ended up downgrading to 5.3 just in case. Peer<br>
>>> status<br>
>>> >> > > > and volume status are OK now.<br>
>>> >> > > ><br>
>>> >> > > > zypper install --oldpackage<br>
>>> glusterfs-5.3-lp150.100.1<br>
>>> >> > > > Loading repository data...<br>
>>> >> > > > Reading installed packages...<br>
>>> >> > > > Resolving package dependencies...<br>
>>> >> > > ><br>
>>> >> > > > Problem: glusterfs-5.3-lp150.100.1.x86_64<br>
>>> requires<br>
>>> >> > > > libgfapi0 = 5.3, but this requirement cannot be<br>
>>> provided<br>
>>> >> > > > not installable providers:<br>
>>> >> > > > libgfapi0-5.3-lp150.100.1.x86_64[glusterfs]<br>
>>> >> > > > Solution 1: Following actions will be done:<br>
>>> >> > > > downgrade of libgfapi0-5.4-lp150.100.1.x86_64<br>
>>> to<br>
>>> >> > > > libgfapi0-5.3-lp150.100.1.x86_64<br>
>>> >> > > > downgrade of<br>
>>> libgfchangelog0-5.4-lp150.100.1.x86_64 to<br>
>>> >> > > > libgfchangelog0-5.3-lp150.100.1.x86_64<br>
>>> >> > > > downgrade of libgfrpc0-5.4-lp150.100.1.x86_64<br>
>>> to<br>
>>> >> > > > libgfrpc0-5.3-lp150.100.1.x86_64<br>
>>> >> > > > downgrade of libgfxdr0-5.4-lp150.100.1.x86_64<br>
>>> to<br>
>>> >> > > > libgfxdr0-5.3-lp150.100.1.x86_64<br>
>>> >> > > > downgrade of<br>
>>> libglusterfs0-5.4-lp150.100.1.x86_64 to<br>
>>> >> > > > libglusterfs0-5.3-lp150.100.1.x86_64<br>
>>> >> > > > Solution 2: do not install<br>
>>> glusterfs-5.3-lp150.100.1.x86_64<br>
>>> >> > > > Solution 3: break<br>
>>> glusterfs-5.3-lp150.100.1.x86_64 by<br>
>>> >> > > > ignoring some of its dependencies<br>
>>> >> > > ><br>
>>> >> > > > Choose from above solutions by number or cancel<br>
>>> >> > > > [1/2/3/c] (c): 1<br>
>>> >> > > > Resolving dependencies...<br>
>>> >> > > > Resolving package dependencies...<br>
>>> >> > > ><br>
>>> >> > > > The following 6 packages are going to be<br>
>>> downgraded:<br>
>>> >> > > > glusterfs libgfapi0 libgfchangelog0 libgfrpc0<br>
>>> >> > > > libgfxdr0 libglusterfs0<br>
>>> >> > > ><br>
>>> >> > > > 6 packages to downgrade.<br>
>>> >> > > ><br>
>>> >> > > > Sincerely,<br>
>>> >> > > > Artem<br>
>>> >> > > ><br>
>>> >> > > > --<br>
>>> >> > > > Founder, Android Police<br>
>>> >> > > > <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK Mirror<br>
>>> >> > > > <<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical Robot<br>
>>> LLC<br>
>>> >> > > > <a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> <<a href="http://beerpla.net/" rel="noreferrer" target="_blank">http://beerpla.net/</a>> |<br>
>>> +ArtemRussakovskii<br>
>>> >> > > > <<a href="https://plus.google.com/+ArtemRussakovskii" rel="noreferrer" target="_blank">https://plus.google.com/+ArtemRussakovskii</a>> |<br>
>>> @ArtemR<br>
>>> >> > > > <<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
>>> >> > > ><br>
>>> >> > > ><br>
>>> >> > > > On Tue, Mar 5, 2019 at 10:57 AM Artem<br>
>>> Russakovskii<br>
>>> >> > > > <<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a> <mailto:<br>
>>> <a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>>> wrote:<br>
>>> >> > > ><br>
>>> >> > > > Noticed the same when upgrading from 5.3 to<br>
>>> 5.4, as<br>
>>> >> > > > mentioned.<br>
>>> >> > > ><br>
>>> >> > > > I'm confused though. Is actual replication<br>
>>> affected,<br>
>>> >> > > > because the 5.4 server and the 3x 5.3<br>
>>> servers still<br>
>>> >> > > > show heal info as all 4 connected, and the<br>
>>> files<br>
>>> >> > > > seem to be replicating correctly as well.<br>
>>> >> > > ><br>
>>> >> > > > So what's actually affected - just the<br>
>>> status<br>
>>> >> > > > command, or leaving 5.4 on one of the nodes<br>
>>> is doing<br>
>>> >> > > > some damage to the underlying fs? Is it<br>
>>> fixable by<br>
>>> >> > > > tweaking transport.socket.ssl-enabled? Does<br>
>>> >> > > > upgrading all servers to 5.4 resolve it, or<br>
>>> should<br>
>>> >> > > > we revert back to 5.3?<br>
>>> >> > > ><br>
>>> >> > > > Sincerely,<br>
>>> >> > > > Artem<br>
>>> >> > > ><br>
>>> >> > > > --<br>
>>> >> > > > Founder, Android Police<br>
>>> >> > > > <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK Mirror<br>
>>> >> > > > <<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical<br>
>>> Robot LLC<br>
>>> >> > > > <a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> <<a href="http://beerpla.net/" rel="noreferrer" target="_blank">http://beerpla.net/</a>> |<br>
>>> >> > > > +ArtemRussakovskii<br>
>>> >> > > > <<a href="https://plus.google.com/+ArtemRussakovskii" rel="noreferrer" target="_blank">https://plus.google.com/+ArtemRussakovskii</a><br>
>>> ><br>
>>> >> > > > | @ArtemR <<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
>>> >> > > ><br>
>>> >> > > ><br>
>>> >> > > > On Tue, Mar 5, 2019 at 2:02 AM Hu Bert<br>
>>> >> > > > <<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a><br>
>>> >> > > > <mailto:<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>>> wrote:<br>
>>> >> > > ><br>
>>> >> > > > fyi: did a downgrade 5.4 -> 5.3 and it<br>
>>> worked.<br>
>>> >> > > > all replicas are up and<br>
>>> >> > > > running. Awaiting updated v5.4.<br>
>>> >> > > ><br>
>>> >> > > > thx :-)<br>
>>> >> > > ><br>
>>> >> > > > Am Di., 5. März 2019 um 09:26 Uhr<br>
>>> schrieb Hari<br>
>>> >> > > > Gowtham <<a href="mailto:hgowtham@redhat.com" target="_blank">hgowtham@redhat.com</a><br>
>>> >> > > > <mailto:<a href="mailto:hgowtham@redhat.com" target="_blank">hgowtham@redhat.com</a>>>:<br>
>>> >> > > > ><br>
>>> >> > > > > There are plans to revert the patch<br>
>>> causing<br>
>>> >> > > > this error and rebuilt 5.4.<br>
>>> >> > > > > This should happen faster. the<br>
>>> rebuilt 5.4<br>
>>> >> > > > should be void of this upgrade issue.<br>
>>> >> > > > ><br>
>>> >> > > > > In the meantime, you can use 5.3 for<br>
>>> this cluster.<br>
>>> >> > > > > Downgrading to 5.3 will work if it<br>
>>> was just<br>
>>> >> > > > one node that was upgrade to 5.4<br>
>>> >> > > > > and the other nodes are still in 5.3</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div></div></div>