<div dir="ltr">This thread has been one of the largest effort to stabilize the systems in recent times.<div><br></div><div>Thanks for patience and number of retries you did, Erik!</div><div><br></div><div>We surely need to get to the glitch you found with the 7.4 version, as with every higher version, we expect more stability!</div><div><br></div><div>Regards,</div><div>Amar</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 17, 2020 at 2:46 AM Erik Jacobson <<a href="mailto:erik.jacobson@hpe.com" target="_blank">erik.jacobson@hpe.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I have some news.<br>
<br>
After many many many trials, reboots of gluster servers, reboots of nodes...<br>
in what should have reproduced the issue several times. I think we're<br>
stable.<br>
<br>
It appears this glusterfs nfs daemon hang only happens in glusterfs74<br>
and not 72.<br>
<br>
So....<br>
1) Your split-brain patch<br>
2) performance.parallel-readdir off<br>
3) glusterfs72<br>
<br>
I declare it stable. I can't make it fail: split-brain, hang, noor seg fault<br>
with one leader down.<br>
<br>
I'm working on putting this in to a SW update.<br>
<br>
We are going to test if performance.parallel-readdir off impacts booting<br>
at scale but we don't have a system to try it on at this time.<br>
<br>
THAK YOU!<br>
<br>
I may have access to the 57 node test system if there is something you'd<br>
like me to try with regards to why glusterfs74 is unstable in this<br>
situation. Just let me know.<br>
<br>
Erik<br>
<br>
On Thu, Apr 16, 2020 at 12:03:33PM -0500, Erik Jacobson wrote:<br>
> So in my test runs since making that change, we have a different odd<br>
> behavior now. As you recall, this is with your patch -- still not<br>
> split-brain -- and now with performance.parallel-readdir off<br>
> <br>
> The NFS server grinds to a hault after a few test runs. It does not core<br>
> dump.<br>
> <br>
> All that shows up in the log is:<br>
> <br>
> "pending frames:" with nothing after it and no date stamp.<br>
> <br>
> I will start looking for interesting break points I guess.<br>
> <br>
> <br>
> The glusterfs for nfs is still alive:<br>
> <br>
> root 30541 1 42 09:57 ? 00:51:06 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/run/gluster/nfs/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/9ddb5561058ff543.socket<br>
> <br>
> <br>
> <br>
> [root@leader3 ~]# strace -f -p 30541<br>
> strace: Process 30541 attached with 40 threads<br>
> [pid 30580] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30579] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30578] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30577] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30576] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30575] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30574] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30573] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30572] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30571] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30570] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30569] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30568] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30567] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30566] futex(0x7f88b8000020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30565] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30564] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30563] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30562] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30561] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30560] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30559] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30558] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30557] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30556] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30555] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30554] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30553] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30552] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30551] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30550] restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...><br>
> [pid 30549] futex(0x7f8904035f60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30548] futex(0x7f88b8000020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=243775} <unfinished ...><br>
> [pid 30546] restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...><br>
> [pid 30545] restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...><br>
> [pid 30544] futex(0x7f88b8000020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30543] rt_sigtimedwait([HUP INT USR1 USR2 TERM], <unfinished ...><br>
> [pid 30542] futex(0x7f88b8000020, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...><br>
> [pid 30541] futex(0x7f890c3a39d0, FUTEX_WAIT, 30548, NULL <unfinished ...><br>
> [pid 30547] <... select resumed> ) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}) = 0 (Timeout)<br>
> [pid 30547] select(0, NULL, NULL, NULL, {tv_sec=1, tv_usec=0}^Cstrace: Process 30541 detached<br>
> strace: Process 30542 detached<br>
> strace: Process 30543 detached<br>
> strace: Process 30544 detached<br>
> strace: Process 30545 detached<br>
> strace: Process 30546 detached<br>
> strace: Process 30547 detached<br>
> <detached ...><br>
> strace: Process 30548 detached<br>
> strace: Process 30549 detached<br>
> strace: Process 30550 detached<br>
> strace: Process 30551 detached<br>
> strace: Process 30552 detached<br>
> strace: Process 30553 detached<br>
> strace: Process 30554 detached<br>
> strace: Process 30555 detached<br>
> strace: Process 30556 detached<br>
> strace: Process 30557 detached<br>
> strace: Process 30558 detached<br>
> strace: Process 30559 detached<br>
> strace: Process 30560 detached<br>
> strace: Process 30561 detached<br>
> strace: Process 30562 detached<br>
> strace: Process 30563 detached<br>
> strace: Process 30564 detached<br>
> strace: Process 30565 detached<br>
> strace: Process 30566 detached<br>
> strace: Process 30567 detached<br>
> strace: Process 30568 detached<br>
> strace: Process 30569 detached<br>
> strace: Process 30570 detached<br>
> strace: Process 30571 detached<br>
> strace: Process 30572 detached<br>
> strace: Process 30573 detached<br>
> strace: Process 30574 detached<br>
> strace: Process 30575 detached<br>
> strace: Process 30576 detached<br>
> strace: Process 30577 detached<br>
> strace: Process 30578 detached<br>
> strace: Process 30579 detached<br>
> strace: Process 30580 detached<br>
> <br>
> <br>
> <br>
> <br>
> > On 16/04/20 8:04 pm, Erik Jacobson wrote:<br>
> > > Quick update just on how this got set.<br>
> > > <br>
> > > gluster volume set cm_shared performance.parallel-readdir on<br>
> > > <br>
> > > Is something we did turn on, thinking it might make our NFS services<br>
> > > faster and not knowing about it possibly being negative.<br>
> > > <br>
> > > Below is a diff of the nfs volume file ON vs OFF. So I will simply turn<br>
> > > this OFF and do a test run.<br>
> > Yes,that should do it. I am not sure if performance.parallel-readdir was<br>
> > intentionally made to have an effect on gnfs volfiles. Usually, for other<br>
> > performance xlators, `gluster volume set` only changes the fuse volfile.<br>
<br>
<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">--<div><a href="https://kadalu.io" target="_blank">https://kadalu.io</a></div><div>Container Storage made easy!</div><div><br></div></div></div>