<div dir="ltr"><div>By the way, on a slightly related note, I'm pretty sure either parallel-readdir or readdir-ahead has a regression in GlusterFS 3.12.x. We are running CentOS 7 with kernel-3.10.0-693.11.6.el7.x86_6.<br></div><div><br></div><div>I updated my servers and clients to 3.12.4 and enabled these two options after reading about them in the 3.10.0 and 3.11.0 release notes. In the days after enabling these two options all of my clients kept getting disconnected from the volume. The error upon attempting to list a directory or read a file was "Transport endpoint is not connected", after which I would force unmount the volume with `umount -fl /home` and remount it, only to have it get disconnected again a few hours later.</div><div><br></div><div>Every time the volume disconnected I looked in the client mount log and only found information such as:</div><div><br></div><div>[2018-01-24 05:52:27.695225] I [MSGID: 108026] [afr-self-heal-common.c:1656:afr_log_selfheal] 2-homes-replicate-1: Completed metadata selfheal on ed3fbafc-734b-41ca-ab30-216399fb9168. sources=[0] sinks=1 <br>[2018-01-24 05:52:27.700611] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 2-homes-replicate-1: performing metadata selfheal on b6a53629-a831-4ee3-a35e-f47c04297aaa<br>[2018-01-24 05:52:27.703021] I [MSGID: 108026] [afr-self-heal-common.c:1656:afr_log_selfheal] 2-homes-replicate-1: Completed metadata selfheal on b6a53629-a831-4ee3-a35e-f47c04297aaa. sources=[0] sinks=1<br></div><div><br></div><div>I enabled debug logging for that volume's client mount with `gluster volume set homes diagnostics.client-log-level DEBUG` and then I saw this in the client mount log the next time it disconnected:</div><div><br></div><div>[2018-01-24 08:55:19.138810] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-homes-io-threads: LOOKUP scheduled as fast fop<br>[2018-01-24 08:55:19.138849] D [MSGID: 0] [dht-common.c:2711:dht_lookup] 0-homes-dht: Calling fresh lookup for /vchebii/revtrans/Hircus-XM_018067032.1.pep.align.fas on homes-readdir-ahead-1<br>[2018-01-24 08:55:19.138928] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-homes-io-threads: FSTAT scheduled as fast fop<br>[2018-01-24 08:55:19.138958] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 0-homes-replicate-1: e6ee0427-b17d-4464-a738-e8ea70d77d95: generation now vs cached: 2, 2<br>[2018-01-24 08:55:19.139187] D [MSGID: 0] [dht-common.c:2294:dht_lookup_cbk] 0-homes-dht: fresh_lookup returned for /vchebii/revtrans/Hircus-XM_018067032.1.pep.align.fas with op_ret 0<br>[2018-01-24 08:55:19.139200] D [MSGID: 0] [dht-layout.c:873:dht_layout_preset] 0-homes-dht: file = 00000000-0000-0000-0000-000000000000, subvol = homes-readdir-ahead-1<br>[2018-01-24 08:55:19.139257] D [MSGID: 0] [io-threads.c:358:iot_schedule] 0-homes-io-threads: READDIRP scheduled as fast fop</div><div><br></div><div>On a hunch I disabled both parallel-readdir and readdir-ahead, which I had only enabled a few days before, and now all of the clients are much more stable, with zero disconnections in the days since I disabled those two volume options.<br></div><div><br></div><div>Please take a look! Thanks,<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Jan 24, 2018 at 5:59 AM Atin Mukherjee <<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Adding Poornima to take a look at it and comment.<br></div><div class="gmail_extra"><br><div class="gmail_quote"></div></div><div class="gmail_extra"><div class="gmail_quote">On Tue, Jan 23, 2018 at 10:39 PM, Alan Orth <span dir="ltr"><<a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a>></span> wrote:<br></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hello,<br><br></div><div>I saw that parallel-readdir was an experimental feature in GlusterFS version 3.10.0, became stable in version 3.11.0, and is now recommended for small file workloads in the Red Hat Gluster Storage Server documentation[2]. I've successfully enabled this on one of my volumes but I notice the following in the client mount log:<br><br>[2018-01-23 10:24:24.048055] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-homes-readdir-ahead-1: option 'parallel-readdir' is not recognized<br>[2018-01-23 10:24:24.048072] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-homes-readdir-ahead-0: option 'parallel-readdir' is not recognized<br><br></div><div>The GlusterFS version on the client and server is 3.12.4. What is going on?<br></div><div><br>[0] <a href="https://github.com/gluster/glusterfs/blob/release-3.10/doc/release-notes/3.10.0.md" target="_blank">https://github.com/gluster/glusterfs/blob/release-3.10/doc/release-notes/3.10.0.md</a><br>[1] <a href="https://github.com/gluster/glusterfs/blob/release-3.11/doc/release-notes/3.11.0.md" target="_blank">https://github.com/gluster/glusterfs/blob/release-3.11/doc/release-notes/3.11.0.md</a><br>[2] <a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/small_file_performance_enhancements" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/small_file_performance_enhancements</a><br><br></div><div>Thank you,<span class="m_-1273123752455108354HOEnZb"><font color="#888888"><br></font></span></div></div><span class="m_-1273123752455108354HOEnZb"><font color="#888888"><br clear="all"><br>-- <br><div dir="ltr" class="m_-1273123752455108354m_-1475148240828132585gmail_signature" data-smartmail="gmail_signature"><p dir="ltr">Alan Orth<br>
<a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br>
<a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br>
<a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br>
<a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a></p>
</div>
</font></span><br></blockquote></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><p dir="ltr">Alan Orth<br>
<a href="mailto:alan.orth@gmail.com">alan.orth@gmail.com</a><br>
<a href="https://picturingjordan.com">https://picturingjordan.com</a><br>
<a href="https://englishbulgaria.net">https://englishbulgaria.net</a><br>
<a href="https://mjanja.ch">https://mjanja.ch</a></p>
</div>