<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>It also matters how you mount it:<br></div><div></div><div>glusterfs defaults,_netdev,negative-timeout=10,attribute-timeout=30,fopen-keep-cache,direct-io-mode=enable,fetch-attempts=5 0 0<br><br></div><div><br></div><div>Options Reconfigured:<br>performance.io-thread-count: 8<br>server.allow-insecure: on<br>cluster.shd-max-threads: 12<br>performance.rda-cache-limit: 128MB<br>cluster.readdir-optimize: on<br>cluster.read-hash-mode: 0<br>performance.strict-o-direct: on<br>cluster.lookup-unhashed: auto<br>performance.nl-cache: on<br>performance.nl-cache-timeout: 600<br>cluster.lookup-optimize: on<br>client.event-threads: 4<br>performance.client-io-threads: on<br>performance.md-cache-timeout: 600<br>server.event-threads: 4<br>features.cache-invalidation: on<br>features.cache-invalidation-timeout: 600<br>performance.stat-prefetch: on<br>performance.cache-invalidation: on<br>network.inode-lru-limit: 90000<br>performance.cache-refresh-timeout: 10<br>performance.enable-least-priority: off<br>performance.cache-size: 2GB<br>cluster.nufa: on<br>cluster.choose-local: on<br>server.outstanding-rpc-limit: 128<br>disperse.eager-lock: off<br>nfs.disable: on<br>transport.address-family: inet<br><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 9, 2018 at 2:33 PM Pedro Costa <pedro@pmc.digital> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="m_-3680116981439725761m_3898616319650389354WordSection1">
<p class="MsoNormal"><span lang="PT" style="font-size:10.0pt;font-family:"Lato",sans-serif">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="PT" style="font-size:10.0pt;font-family:"Lato",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">I’ve a 1 x 3 replicated glusterfs 4.1.5 volume, that mounts using fuse on each server into /www for various Node apps that are proxied with nginx. Servers are then load balanced
to split traffic. Here’s the gvol1 configuration at the moment:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Volume Name: gvol1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Type: Replicate<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Volume ID: 384acec2-XXXX-40da-YYYY-5c53d12b3ae2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Status: Started<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Snapshot Count: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Number of Bricks: 1 x 3 = 3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Transport-type: tcp<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Bricks:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick1: vm0:/srv/brick1/gvol1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick2: vm1:/srv/brick1/gvol1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick3: vm2:/srv/brick1/gvol1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Options Reconfigured:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">cluster.strict-readdir: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">client.event-threads: 4<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">cluster.lookup-optimize: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">network.inode-lru-limit: 90000<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.md-cache-timeout: 600<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.cache-invalidation: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.cache-samba-metadata: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.stat-prefetch: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">features.cache-invalidation-timeout: 600<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">features.cache-invalidation: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">transport.address-family: inet<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">nfs.disable: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.client-io-threads: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">storage.fips-mode-rchecksum: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">features.utime: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">storage.ctime: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">server.event-threads: 4<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.cache-size: 500MB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.read-ahead: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">cluster.readdir-optimize: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">cluster.shd-max-threads: 6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.strict-o-direct: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">server.outstanding-rpc-limit: 128<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.enable-least-priority: off<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">cluster.nufa: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.nl-cache: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.nl-cache-timeout: 60<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.cache-refresh-timeout: 10<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.rda-cache-limit: 128MB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.readdir-ahead: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.parallel-readdir: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">disperse.eager-lock: off<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">network.ping-timeout: 5<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">cluster.background-self-heal-count: 20<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">cluster.self-heal-window-size: 2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">cluster.self-heal-readdir-size: 2KB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">On each restart the apps delete a particular folder and rebuild it from internal packages. On one such operation on a particular client to the volume I get repeated logs hundreds
of times for the same guid even, sometimes:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">[2018-10-09 13:40:40.579161] W [MSGID: 114061] [client-common.c:2658:client_pre_flush_v2] 0-gvol1-client-2: (7955fd7a-3147-48b3-bf6a-5306ac97e10d) remote_fd is -1. EBADFD [File
descriptor in bad state]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">[2018-10-09 13:40:40.579313] W [MSGID: 114061] [client-common.c:2658:client_pre_flush_v2] 0-gvol1-client-2: (0ac67ee4-a31e-4989-ba1e-e4f513c1f757) remote_fd is -1. EBADFD [File
descriptor in bad state]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">[2018-10-09 13:40:40.579707] W [MSGID: 114061] [client-common.c:2658:client_pre_flush_v2] 0-gvol1-client-2: (7ea6106d-29f4-4a19-8eb6-6515ffefb9d3) remote_fd is -1. EBADFD [File
descriptor in bad state]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">[2018-10-09 13:40:40.579911] W [MSGID: 114061] [client-common.c:2658:client_pre_flush_v2] 0-gvol1-client-2: (7ea6106d-29f4-4a19-8eb6-6515ffefb9d3) remote_fd is -1. EBADFD [File
descriptor in bad state]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">I assume this is probably because the client didn’t catch up with the previous delete? I do control the server (client to the gluster volume) that the restart occurs, and I prevent
having more than one rebuilding the same app at the same time, which makes these logs odd.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">I’ve implemented the volume options above after reading most of the entries in the archive here over the last few weeks, but I’m not sure what else to tweak because other than
the restart of the apps it is working pretty well.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">If there’s any input you may have to help on this particular scenario I’d be much appreciated.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">P.<u></u><u></u></span></p>
</div>
</div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>