<div dir="ltr">I resolved this with the following settings, particularly disabling features.ctr-enabled <br><br>olume Name: storage2<br>Type: Distributed-Replicate<br>Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 3 x (2 + 1) = 9<br>Transport-type: tcp<br>Bricks:<br>Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2<br>Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2<br>Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter)<br>Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2<br>Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2<br>Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter)<br>Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2<br>Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2<br>Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter)<br>Options Reconfigured:<br>performance.write-behind-window-size: 4MB<br>performance.cache-invalidation: on<br>transport.keepalive: on<br>performance.write-behind: on<br>performance.read-ahead: on<br>performance.io-cache: on<br>performance.stat-prefetch: on<br>performance.open-behind: on<br>cluster.use-compound-fops: on<br>performance.cache-ima-xattrs: on<br>features.cache-invalidation: on<br>client.event-threads: 4<br>cluster.data-self-heal-algorithm: full<br>performance.client-io-threads: on<br>server.event-threads: 4<br>performance.quick-read: on<br>features.scrub: Active<br>features.bitrot: on<br>features.shard: on<br>transport.address-family: inet6<br>nfs.disable: on<br>server.allow-insecure: on<br>user.cifs: off<br>cluster.quorum-type: auto<br>cluster.server-quorum-type: server<br>cluster.tier-compact: on<br>diagnostics.brick-log-level: WARNING<br>diagnostics.client-log-level: WARNING<br>cluster.self-heal-daemon: enable<br>performance.cache-samba-metadata: on<br>cluster.brick-multiplex: off<br>cluster.enable-shared-storage: enable<br>nfs-ganesha: enable<br><br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><br>-ben<br></div></div>
<br><div class="gmail_quote">On Sat, May 13, 2017 at 12:20 PM, Benjamin Kingston <span dir="ltr"><<a href="mailto:ben@nexusnebula.net" target="_blank">ben@nexusnebula.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hers's some log entries from nfs-ganesha gfapi<br><br>[2017-05-13 19:02:54.105936] E [MSGID: 133010] [shard.c:1706:shard_common_<wbr>lookup_shards_cbk] 0-storage2-shard: Lookup on shard 11 failed. Base file gfid = 1494c083-a618-4eba-80a0-<wbr>147e656dd9d0 [Input/output error]<br>[2017-05-13 19:02:54.106176] E [MSGID: 133010] [shard.c:1706:shard_common_<wbr>lookup_shards_cbk] 0-storage2-shard: Lookup on shard 2 failed. Base file gfid = 1494c083-a618-4eba-80a0-<wbr>147e656dd9d0 [Input/output error]<br>[2017-05-13 19:02:54.106288] E [MSGID: 133010] [shard.c:1706:shard_common_<wbr>lookup_shards_cbk] 0-storage2-shard: Lookup on shard 1 failed. Base file gfid = 1494c083-a618-4eba-80a0-<wbr>147e656dd9d0 [Input/output error]<br>[2017-05-13 19:02:54.384922] I [MSGID: 108026] [afr-self-heal-metadata.c:52:_<wbr>_afr_selfheal_metadata_do] 0-storage2-replicate-2: performing metadata selfheal on fe651475-226e-42a3-be2d-<wbr>751d4f58e383<br>[2017-05-13 19:02:54.385894] W [MSGID: 114031] [client-rpc-fops.c:2258:<wbr>client3_3_setattr_cbk] 0-storage2-client-8: remote operation failed [Operation not permitted]<br>[2017-05-13 19:02:54.401187] I [MSGID: 108026] [afr-self-heal-common.c:1255:<wbr>afr_log_selfheal] 0-storage2-replicate-2: Completed metadata selfheal on fe651475-226e-42a3-be2d-<wbr>751d4f58e383. sources=[0] 1 sinks=<br>[2017-05-13 19:02:57.830019] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>par2.tmp (hash=storage2-readdir-ahead-<wbr>2/cache=storage2-readdir-<wbr>ahead-2) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>par2 (hash=storage2-readdir-ahead-<wbr>0/cache=<nul>)<br><br>[2017-05-13 19:08:22.014899] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>srr.tmp (hash=storage2-readdir-ahead-<wbr>1/cache=storage2-readdir-<wbr>ahead-1) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.srr (hash=storage2-readdir-ahead-<wbr>1/cache=<nul>)<br>[2017-05-13 19:08:22.463840] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r04.tmp (hash=storage2-readdir-ahead-<wbr>2/cache=storage2-readdir-<wbr>ahead-2) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r04 (hash=storage2-readdir-ahead-<wbr>0/cache=<nul>)<br>[2017-05-13 19:08:22.769542] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r01.tmp (hash=storage2-readdir-ahead-<wbr>2/cache=storage2-readdir-<wbr>ahead-2) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r01 (hash=storage2-readdir-ahead-<wbr>0/cache=<nul>)<br>[2017-05-13 19:08:23.141069] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>nfo.tmp (hash=storage2-readdir-ahead-<wbr>1/cache=storage2-readdir-<wbr>ahead-1) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.nfo (hash=storage2-readdir-ahead-<wbr>0/cache=<nul>)<br>[2017-05-13 19:08:23.468554] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r00.tmp (hash=storage2-readdir-ahead-<wbr>0/cache=storage2-readdir-<wbr>ahead-0) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r00 (hash=storage2-readdir-ahead-<wbr>2/cache=<nul>)<br>[2017-05-13 19:08:23.671753] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>sfv.tmp (hash=storage2-readdir-ahead-<wbr>2/cache=storage2-readdir-<wbr>ahead-2) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.sfv (hash=storage2-readdir-ahead-<wbr>2/cache=<nul>)<br>[2017-05-13 19:08:23.812152] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r11.tmp (hash=storage2-readdir-ahead-<wbr>0/cache=storage2-readdir-<wbr>ahead-0) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r11 (hash=storage2-readdir-ahead-<wbr>1/cache=<nul>)<br>[2017-05-13 19:08:24.244584] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r12.tmp (hash=storage2-readdir-ahead-<wbr>0/cache=storage2-readdir-<wbr>ahead-0) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r12 (hash=storage2-readdir-ahead-<wbr>1/cache=<nul>)<br>[2017-05-13 19:08:24.711781] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r09.tmp (hash=storage2-readdir-ahead-<wbr>1/cache=storage2-readdir-<wbr>ahead-1) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r09 (hash=storage2-readdir-ahead-<wbr>1/cache=<nul>)<br>[2017-05-13 19:08:25.029102] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r06.tmp (hash=storage2-readdir-ahead-<wbr>0/cache=storage2-readdir-<wbr>ahead-0) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r06 (hash=storage2-readdir-ahead-<wbr>0/cache=<nul>)<br>[2017-05-13 19:08:25.206011] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r05.tmp (hash=storage2-readdir-ahead-<wbr>0/cache=storage2-readdir-<wbr>ahead-0) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r05 (hash=storage2-readdir-ahead-<wbr>0/cache=<nul>)<br>[2017-05-13 19:08:25.467312] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r03.tmp (hash=storage2-readdir-ahead-<wbr>1/cache=storage2-readdir-<wbr>ahead-1) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r03 (hash=storage2-readdir-ahead-<wbr>1/cache=<nul>)<br>[2017-05-13 19:08:25.726208] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r07.tmp (hash=storage2-readdir-ahead-<wbr>0/cache=storage2-readdir-<wbr>ahead-0) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r07 (hash=storage2-readdir-ahead-<wbr>0/cache=<nul>)<br>[2017-05-13 19:08:25.860386] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>rar.tmp (hash=storage2-readdir-ahead-<wbr>1/cache=storage2-readdir-<wbr>ahead-1) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.rar (hash=storage2-readdir-ahead-<wbr>1/cache=<nul>)<br>[2017-05-13 19:09:00.803859] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r08.tmp (hash=storage2-readdir-ahead-<wbr>2/cache=storage2-readdir-<wbr>ahead-2) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r08 (hash=storage2-readdir-ahead-<wbr>0/cache=<nul>)<br>[2017-05-13 19:09:01.826525] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r02.tmp (hash=storage2-readdir-ahead-<wbr>2/cache=storage2-readdir-<wbr>ahead-2) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r02 (hash=storage2-readdir-ahead-<wbr>2/cache=<nul>)<br>[2017-05-13 19:09:02.531354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Videos/Cinema/Bambi (1942)/.Bambi (1942).mkv.hp9oW8 (hash=storage2-readdir-ahead-<wbr>2/cache=storage2-readdir-<wbr>ahead-2) => /content/Videos/Cinema/Bambi (1942)/.<wbr>nfsa223c532d51b3c7700000006 (hash=storage2-readdir-ahead-<wbr>2/cache=<nul>)<br>[2017-05-13 19:09:39.695354] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.<wbr>r10.tmp (hash=storage2-readdir-ahead-<wbr>2/cache=storage2-readdir-<wbr>ahead-2) => /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r10 (hash=storage2-readdir-ahead-<wbr>2/cache=<nul>)<br>[2017-05-13 19:10:59.938286] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-<wbr>1732d24f30f7): trying duplicate remote fd set. <br>[2017-05-13 19:10:59.938510] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-<wbr>1732d24f30f7): trying duplicate remote fd set. <br>[2017-05-13 19:10:59.939004] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r11 (7dcd980b-4f8b-4e0c-8fb9-<wbr>1732d24f30f7): trying duplicate remote fd set. <br>[2017-05-13 19:10:59.939246] E [shard.c:426:shard_modify_<wbr>size_and_block_count] (-->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/cluster/distribute.<wbr>so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/features/shard.so(+<wbr>0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/features/shard.so(+<wbr>0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-<wbr>size for 7dcd980b-4f8b-4e0c-8fb9-<wbr>1732d24f30f7<br>[2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-<wbr>fb0205f5d99e): trying duplicate remote fd set. <br>[2017-05-13 19:11:01.884482] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-<wbr>fb0205f5d99e): trying duplicate remote fd set. <br>[2017-05-13 19:11:01.884557] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r12 (2b3e3ab2-ba6a-4ad3-af25-<wbr>fb0205f5d99e): trying duplicate remote fd set. <br>[2017-05-13 19:11:01.884918] E [shard.c:426:shard_modify_<wbr>size_and_block_count] (-->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/cluster/distribute.<wbr>so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/features/shard.so(+<wbr>0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/features/shard.so(+<wbr>0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-<wbr>size for 2b3e3ab2-ba6a-4ad3-af25-<wbr>fb0205f5d99e<br>[2017-05-13 19:11:05.136517] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-0: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-<wbr>22e9597fa787): trying duplicate remote fd set. <br>[2017-05-13 19:11:05.136518] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-1: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-<wbr>22e9597fa787): trying duplicate remote fd set. <br>[2017-05-13 19:11:05.136648] I [MSGID: 114024] [client-helpers.c:99:this_fd_<wbr>set_ctx] 0-storage2-client-6: /content/Downloads/incomplete/<wbr>usenet/Attack.on.Titan.S02E05.<wbr>720p.WEB.x264-ANiURL.#27/<wbr>aniurl-aot.s02e05.720p.web.r00 (b2745d17-1972-4738-afa9-<wbr>22e9597fa787): trying duplicate remote fd set. <br>[2017-05-13 19:11:05.136871] E [shard.c:426:shard_modify_<wbr>size_and_block_count] (-->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/cluster/distribute.<wbr>so(+0x65df0) [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/features/shard.so(+<wbr>0xba6e) [0x7f8c495eca6e] -->/usr/lib64/glusterfs/3.10.<wbr>1/xlator/features/shard.so(+<wbr>0xb29b) [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get trusted.glusterfs.shard.file-<wbr>size for b2745d17-1972-4738-afa9-<wbr>22e9597fa787<br></div><div class="gmail_extra"><br clear="all"><div><div class="m_-7491372376951046077gmail_signature" data-smartmail="gmail_signature"><br>-ben<br></div></div><div><div class="h5">
<br><div class="gmail_quote">On Fri, May 12, 2017 at 11:46 PM, Benjamin Kingston <span dir="ltr"><<a href="mailto:ben@nexusnebula.net" target="_blank">ben@nexusnebula.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr"><div><span><div><div><div><div>Hello all,<br><br></div>I'm trying to take advantage of the shard xlator, however I've found it causes a lot of issues that I hope is easily resolvable<br><br></div>1) large file operations work well (copy file from folder a to folder b<br></div>2) seek operations and list operations frequently fail (ls directory, read bytes xyz at offset 235567)<br></div></span><div>3) Another issue is samba shares through samba-vfs show all files as 4MB, I've also seen this when mounting with fuse, however nfs-ganesha reflects correct file sizes always-<br><br></div><span><div><br></div>Turning off the shard feature resolves this issue for new files created in the volume. mounted using the gluster fuse mount<br><br></span></div><span>here's my volume settings, please let me know if there's some changes I can make.<br><br></span></div></div>Volume Name: storage2<br>Type: Distributed-Replicate<br>Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc<wbr>0143a8<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 3 x (2 + 1) = 9<br>Transport-type: tcp<br>Bricks:<br>Brick1: fd00:0:0:3::6:/mnt/gluster/sto<wbr>rage/brick0/glusterfs2<br>Brick2: fd00:0:0:3::8:/mnt/gluster/sto<wbr>rage/brick0/glusterfs2<br>Brick3: fd00:0:0:3::10:/mnt/gluster/st<wbr>orage/brick0/glusterfs (arbiter)<br>Brick4: fd00:0:0:3::6:/mnt/gluster/sto<wbr>rage/brick1/glusterfs2<br>Brick5: fd00:0:0:3::8:/mnt/gluster/sto<wbr>rage/brick1/glusterfs2<br>Brick6: fd00:0:0:3::10:/mnt/gluster/st<wbr>orage/brick1/glusterfs (arbiter)<br>Brick7: fd00:0:0:3::6:/mnt/gluster/sto<wbr>rage/brick2/glusterfs2<br>Brick8: fd00:0:0:3::8:/mnt/gluster/sto<wbr>rage/brick2/glusterfs2<br>Brick9: fd00:0:0:3::10:/mnt/gluster/st<wbr>orage/brick2/glusterfs (arbiter)<br>Options Reconfigured:<br>features.ctr-enabled: on<br>features.shard-block-size: 4MB<br>network.inode-lru-limit: 90000<br>features.cache-invalidation: on<br>performance.readdir-ahead: on<br>client.event-threads: 3<br>performance.cache-ima-xattrs: on<br>cluster.data-self-heal-algorit<wbr>hm: diff<br>network.remote-dio: disable<br>cluster.use-compound-fops: on<br>cluster.read-freq-threshold: 2<br>cluster.write-freq-threshold: 2<br>features.record-counters: on<br>disperse.shd-max-threads: 4<br>performance.parallel-readdir: on<br>performance.client-io-threads: on<br>server.event-threads: 3<br>cluster.lookup-optimize: on<br>performance.open-behind: on<br>performance.stat-prefetch: on<br>performance.quick-read: off<br>performance.io-cache: on<br>performance.read-ahead: off<br>performance.write-behind: on<br>features.scrub: Active<br>features.bitrot: on<br>features.leases: on<br>features.shard: off<br>transport.address-family: inet6<br>nfs.disable: on<br>server.allow-insecure: on<br>cluster.shd-max-threads: 8<br>performance.low-prio-threads: 32<br>cluster.locking-scheme: granular<br>cluster.shd-wait-qlength: 10000<br>user.cifs: off<br>cluster.eager-lock: enable<br>cluster.quorum-type: auto<br>cluster.server-quorum-type: server<br>cluster.tier-compact: on<br>storage.linux-aio: on<br>transport.keepalive: on<br>performance.write-behind-windo<wbr>w-size: 2GB<br>performance.flush-behind: on<br>performance.cache-size: 1GB<br>cluster.choose-local: on<br>performance.io-thread-count: 64<br>cluster.brick-multiplex: off<br>cluster.enable-shared-storage: enable<br>nfs-ganesha: enable<br></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>