[Gluster-users] Reliability issues with Gluster 3.10 and shard

[Nithya Balachandran]

On 15 May 2017 at 11:01, Benjamin Kingston <ben at nexusnebula.net> wrote:

> I resolved this with the following settings, particularly disabling
> features.ctr-enabled
>

That's odd. CTR should be enabled for tiered volumes. Was it enabled by
default?



>
> olume Name: storage2
> Type: Distributed-Replicate
> Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 3 x (2 + 1) = 9
> Transport-type: tcp
> Bricks:
> Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2
> Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2
> Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter)
> Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2
> Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2
> Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter)
> Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2
> Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2
> Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter)
> Options Reconfigured:
> performance.write-behind-window-size: 4MB
> performance.cache-invalidation: on
> transport.keepalive: on
> performance.write-behind: on
> performance.read-ahead: on
> performance.io-cache: on
> performance.stat-prefetch: on
> performance.open-behind: on
> cluster.use-compound-fops: on
> performance.cache-ima-xattrs: on
> features.cache-invalidation: on
> client.event-threads: 4
> cluster.data-self-heal-algorithm: full
> performance.client-io-threads: on
> server.event-threads: 4
> performance.quick-read: on
> features.scrub: Active
> features.bitrot: on
> features.shard: on
> transport.address-family: inet6
> nfs.disable: on
> server.allow-insecure: on
> user.cifs: off
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.tier-compact: on
> diagnostics.brick-log-level: WARNING
> diagnostics.client-log-level: WARNING
> cluster.self-heal-daemon: enable
> performance.cache-samba-metadata: on
> cluster.brick-multiplex: off
> cluster.enable-shared-storage: enable
> nfs-ganesha: enable
>
>
>
> -ben
>
> On Sat, May 13, 2017 at 12:20 PM, Benjamin Kingston <ben at nexusnebula.net>
> wrote:
>
>> Hers's some log entries from nfs-ganesha gfapi
>>
>> [2017-05-13 19:02:54.105936] E [MSGID: 133010]
>> [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup
>> on shard 11 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
>> [Input/output error]
>> [2017-05-13 19:02:54.106176] E [MSGID: 133010]
>> [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup
>> on shard 2 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
>> [Input/output error]
>> [2017-05-13 19:02:54.106288] E [MSGID: 133010]
>> [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup
>> on shard 1 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
>> [Input/output error]
>> [2017-05-13 19:02:54.384922] I [MSGID: 108026]
>> [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
>> 0-storage2-replicate-2: performing metadata selfheal on
>> fe651475-226e-42a3-be2d-751d4f58e383
>> [2017-05-13 19:02:54.385894] W [MSGID: 114031]
>> [client-rpc-fops.c:2258:client3_3_setattr_cbk] 0-storage2-client-8:
>> remote operation failed [Operation not permitted]
>> [2017-05-13 19:02:54.401187] I [MSGID: 108026]
>> [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2:
>> Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383.
>> sources=[0] 1  sinks=
>> [2017-05-13 19:02:57.830019] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2
>> (hash=storage2-readdir-ahead-0/cache=<nul>)
>>
>> [2017-05-13 19:08:22.014899] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp
>> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr
>> (hash=storage2-readdir-ahead-1/cache=<nul>)
>> [2017-05-13 19:08:22.463840] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04
>> (hash=storage2-readdir-ahead-0/cache=<nul>)
>> [2017-05-13 19:08:22.769542] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01
>> (hash=storage2-readdir-ahead-0/cache=<nul>)
>> [2017-05-13 19:08:23.141069] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo.tmp
>> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo
>> (hash=storage2-readdir-ahead-0/cache=<nul>)
>> [2017-05-13 19:08:23.468554] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00.tmp
>> (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00
>> (hash=storage2-readdir-ahead-2/cache=<nul>)
>> [2017-05-13 19:08:23.671753] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv
>> (hash=storage2-readdir-ahead-2/cache=<nul>)
>> [2017-05-13 19:08:23.812152] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11.tmp
>> (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11
>> (hash=storage2-readdir-ahead-1/cache=<nul>)
>> [2017-05-13 19:08:24.244584] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12.tmp
>> (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12
>> (hash=storage2-readdir-ahead-1/cache=<nul>)
>> [2017-05-13 19:08:24.711781] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09.tmp
>> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r09
>> (hash=storage2-readdir-ahead-1/cache=<nul>)
>> [2017-05-13 19:08:25.029102] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06.tmp
>> (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r06
>> (hash=storage2-readdir-ahead-0/cache=<nul>)
>> [2017-05-13 19:08:25.206011] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05.tmp
>> (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r05
>> (hash=storage2-readdir-ahead-0/cache=<nul>)
>> [2017-05-13 19:08:25.467312] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03.tmp
>> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r03
>> (hash=storage2-readdir-ahead-1/cache=<nul>)
>> [2017-05-13 19:08:25.726208] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07.tmp
>> (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r07
>> (hash=storage2-readdir-ahead-0/cache=<nul>)
>> [2017-05-13 19:08:25.860386] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar.tmp
>> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.rar
>> (hash=storage2-readdir-ahead-1/cache=<nul>)
>> [2017-05-13 19:09:00.803859] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r08
>> (hash=storage2-readdir-ahead-0/cache=<nul>)
>> [2017-05-13 19:09:01.826525] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r02
>> (hash=storage2-readdir-ahead-2/cache=<nul>)
>> [2017-05-13 19:09:02.531354] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Videos/Cinema/Bambi (1942)/.Bambi (1942).mkv.hp9oW8
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Videos/Cinema/Bambi (1942)/.nfsa223c532d51b3c7700000006
>> (hash=storage2-readdir-ahead-2/cache=<nul>)
>> [2017-05-13 19:09:39.695354] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r10
>> (hash=storage2-readdir-ahead-2/cache=<nul>)
>> [2017-05-13 19:10:59.938286] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11
>> (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set.
>> [2017-05-13 19:10:59.938510] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11
>> (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set.
>> [2017-05-13 19:10:59.939004] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11
>> (7dcd980b-4f8b-4e0c-8fb9-1732d24f30f7): trying duplicate remote fd set.
>> [2017-05-13 19:10:59.939246] E [shard.c:426:shard_modify_size_and_block_count]
>> (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0)
>> [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1
>> /xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e]
>> -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b)
>> [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get
>> trusted.glusterfs.shard.file-size for 7dcd980b-4f8b-4e0c-8fb9-1732d2
>> 4f30f7
>> [2017-05-13 19:11:01.884482] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12
>> (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set.
>> [2017-05-13 19:11:01.884482] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12
>> (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set.
>> [2017-05-13 19:11:01.884557] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r12
>> (2b3e3ab2-ba6a-4ad3-af25-fb0205f5d99e): trying duplicate remote fd set.
>> [2017-05-13 19:11:01.884918] E [shard.c:426:shard_modify_size_and_block_count]
>> (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0)
>> [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1
>> /xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e]
>> -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b)
>> [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get
>> trusted.glusterfs.shard.file-size for 2b3e3ab2-ba6a-4ad3-af25-fb0205
>> f5d99e
>> [2017-05-13 19:11:05.136517] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-0:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00
>> (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set.
>> [2017-05-13 19:11:05.136518] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-1:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00
>> (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set.
>> [2017-05-13 19:11:05.136648] I [MSGID: 114024]
>> [client-helpers.c:99:this_fd_set_ctx] 0-storage2-client-6:
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00
>> (b2745d17-1972-4738-afa9-22e9597fa787): trying duplicate remote fd set.
>> [2017-05-13 19:11:05.136871] E [shard.c:426:shard_modify_size_and_block_count]
>> (-->/usr/lib64/glusterfs/3.10.1/xlator/cluster/distribute.so(+0x65df0)
>> [0x7f8c49869df0] -->/usr/lib64/glusterfs/3.10.1
>> /xlator/features/shard.so(+0xba6e) [0x7f8c495eca6e]
>> -->/usr/lib64/glusterfs/3.10.1/xlator/features/shard.so(+0xb29b)
>> [0x7f8c495ec29b] ) 0-storage2-shard: Failed to get
>> trusted.glusterfs.shard.file-size for b2745d17-1972-4738-afa9-22e959
>> 7fa787
>>
>>
>> -ben
>>
>> On Fri, May 12, 2017 at 11:46 PM, Benjamin Kingston <ben at nexusnebula.net>
>> wrote:
>>
>>>
>>> Hello all,
>>>
>>> I'm trying to take advantage of the shard xlator, however I've found it
>>> causes a lot of issues that I hope is easily resolvable
>>>
>>> 1) large file operations work well (copy file from folder a to folder b
>>> 2) seek operations and list operations frequently fail (ls directory,
>>> read bytes xyz at offset 235567)
>>> 3) Another issue is samba shares through samba-vfs show all files as
>>> 4MB, I've also seen this when mounting with fuse, however nfs-ganesha
>>> reflects correct file sizes always-
>>>
>>>
>>> Turning off the shard feature resolves this issue for new files created
>>> in the volume. mounted using the gluster fuse mount
>>>
>>> here's my volume settings, please let me know if there's some changes I
>>> can make.
>>>
>>> Volume Name: storage2
>>> Type: Distributed-Replicate
>>> Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 3 x (2 + 1) = 9
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2
>>> Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2
>>> Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter)
>>> Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2
>>> Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2
>>> Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter)
>>> Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2
>>> Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2
>>> Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter)
>>> Options Reconfigured:
>>> features.ctr-enabled: on
>>> features.shard-block-size: 4MB
>>> network.inode-lru-limit: 90000
>>> features.cache-invalidation: on
>>> performance.readdir-ahead: on
>>> client.event-threads: 3
>>> performance.cache-ima-xattrs: on
>>> cluster.data-self-heal-algorithm: diff
>>> network.remote-dio: disable
>>> cluster.use-compound-fops: on
>>> cluster.read-freq-threshold: 2
>>> cluster.write-freq-threshold: 2
>>> features.record-counters: on
>>> disperse.shd-max-threads: 4
>>> performance.parallel-readdir: on
>>> performance.client-io-threads: on
>>> server.event-threads: 3
>>> cluster.lookup-optimize: on
>>> performance.open-behind: on
>>> performance.stat-prefetch: on
>>> performance.quick-read: off
>>> performance.io-cache: on
>>> performance.read-ahead: off
>>> performance.write-behind: on
>>> features.scrub: Active
>>> features.bitrot: on
>>> features.leases: on
>>> features.shard: off
>>> transport.address-family: inet6
>>> nfs.disable: on
>>> server.allow-insecure: on
>>> cluster.shd-max-threads: 8
>>> performance.low-prio-threads: 32
>>> cluster.locking-scheme: granular
>>> cluster.shd-wait-qlength: 10000
>>> user.cifs: off
>>> cluster.eager-lock: enable
>>> cluster.quorum-type: auto
>>> cluster.server-quorum-type: server
>>> cluster.tier-compact: on
>>> storage.linux-aio: on
>>> transport.keepalive: on
>>> performance.write-behind-window-size: 2GB
>>> performance.flush-behind: on
>>> performance.cache-size: 1GB
>>> cluster.choose-local: on
>>> performance.io-thread-count: 64
>>> cluster.brick-multiplex: off
>>> cluster.enable-shared-storage: enable
>>> nfs-ganesha: enable
>>>
>>
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170515/bc48d415/attachment.html>