<div dir="ltr"><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 29, 2020 at 1:28 PM jifeng-call <<a href="mailto:17607319886@163.com">17607319886@163.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:Arial"><div><p style="margin:0px"><span style="font-family:arial;white-space:pre-wrap">Hi All,</span></p></div><div>I have 6 servers that form a glusterfs 2x3 distributed replication volume, the details are as follows:</div><div><br></div><div><div>[root@node1 ~]# gluster volume info</div><div>Volume Name: ksvd_vol</div><div>Type: Distributed-Replicate</div><div>Volume ID: c9848daa-b06f-4f82-a2f8-1b425b8e869c</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 2 x 3 = 6</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: node1:/export/kylin/ksvd</div><div>Brick2: node2:/export/kylin/ksvd</div><div>Brick3: node3:/export/kylin/ksvd</div><div>Brick4: node4:/export/kylin/ksvd</div><div>Brick5: node5:/export/kylin/ksvd</div><div>Brick6: node6:/export/kylin/ksvd</div><div>Options Reconfigured:</div><div>diagnostics.client-log-level: DEBUG</div><div>diagnostics.brick-log-level: DEBUG</div><div>performance.client-io-threads: off</div><div>nfs.disable: on</div><div>transport.address-family: inet</div><div>cluster.quorum-type: auto</div></div><div><br></div><div>Restart 6 servers at a time, the server automatically mounts the ksvd_vol volume to the / home / kylin-ksvd directory during the startup process：</div><div><br></div><div><div>[root@node1 ~]# df -h | grep ksvd_vol</div><div>node1:/ksvd_vol 4.5T 362G 4.1T 9% /home/kylin-ksvd</div></div><div><br></div><div>Failed to create a file in the glusterfs mount directory of node3, the log shows the error as follows (the files created by other servers are normal)：</div><div><br></div></div></blockquote><div>Possibly the layout setxattr failed which should not have happened given it is a replicated volume. You can check log on client side and brick side for the parent directory for more information.</div><div>Post remount, a fresh triggers which heals the layout missing before and create goes in.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:Arial"><div></div><div><div>[2020-05-29 05:55:03.065656] E [MSGID: 109011] [dht-common.c:8683:dht_create] 0-ksvd_vol-dht: no subvolume in layout for path=/.ksvd.time.405fa8a6-575d-4474-a97f-a107cbf1c673.18512</div><div>[2020-05-29 05:55:03.065719] W [fuse-bridge.c:2122:fuse_create_cbk] 0-glusterfs-fuse: 2454790: /.ksvd.time.405fa8a6-575d-4474-a97f-a107cbf1c673.18512 => -1 (输入/输出错误)</div><div>[2020-05-29 05:55:03.680303] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696</div><div>[2020-05-29 05:55:04.687456] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 491618322</div><div>[2020-05-29 05:55:04.688612] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696</div><div>[2020-05-29 05:55:05.694446] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696</div><div>[2020-05-29 05:55:05.830555] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 491618322</div><div>[2020-05-29 05:55:06.700423] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696</div><div>[2020-05-29 05:55:07.706536] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696</div><div>[2020-05-29 05:55:07.833049] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 491618322</div><div>[2020-05-29 05:55:08.712128] W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696</div><div>The message "W [MSGID: 109011] [dht-layout.c:186:dht_layout_search] 0-ksvd_vol-dht: no subvolume for hash (value) = 1738969696" repeated 2 times between [2020-05-29 05:55:08.712128] and [2020-05-29 05:55:09.718541]</div></div><div><br></div><div>The pathinfo information in the / home / kylin-ksvd directory is displayed as follows:</div><div><br></div><div><div>[root@node3 kylin-ksvd]# getfattr -d -m . -n trusted.glusterfs.pathinfo /home/kylin-ksvd/</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: home/kylin-ksvd/</div><div>trusted.glusterfs.pathinfo="((<DISTRIBUTE:ksvd_vol-dht> (<REPLICATE:ksvd_vol-replicate-0> <POSIX(/export/kylin/ksvd):node2:/export/kylin/ksvd/> <POSIX(/export/kylin/ksvd):node3:/export/kylin/ksvd/> <POSIX(/export/kylin/ksvd):node1:/export/kylin/ksvd/>) (<REPLICATE:ksvd_vol-replicate-1> <POSIX(/export/kylin/ksvd):node5:/export/kylin/ksvd/> <POSIX(/export/kylin/ksvd):node6:/export/kylin/ksvd/> <POSIX(/export/kylin/ksvd):node4:/export/kylin/ksvd/>)) (ksvd_vol-dht-layout (ksvd_vol-replicate-0 0 0) (ksvd_vol-replicate-1 3539976838 4294967295)))"</div></div><div><br></div><div>It can be seen from the above information that ksvd_vol-dht-layout is missing the interval 0 to 3539976837</div><div><br></div><div>After umount, remounting returned to normal ... What is the reason?</div><div><br></div><div><pre>Best regards</pre></div><div><br></div></div><br><br><span title="neteasefooter"><p><br> </p></span>________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div></div>