[Gluster-users] dht_layout_dir_mismatch after simultaneously mkdir

Sat Dec 1 08:06:28 UTC 2012

Dear experts,

I'm using gluster 3.2.7. It believe it has good performance. That's good,
but troubles sometimes occur for mkdir.
The scenario is as below
1: A volume is created by 9 bricks on 9 gluster server.
2: Many client hosts mount it with fuse.
3: Several clients simultaneously  make one directory.
4: Except with one host, all other hosts fail to be make the directory.
(This is usual.)
5: But, problem host is found here, it becomes to be not able to make
directory and create file in the directory of step 1.

At the time, in problem host I found error message:

mkdir: cannot create directory `/gluster/test/x': Invalid argument.

touch  /gluster/test/x
touch: cannot touch `/gluster/test/x': No such file or directory

then I can find some warning log in /var/log/gluster/[logs] as following:

[2012-11-29 19:36:50.52787] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-1; inode layout - 0 - 0; disk layout - 477218588 -
954437175
[2012-11-29 19:36:50.52824] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.52873] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-7; inode layout - 0 - 0; disk layout - 3340530116 -
3817748703
[2012-11-29 19:36:50.52886] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.52901] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-2; inode layout - 0 - 0; disk layout - 954437176 -
1431655763
[2012-11-29 19:36:50.52917] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.52936] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-5; inode layout - 0 - 0; disk layout - 2386092940 -
2863311527
[2012-11-29 19:36:50.52947] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.52961] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-3; inode layout - 0 - 0; disk layout - 1431655764 -
1908874351
[2012-11-29 19:36:50.52970] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.52983] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-0; inode layout - 0 - 0; disk layout - 0 - 477218587
[2012-11-29 19:36:50.52993] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.53007] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-6; inode layout - 0 - 0; disk layout - 2863311528 -
3340530115
[2012-11-29 19:36:50.53016] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.53029] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-8; inode layout - 0 - 0; disk layout - 3817748704 -
4294967295
[2012-11-29 19:36:50.53038] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.53052] I
[dht-layout.c:682:dht_layout_dir_mismatch] 0-vol18-dht: subvol:
vol18-client-4; inode layout - 0 - 0; disk layout - 1908874352 -
2386092939
[2012-11-29 19:36:50.53060] I [dht-common.c:525:dht_revalidate_cbk]
0-vol18-dht: mismatching layouts for /test/x
[2012-11-29 19:36:50.53923] I [dht-layout.c:192:dht_layout_search]
0-vol18-dht: no subvolume for hash (value) = 3127134579
[2012-11-29 19:36:50.54422] I [dht-layout.c:192:dht_layout_search]
0-vol18-dht: no subvolume for hash (value) = 3127134579
[2012-11-29 19:36:50.54442] W [fuse-bridge.c:231:fuse_entry_cbk]
0-glusterfs-fuse: 127332: MKDIR() /test/x => -1 (Invalid argument)

So I tried to check the DHT with getfattr command in problem host.

[host1]$ sudo getfattr -m . -n trusted.glusterfs.pathinfo  /gluster/test
getfattr: Removing leading '/' from absolute path names
# file: gluster/test
trusted.glusterfs.pathinfo="(vol18-dht-layout (vol18-client-7 0 0)
(vol18-client-8 0 0) (vol18-client-4 0 0) (vol18-client-0 0 0)
(vol18-client-6 0 0) (vol18-client-1 0 0) (vol18-client-2 0 0)
(vol18-client-3 0 0) (vol18-client-5 0 0))"

I seems there is incorrect table.

If it's checked in no problem host, below tables are displayed.

[host2]$ sudo getfattr -m . -n trusted.glusterfs.pathinfo  /gluster/test
getfattr: Removing leading '/' from absolute path names
# file: gluster/test
trusted.glusterfs.pathinfo="(vol18-dht-layout (vol18-client-0 0
477218587) (vol18-client-1 477218588 954437175) (vol18-client-2
954437176 1431655763) (vol18-client-3 1431655764 1908874351)
(vol18-client-4 1908874352 2386092939) (vol18-client-5 2386092940
2863311527) (vol18-client-6 2863311528 3340530115) (vol18-client-7
3340530116 3817748703) (vol18-client-8 3817748704 4294967295))"

In my experience, if re-mount is tried on problem host, this problem
disappear and it becomes to be able to make the directory.
Is this problem a bug?

Best regards,
Kondo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121201/871bb808/attachment.html>