[Gluster-devel] Confusing the clients with echo, mv and cat

[David Rotermund]

  Hi,

we tried to install glusterfs on our new cluster. 8 servers were set up, 
with a RAID5 storage volume each,  and 19 clients are accessing the 
unified glusterfs volume. Everything worked quit well but then we 
encounter the following problem:

Client A:
echo "malediction" > x
Client B:
cat x
Client A:
mv x y; echo "malediction" > x
Client B:
cat x
ls -ltr

In many cases (not every-time but with high probability), the last "cat 
x" on client B generates an access error and "ls -ltr" delivers
-????????? ? ?    ?     ?             ? x
Our guess is: The second "echo" generates the new file on a different 
server and client B is not able to recognise this (client A has no 
problem). Maybe this happens due to an unkown cache (maybe in fuse ?). 
This is strange because this error happens also with a minimal setting 
without write-behind or read-ahead (see the config file at the end). 
There should be no reason to cache something.

Is there a workaround to solve this problem?

best regards
David

Additional Information : --------------------------

The client generates with the debug-function for "ls x" on client B:
2007-09-06 16:10:48 D [inode.c:340:__active_inode] fuse/inode: 
activating inode(205783062), lru=15/1024
2007-09-06 16:10:48 D [fuse-bridge.c:389:fuse_lookup] glusterfs-fuse: 
LOOKUP 205783057/x (/davrot/x)
2007-09-06 16:10:48 D [fuse-bridge.c:367:fuse_entry_cbk] glusterfs-fuse: 
ERR => -1 (2)
2007-09-06 16:10:48 D [inode.c:370:__passive_inode] fuse/inode: 
passivating inode(205783062), lru=16/1024
2007-09-06 16:10:48 D [inode.c:340:__active_inode] fuse/inode: 
activating inode(205783062), lru=15/1024
2007-09-06 16:10:48 D [fuse-bridge.c:389:fuse_lookup] glusterfs-fuse: 
LOOKUP 205783057/x (/davrot/x)
2007-09-06 16:10:48 D [fuse-bridge.c:367:fuse_entry_cbk] glusterfs-fuse: 
ERR => -1 (2)
2007-09-06 16:10:48 D [inode.c:370:__passive_inode] fuse/inode: 
passivating inode(205783062), lru=16/1024
2007-09-06 16:10:53 D [fuse-bridge.c:507:fuse_getattr] glusterfs-fuse: 
GETATTR 205783057 (/davrot)

The computers are running on a FEDORA 7 (64 bit version) with fuse 
(2.7.0-3.fc7.x86_64) and glusterfs 1.3.1 (also the same problem with 
1.3.0).

[Client - Config - File(s) ->]
volume client1
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
 option remote-host 192.168.1.1      # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume
volume client_ns
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
 option remote-host 192.168.1.1      # IP address of the remote brick
 option remote-subvolume brick_ns        # name of the remote volume
end-volume
volume client2
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
 option remote-host 192.168.1.2      # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume

[...the analog entries for the other clients 3 - 7 ...]

volume client8
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
 option remote-host 192.168.1.8      # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume
volume bricks
 type cluster/unify
 subvolumes client1 client2 client3 client4 client5 client6 client7 client8
 option namespace client_ns
option scheduler rr
end-volume
[<- Client - Config - File(s)]

[Server - Config - File(s) ->]
volume brick
 type storage/posix                   # POSIX FS translator
 option directory /raid_0/fs        # Export this directory
end-volume
volume server
 type protocol/server
 option transport-type tcp/server  
option auth.ip.brick.allow *
end-volume
[<- Server - Config - File(s) ]