[Gluster-users] file not found on DHT, but it exists

[Raghavendra G]

Are other operations like chown, chmod etc are succeeding? stat call is
generally received as lookup in glusterfs (lookup callback returns a stat
structure) and lookup is sent to all subvolumes, whereas calls like open are
sent only to those nodes where file is present.

On Wed, Mar 31, 2010 at 3:49 PM, Arend-Jan Wijtzes <ajwytzes at wise-guys.nl>wrote:

> Hi gluster developers,
>
> I have encountered a situation where a file can not be found,
> but it does exist and it is on the correct node. The file can
> be stat()-ed but not opened. After a Gluster restart the file
> is accessable again.
>
> Glusterfs: 3.0.3 with altered hashing function (by me).
>
> == On the Gluster mounted volume:
> archive at cgmarchive0:~/archive/incoming$ ls -l
> www.funkyfish.nl#59493#cgmspider0
> -rw-rw-r-- <http://www.funkyfish.nl#59493#cgmspider0%0A-rw-rw-r-->    1
> archive  archive    599065 Mar 30 15:16 www.funkyfish.nl#59493#cgmspider0
> archive at cgmarchive0:~/archive/incoming$ wc -l
> www.funkyfish.nl#59493#cgmspider0
> wc: www.funkyfish.nl#59493#cgmspider0: No such file or directory
>
> == On the local (node0) volume
> archive at cgmarchive0:/local.mnt/md0/glfs-data/incoming$ ls -l
> www.funkyfish.nl#59493#cgmspider0
> -rw-rw-r-- <http://www.funkyfish.nl#59493#cgmspider0%0A-rw-rw-r-->    1
> vagabond vagabondo   599065 Mar 30 15:16 www.funkyfish.nl#59493#cgmspider0
> archive at cgmarchive0:/local.mnt/md0/glfs-data/incoming$ wc -l
> www.funkyfish.nl#59493#cgmspider0
>  10767 www.funkyfish.nl#59493#cgmspider0
>
> == Error log
> [2010-03-31 12:02:47] D [dht-common.c:1590:dht_fd_cbk] dht: subvolume node0
> returned -1 (No such file or directory)
> [2010-03-31 12:02:47] W [fuse-bridge.c:858:fuse_fd_cbk] glusterfs-fuse:
> 10346982: OPEN() /incoming/www.funkyfish.nl#59493#cgmspider0 => -1 (No
> such file or directory)
>
> Then after a gluster restart (umount/mount sequence):
>
> == On the Gluster mounted volume:
> archive at cgmarchive0:~/archive/incoming$  ls -l
> www.funkyfish.nl#59493#cgmspider0
> -rw-rw-r-- <http://www.funkyfish.nl#59493#cgmspider0%0A-rw-rw-r-->    1
> archive  archive    599065 Mar 30 15:16 www.funkyfish.nl#59493#cgmspider0
> archive at cgmarchive0:~/archive/incoming$  wc -l
> www.funkyfish.nl#59493#cgmspider0
>  10767 www.funkyfish.nl#59493#cgmspider0
>
>
> The application access pattern for these files is:
> * a file is copied onto the filesystem with a temporary name
> * the file is renamed to it's final name
> * the file is read once, then deleted
> * the filename is normally not used again or at least not any time soon
>
> All file operations went through the gluster fs (no direct local access).
> The hashing function has been replaced by one that implements a 'consistent
> hashing' scheme and adapted so that the temporary filename and the final
> filename always go to the same node.
>
> The problem is not isolated to a single case, but it does take
> a long time (days) to occur. In the long term it can be reproduced
> so if you need more debugging info I can try to extract it for you.
>
> Any ideas?
>
>
> == Volume file
> volume posix
>  type storage/posix
>  option directory /local.mnt/md0/glfs-data
> end-volume
>
> volume locks
>  type features/posix-locks
>  subvolumes posix
> end-volume
>
> volume fixed-id
>  type features/filter
>  option fixed-uid 2224
>  option fixed-gid 224
>  subvolumes locks
> end-volume
>
> volume brick
>  type performance/io-threads
>  subvolumes fixed-id
> end-volume
>
> volume server
>  type protocol/server
>  option transport-type tcp
>  option auth.addr.brick.allow 10.0.0.*,10.1.0.*
>  subvolumes brick
> end-volume
>
> volume node0
>  type protocol/client
>  option transport-type tcp
>  option remote-host cgmarchive0
>  option remote-subvolume brick
> end-volume
>
> volume node1
>  type protocol/client
>  option transport-type tcp
>  option remote-host cgmarchive1
>  option remote-subvolume brick
> end-volume
>
> volume dht
>  type cluster/dht
>  subvolumes node0 node1
> end-volume
>
>
> --
> Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>



-- 
Raghavendra G