[Gluster-devel] error while copying files to volume

Brian Hirt bhirt at mobygames.com
Tue Dec 23 17:09:25 UTC 2008


*i'm sending this message a send time, just in case the first one got  
lost.   If there is anything I can do to help with this, please let me  
know*

Hi,

Thanks for your help thus far.  I'm continuing my testing and have run  
across some errors in my logfiles.   I'm using rsync from a local  
directory to the gluster volume to load up some test data.  This is  
the only processing modifying the gluster filesystem.  Occasionally,  
I'll see self-heals triggered while the the rsync is running.  With  
the exception of this one, they all finish without any errors reported.

I thought it was strange that selfheals were being triggered since all  
of the servers are up and rsync is the only process modifying the  
gluster filesystem, but since it seemed harmless I ignored them.    
This time it seemed to be more of an error.  While rsync is copying  
files, it uses a different filename than the file that's actually  
being copied, and once the copy is complete, the file is renamed to  
it's original name.   There seems to have been a race condition here  
where a self-heal is triggered and during that time, the file is  
renamed.   The logs on brick1 and brick3 seem consistent with this  
theory.   The failed locks seem to make sense also with the error "No  
such file or directory"  (the old name is no longer in use).   I see 2  
potential problems here.   The first is that self-heals are  
occasionally being triggered during a copy when everything is up and  
functioning properly.   The 2nd issue seems to be that during a self- 
heal, there is a race condition where a file can be renamed and causes  
these errors below.

Running 1.4RC3, logs and volfiles are below.  Nothing of mention was  
in glusterfs.log except on brick1.  Please let me know what i can do  
to help.

server x.x.x.102 (brick1)
==> glusterfsd.log <==
2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns:  
paths differ for inode(19824657): client path = /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc. dentry path = /mobygames/htdocs/ 
images/i/22/23/139623.png
2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns:  
paths differ for inode(19824657): client path = /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc. dentry path = /mobygames/htdocs/ 
images/i/22/23/139623.png
2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns:  
paths differ for inode(19824657): client path = /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc. dentry path = /mobygames/htdocs/ 
images/i/22/23/139623.png
2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns:  
paths differ for inode(19824657): client path = /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc. dentry path = /mobygames/htdocs/ 
images/i/22/23/139623.png

==> glusterfs.log <==
2008-12-18 11:16:20 W [afr-self-heal-common.c:1005:afr_self_heal] afr- 
ns: performing self heal on /mobygames/htdocs/images/i/ 
22/23/.139623.png.S254kc (metadata=1 data=1 entry=1)
2008-12-18 11:16:20 E [afr-self-heal-metadata.c: 
686:afr_sh_metadata_lk_cbk] afr-ns: locking of /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc on child 3 failed: No such file or  
directory
2008-12-18 11:16:20 E [afr-self-heal-metadata.c: 
686:afr_sh_metadata_lk_cbk] afr-ns: locking of /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc on child 1 failed: No such file or  
directory
2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk]  
afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc  
on child 3 failed: No such file or directory
2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk]  
afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc  
on child 1 failed: No such file or directory
2008-12-18 11:16:20 E [afr-self-heal-data.c:182:afr_sh_data_unlck_cbk]  
afr-ns: locking inode of /mobygames/htdocs/images/i/ 
22/23/.139623.png.S254kc on child 1 failed: No such file or directory
2008-12-18 11:16:20 E [afr-self-heal-data.c:182:afr_sh_data_unlck_cbk]  
afr-ns: locking inode of /mobygames/htdocs/images/i/ 
22/23/.139623.png.S254kc on child 3 failed: No such file or directory



server x.x.x.106 (brick2)
==> glusterfsd.log <==
2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
server: INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc)  
on brick-ns returning error: -1 (2)
2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
server: INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc)  
on brick-ns returning error: -1 (2)
2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
server: INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc)  
on brick-ns returning error: -1 (2)
2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
server: INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc)  
on brick-ns returning error: -1 (2)


server x.x.x107 (brick3)
==> glusterfsd.log <==
2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns:  
paths differ for inode(5570577): client path = /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc. dentry path = /mobygames/htdocs/ 
images/i/22/23/139623.png
2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns:  
paths differ for inode(5570577): client path = /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc. dentry path = /mobygames/htdocs/ 
images/i/22/23/139623.png
2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns:  
paths differ for inode(5570577): client path = /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc. dentry path = /mobygames/htdocs/ 
images/i/22/23/139623.png
2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns:  
paths differ for inode(5570577): client path = /mobygames/htdocs/ 
images/i/22/23/.139623.png.S254kc. dentry path = /mobygames/htdocs/ 
images/i/22/23/139623.png

server x.x.x.103 (brick4)
==> glusterfsd.log <==
2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
server: INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc)  
on brick-ns returning error: -1 (2)
2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
server: INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc)  
on brick-ns returning error: -1 (2)
2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
server: INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc)  
on brick-ns returning error: -1 (2)
2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
server: INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc)  
on brick-ns returning error: -1 (2)


VOLFILES

****Server****

volume brick-store
type storage/posix
option directory /moby/gluster/
end-volume

volume brick
type features/locks
subvolumes brick-store
end-volume

volume brick-store-ns
type storage/posix
option directory /moby/gluster-ns/
end-volume

volume brick-ns
type features/locks
subvolumes brick-store-ns
end-volume

volume server
type protocol/server
option transport-type tcp/server
subvolumes brick brick-ns
option auth.addr.brick.allow x.x.x.* # Allow access to "brick" volume
option auth.addr.brick-ns.allow x.x.x.* # Allow access to "brick-ns"  
volume
end-volume


****Client****

volume brick1
type protocol/client
option transport-type tcp/client
option remote-host x.x.x.102
option remote-subvolume brick
end-volume

volume brick2
type protocol/client
option transport-type tcp/client
option remote-host x.x.x.106
option remote-subvolume brick
end-volume

volume brick3
type protocol/client
option transport-type tcp/client
option remote-host x.x.x.107
option remote-subvolume brick
end-volume

volume brick4
type protocol/client
option transport-type tcp/client
option remote-host x.x.x.103
option remote-subvolume brick
end-volume

volume brick-ns1
type protocol/client
option transport-type tcp/client
option remote-host x.x.x.102
option remote-subvolume brick-ns
end-volume

volume brick-ns2
type protocol/client
option transport-type tcp/client
option remote-host x.x.x.106
option remote-subvolume brick-ns
end-volume

volume brick-ns3
type protocol/client
option transport-type tcp/client
option remote-host x.x.x.107
option remote-subvolume brick-ns
end-volume

volume brick-ns4
type protocol/client
option transport-type tcp/client
option remote-host x.x.x.103
option remote-subvolume brick-ns
end-volume

volume afr1
type cluster/afr
subvolumes brick1 brick3
end-volume

volume afr2
type cluster/afr
subvolumes brick2 brick4
end-volume

volume afr-ns
type cluster/afr
subvolumes brick-ns1 brick-ns2 brick-ns3 brick-ns4
end-volume

volume unify
type cluster/unify
option namespace afr-ns
option scheduler rr
subvolumes afr1 afr2
end-volume



_______________________________________________
Gluster-devel mailing list
Gluster-devel at nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel






More information about the Gluster-devel mailing list