[Gluster-devel] error while copying files to volume

Tue Dec 23 21:42:26 UTC 2008

I stopped it after I saw the log messages.  When gluster started  
reporting things as errors (instead of warnings) I figured something  
was wrong.   I'll fire it back up and see how it goes.

--brian

On Dec 23, 2008, at 1:03 PM, Anand Avati wrote:

> Brian,
>  These logs related to rsync need not be worried. Is rsync actually
> completing? these log messages will be fixed soon.
>
> Thanks,
> avati
>
>> Hi,
>>
>> Thanks for your help thus far.  I'm continuing my testing and have  
>> run
>> across some errors in my logfiles.   I'm using rsync from a local  
>> directory
>> to the gluster volume to load up some test data.  This is the only
>> processing modifying the gluster filesystem.  Occasionally, I'll see
>> self-heals triggered while the the rsync is running.  With the  
>> exception of
>> this one, they all finish without any errors reported.
>>
>> I thought it was strange that selfheals were being triggered since  
>> all of
>> the servers are up and rsync is the only process modifying the  
>> gluster
>> filesystem, but since it seemed harmless I ignored them.   This  
>> time it
>> seemed to be more of an error.  While rsync is copying files, it  
>> uses a
>> different filename than the file that's actually being copied, and  
>> once the
>> copy is complete, the file is renamed to it's original name.    
>> There seems
>> to have been a race condition here where a self-heal is triggered  
>> and during
>> that time, the file is renamed.   The logs on brick1 and brick3 seem
>> consistent with this theory.   The failed locks seem to make sense  
>> also with
>> the error "No such file or directory"  (the old name is no longer  
>> in use).
>> I see 2 potential problems here.   The first is that self-heals are
>> occasionally being triggered during a copy when everything is up and
>> functioning properly.   The 2nd issue seems to be that during a  
>> self-heal,
>> there is a race condition where a file can be renamed and causes  
>> these
>> errors below.
>>
>> Running 1.4RC3, logs and volfiles are below.  Nothing of mention  
>> was in
>> glusterfs.log except on brick1.  Please let me know what i can do  
>> to help.
>>
>> server x.x.x.102 (brick1)
>> ==> glusterfsd.log <==
>> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
>> ns: paths
>> differ for inode(19824657): client path =
>> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
>> /mobygames/htdocs/images/i/22/23/139623.png
>> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
>> ns: paths
>> differ for inode(19824657): client path =
>> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
>> /mobygames/htdocs/images/i/22/23/139623.png
>> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
>> ns: paths
>> differ for inode(19824657): client path =
>> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
>> /mobygames/htdocs/images/i/22/23/139623.png
>> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
>> ns: paths
>> differ for inode(19824657): client path =
>> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
>> /mobygames/htdocs/images/i/22/23/139623.png
>>
>> ==> glusterfs.log <==
>> 2008-12-18 11:16:20 W [afr-self-heal-common.c:1005:afr_self_heal]  
>> afr-ns:
>> performing self heal on /mobygames/htdocs/images/i/ 
>> 22/23/.139623.png.S254kc
>> (metadata=1 data=1 entry=1)
>> 2008-12-18 11:16:20 E [afr-self-heal-metadata.c: 
>> 686:afr_sh_metadata_lk_cbk]
>> afr-ns: locking of /mobygames/htdocs/images/i/ 
>> 22/23/.139623.png.S254kc on
>> child 3 failed: No such file or directory
>> 2008-12-18 11:16:20 E [afr-self-heal-metadata.c: 
>> 686:afr_sh_metadata_lk_cbk]
>> afr-ns: locking of /mobygames/htdocs/images/i/ 
>> 22/23/.139623.png.S254kc on
>> child 1 failed: No such file or directory
>> 2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk]
>> afr-ns: locking of /mobygames/htdocs/images/i/ 
>> 22/23/.139623.png.S254kc on
>> child 3 failed: No such file or directory
>> 2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk]
>> afr-ns: locking of /mobygames/htdocs/images/i/ 
>> 22/23/.139623.png.S254kc on
>> child 1 failed: No such file or directory
>> 2008-12-18 11:16:20 E [afr-self-heal-data.c: 
>> 182:afr_sh_data_unlck_cbk]
>> afr-ns: locking inode of /mobygames/htdocs/images/i/ 
>> 22/23/.139623.png.S254kc
>> on child 1 failed: No such file or directory
>> 2008-12-18 11:16:20 E [afr-self-heal-data.c: 
>> 182:afr_sh_data_unlck_cbk]
>> afr-ns: locking inode of /mobygames/htdocs/images/i/ 
>> 22/23/.139623.png.S254kc
>> on child 3 failed: No such file or directory
>>
>>
>>
>> server x.x.x.106 (brick2)
>> ==> glusterfsd.log <==
>> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
>> server:
>> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
>> brick-ns
>> returning error: -1 (2)
>> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
>> server:
>> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
>> brick-ns
>> returning error: -1 (2)
>> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
>> server:
>> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
>> brick-ns
>> returning error: -1 (2)
>> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
>> server:
>> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
>> brick-ns
>> returning error: -1 (2)
>>
>>
>> server x.x.x107 (brick3)
>> ==> glusterfsd.log <==
>> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
>> ns: paths
>> differ for inode(5570577): client path =
>> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
>> /mobygames/htdocs/images/i/22/23/139623.png
>> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
>> ns: paths
>> differ for inode(5570577): client path =
>> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
>> /mobygames/htdocs/images/i/22/23/139623.png
>> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
>> ns: paths
>> differ for inode(5570577): client path =
>> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
>> /mobygames/htdocs/images/i/22/23/139623.png
>> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
>> ns: paths
>> differ for inode(5570577): client path =
>> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
>> /mobygames/htdocs/images/i/22/23/139623.png
>>
>> server x.x.x.103 (brick4)
>> ==> glusterfsd.log <==
>> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
>> server:
>> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
>> brick-ns
>> returning error: -1 (2)
>> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
>> server:
>> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
>> brick-ns
>> returning error: -1 (2)
>> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
>> server:
>> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
>> brick-ns
>> returning error: -1 (2)
>> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
>> server:
>> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
>> brick-ns
>> returning error: -1 (2)
>>
>>
>> VOLFILES
>>
>> ****Server****
>>
>> volume brick-store
>> type storage/posix
>> option directory /moby/gluster/
>> end-volume
>>
>> volume brick
>> type features/locks
>> subvolumes brick-store
>> end-volume
>>
>> volume brick-store-ns
>> type storage/posix
>> option directory /moby/gluster-ns/
>> end-volume
>>
>> volume brick-ns
>> type features/locks
>> subvolumes brick-store-ns
>> end-volume
>>
>> volume server
>> type protocol/server
>> option transport-type tcp/server
>> subvolumes brick brick-ns
>> option auth.addr.brick.allow x.x.x.* # Allow access to "brick" volume
>> option auth.addr.brick-ns.allow x.x.x.* # Allow access to "brick- 
>> ns" volume
>> end-volume
>>
>>
>> ****Client****
>>
>> volume brick1
>> type protocol/client
>> option transport-type tcp/client
>> option remote-host x.x.x.102
>> option remote-subvolume brick
>> end-volume
>>
>> volume brick2
>> type protocol/client
>> option transport-type tcp/client
>> option remote-host x.x.x.106
>> option remote-subvolume brick
>> end-volume
>>
>> volume brick3
>> type protocol/client
>> option transport-type tcp/client
>> option remote-host x.x.x.107
>> option remote-subvolume brick
>> end-volume
>>
>> volume brick4
>> type protocol/client
>> option transport-type tcp/client
>> option remote-host x.x.x.103
>> option remote-subvolume brick
>> end-volume
>>
>> volume brick-ns1
>> type protocol/client
>> option transport-type tcp/client
>> option remote-host x.x.x.102
>> option remote-subvolume brick-ns
>> end-volume
>>
>> volume brick-ns2
>> type protocol/client
>> option transport-type tcp/client
>> option remote-host x.x.x.106
>> option remote-subvolume brick-ns
>> end-volume
>>
>> volume brick-ns3
>> type protocol/client
>> option transport-type tcp/client
>> option remote-host x.x.x.107
>> option remote-subvolume brick-ns
>> end-volume
>>
>> volume brick-ns4
>> type protocol/client
>> option transport-type tcp/client
>> option remote-host x.x.x.103
>> option remote-subvolume brick-ns
>> end-volume
>>
>> volume afr1
>> type cluster/afr
>> subvolumes brick1 brick3
>> end-volume
>>
>> volume afr2
>> type cluster/afr
>> subvolumes brick2 brick4
>> end-volume
>>
>> volume afr-ns
>> type cluster/afr
>> subvolumes brick-ns1 brick-ns2 brick-ns3 brick-ns4
>> end-volume
>>
>> volume unify
>> type cluster/unify
>> option namespace afr-ns
>> option scheduler rr
>> subvolumes afr1 afr2
>> end-volume
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>