[Gluster-devel] error while copying files to volume

Anand Avati avati at zresearch.com
Tue Dec 23 20:03:51 UTC 2008 

Brian,
  These logs related to rsync need not be worried. Is rsync actually
completing? these log messages will be fixed soon.

Thanks,
avati

> Hi,
>
> Thanks for your help thus far.  I'm continuing my testing and have run
> across some errors in my logfiles.   I'm using rsync from a local directory
> to the gluster volume to load up some test data.  This is the only
> processing modifying the gluster filesystem.  Occasionally, I'll see
> self-heals triggered while the the rsync is running.  With the exception of
> this one, they all finish without any errors reported.
>
> I thought it was strange that selfheals were being triggered since all of
> the servers are up and rsync is the only process modifying the gluster
> filesystem, but since it seemed harmless I ignored them.   This time it
> seemed to be more of an error.  While rsync is copying files, it uses a
> different filename than the file that's actually being copied, and once the
> copy is complete, the file is renamed to it's original name.   There seems
> to have been a race condition here where a self-heal is triggered and during
> that time, the file is renamed.   The logs on brick1 and brick3 seem
> consistent with this theory.   The failed locks seem to make sense also with
> the error "No such file or directory"  (the old name is no longer in use).
> I see 2 potential problems here.   The first is that self-heals are
> occasionally being triggered during a copy when everything is up and
> functioning properly.   The 2nd issue seems to be that during a self-heal,
> there is a race condition where a file can be renamed and causes these
> errors below.
>
> Running 1.4RC3, logs and volfiles are below.  Nothing of mention was in
> glusterfs.log except on brick1.  Please let me know what i can do to help.
>
> server x.x.x.102 (brick1)
> ==> glusterfsd.log <==
> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths
> differ for inode(19824657): client path =
> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> /mobygames/htdocs/images/i/22/23/139623.png
> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths
> differ for inode(19824657): client path =
> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> /mobygames/htdocs/images/i/22/23/139623.png
> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths
> differ for inode(19824657): client path =
> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> /mobygames/htdocs/images/i/22/23/139623.png
> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths
> differ for inode(19824657): client path =
> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> /mobygames/htdocs/images/i/22/23/139623.png
>
> ==> glusterfs.log <==
> 2008-12-18 11:16:20 W [afr-self-heal-common.c:1005:afr_self_heal] afr-ns:
> performing self heal on /mobygames/htdocs/images/i/22/23/.139623.png.S254kc
> (metadata=1 data=1 entry=1)
> 2008-12-18 11:16:20 E [afr-self-heal-metadata.c:686:afr_sh_metadata_lk_cbk]
> afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc on
> child 3 failed: No such file or directory
> 2008-12-18 11:16:20 E [afr-self-heal-metadata.c:686:afr_sh_metadata_lk_cbk]
> afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc on
> child 1 failed: No such file or directory
> 2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk]
> afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc on
> child 3 failed: No such file or directory
> 2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk]
> afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc on
> child 1 failed: No such file or directory
> 2008-12-18 11:16:20 E [afr-self-heal-data.c:182:afr_sh_data_unlck_cbk]
> afr-ns: locking inode of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc
> on child 1 failed: No such file or directory
> 2008-12-18 11:16:20 E [afr-self-heal-data.c:182:afr_sh_data_unlck_cbk]
> afr-ns: locking inode of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc
> on child 3 failed: No such file or directory
>
>
>
> server x.x.x.106 (brick2)
> ==> glusterfsd.log <==
> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server:
> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns
> returning error: -1 (2)
> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server:
> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns
> returning error: -1 (2)
> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server:
> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns
> returning error: -1 (2)
> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server:
> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns
> returning error: -1 (2)
>
>
> server x.x.x107 (brick3)
> ==> glusterfsd.log <==
> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths
> differ for inode(5570577): client path =
> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> /mobygames/htdocs/images/i/22/23/139623.png
> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths
> differ for inode(5570577): client path =
> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> /mobygames/htdocs/images/i/22/23/139623.png
> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths
> differ for inode(5570577): client path =
> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> /mobygames/htdocs/images/i/22/23/139623.png
> 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths
> differ for inode(5570577): client path =
> /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> /mobygames/htdocs/images/i/22/23/139623.png
>
> server x.x.x.103 (brick4)
> ==> glusterfsd.log <==
> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server:
> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns
> returning error: -1 (2)
> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server:
> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns
> returning error: -1 (2)
> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server:
> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns
> returning error: -1 (2)
> 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server:
> INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns
> returning error: -1 (2)
>
>
> VOLFILES
>
> ****Server****
>
> volume brick-store
> type storage/posix
> option directory /moby/gluster/
> end-volume
>
> volume brick
> type features/locks
> subvolumes brick-store
> end-volume
>
> volume brick-store-ns
> type storage/posix
> option directory /moby/gluster-ns/
> end-volume
>
> volume brick-ns
> type features/locks
> subvolumes brick-store-ns
> end-volume
>
> volume server
> type protocol/server
> option transport-type tcp/server
> subvolumes brick brick-ns
> option auth.addr.brick.allow x.x.x.* # Allow access to "brick" volume
> option auth.addr.brick-ns.allow x.x.x.* # Allow access to "brick-ns" volume
> end-volume
>
>
> ****Client****
>
> volume brick1
> type protocol/client
> option transport-type tcp/client
> option remote-host x.x.x.102
> option remote-subvolume brick
> end-volume
>
> volume brick2
> type protocol/client
> option transport-type tcp/client
> option remote-host x.x.x.106
> option remote-subvolume brick
> end-volume
>
> volume brick3
> type protocol/client
> option transport-type tcp/client
> option remote-host x.x.x.107
> option remote-subvolume brick
> end-volume
>
> volume brick4
> type protocol/client
> option transport-type tcp/client
> option remote-host x.x.x.103
> option remote-subvolume brick
> end-volume
>
> volume brick-ns1
> type protocol/client
> option transport-type tcp/client
> option remote-host x.x.x.102
> option remote-subvolume brick-ns
> end-volume
>
> volume brick-ns2
> type protocol/client
> option transport-type tcp/client
> option remote-host x.x.x.106
> option remote-subvolume brick-ns
> end-volume
>
> volume brick-ns3
> type protocol/client
> option transport-type tcp/client
> option remote-host x.x.x.107
> option remote-subvolume brick-ns
> end-volume
>
> volume brick-ns4
> type protocol/client
> option transport-type tcp/client
> option remote-host x.x.x.103
> option remote-subvolume brick-ns
> end-volume
>
> volume afr1
> type cluster/afr
> subvolumes brick1 brick3
> end-volume
>
> volume afr2
> type cluster/afr
> subvolumes brick2 brick4
> end-volume
>
> volume afr-ns
> type cluster/afr
> subvolumes brick-ns1 brick-ns2 brick-ns3 brick-ns4
> end-volume
>
> volume unify
> type cluster/unify
> option namespace afr-ns
> option scheduler rr
> subvolumes afr1 afr2
> end-volume
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>

More information about the Gluster-devel mailing list