[Gluster-users] rsync to gluster mount: self-heal and bad performance

Mon Nov 16 10:00:20 UTC 2015

Hello Ben,

Thank you for your answer. I don't see the same errors when just creating a
number of files, eg. touch test{0000..9999}. Performance is not great, but
after a few minutes it finishes successfully.

I'm running rsync through lsyncd, the options are:

/usr/bin/rsync --delete --ignore-errors -zslt -r $source $destination

I'm running it over a LAN network, between two VMs. The volume is indeed
mounted with --acl, but on the directory I'm syncing to I haven't set them
explicitly:

[tiemen at iron2 test]$ getfacl stg/
# file: stg/
# owner: root
# group: rdcompany
# flags: -s-
user::rwx
group::rwx
other::r-x

Volume options:

[tiemen at iron2 test]$ sudo gluster volume info lpxassets

Volume Name: lpxassets
Type: Replicate
Volume ID: fea00430-63b1-4a4e-bc38-b74d3732acf4
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: iron2:/data/brick/lpxassets
Brick2: cobalt2:/data/brick/lpxassets
Brick3: arbiter:/data/arbiter/lpxassets
Options Reconfigured:
nfs.disable: on
performance.readdir-ahead: on
cluster.quorum-type: auto
cluster.enable-shared-storage: enable
nfs-ganesha: disable

Any other info I could provide?

On 15 November 2015 at 18:11, Ben Turner <bturner at redhat.com> wrote:

> ----- Original Message -----
> > From: "Tiemen Ruiten" <t.ruiten at rdmedia.com>
> > To: "gluster-users" <gluster-users at gluster.org>
> > Sent: Sunday, November 15, 2015 5:22:08 AM
> > Subject: Re: [Gluster-users] rsync to gluster mount: self-heal and bad
>       performance
> >
> > Any other suggestions?
>
> You are correct, rsync should not cause self heal on every file.  It makes
> me think that Ernie is correct and that something isn't correct.  If you
> just create a bunch of files out side of rsync do you see the same
> behavior?  What rsync command are you running, where are you syncing the
> data from?  I see you have the acl mount option, are you using ACLs?
>
> -b
>
>
> >
> > On 13 November 2015 at 09:56, Tiemen Ruiten < t.ruiten at rdmedia.com >
> wrote:
> >
> >
> >
> > Hello Ernie, list,
> >
> > No, that's not the case. The volume is mounted through glusterfs-fuse -
> on
> > the same server running one of the bricks. The fstab:
> >
> > # /etc/fstab
> > # Created by anaconda on Tue Aug 18 18:10:49 2015
> > #
> > # Accessible filesystems, by reference, are maintained under '/dev/disk'
> > # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more
> info
> > #
> > UUID=56778fed-bf3f-435e-8c32-edaa8c707f29 / xfs defaults 0 0
> > UUID=a44e32ed-cfbe-4ba0-896f-1efff9397ba1 /boot xfs defaults 0 0
> > UUID=a344d2bc-266d-4905-85b1-fbb7fe927659 swap swap defaults 0 0
> > /dev/vdb1 /data/brick xfs defaults 1 2
> > iron2:/lpxassets /mnt/lpxassets glusterfs _netdev,acl 0 0
> >
> >
> >
> >
> > On 12 November 2015 at 22:50, Ernie Dunbar < maillist at lightspeed.ca >
> wrote:
> >
> >
> > Hi Tiemen
> >
> > It sounds like you're trying to rsync files onto your Gluster server,
> rather
> > than to the Gluster filesystem. You want to copy these files into the
> > mounted filesystem (typically on some other system than the Gluster
> > servers), because Gluster is designed to handle it that way.
> >
> > I can't remember the nitty gritty details about why this is, but I've
> made
> > this mistake before as well. Hope that helps. :)
> >
> >
> > On 2015-11-12 11:31, Tiemen Ruiten wrote:
> >
> >
> >
> > Hello,
> >
> > While rsyncing to a directory mounted through glusterfs fuse,
> > performance is very bad and it appears every synced file generates a
> > (metadata) self-heal.
> >
> > The volume is mounted with option acl and acl's are set on a
> > subdirectory.
> >
> > Setup is as follows:
> >
> > Two Centos 7 VM's (KVM), with Gluster 3.7.6 and one physical CentOS 6
> > node, also Gluster 3.7.6. Physical node functions as arbiter. So it's
> > a replica 3 arbiter 1 volume. The bricks are LVM's with XFS
> > filesystem.
> >
> > While I don't think I should expect top performance for rsync on
> > Gluster, I wouldn't expect every file synced to trigger a self-heal.
> > Anything I can do to improve this? Should I file a bug?
> >
> > Another thing that looks related, I see a lot of these messages,
> > especially when doing IO:
> >
> > [2015-11-12 19:25:42.185904] I [dict.c:473:dict_get]
> >
> (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x121)
> > [0x7fdcc2d31161]
> >
> -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x242)
> > [0x7fdcc2b1b212] -->/lib64/libglusterfs.so.0(dict_get+0xac)
> > [0x7fdcd5e770cc] ) 0-dict: !this || key=system.posix_acl_default
> > [Invalid argument]
> >
> > --
> >
> > Tiemen Ruiten
> > Systems Engineer
> > R&D Media
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> >
> >
> > --
> > Tiemen Ruiten
> > Systems Engineer
> > R&D Media
> >
> >
> >
> > --
> > Tiemen Ruiten
> > Systems Engineer
> > R&D Media
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>

-- 
Tiemen Ruiten
Systems Engineer
R&D Media
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151116/84a00455/attachment.html>