[Gluster-users] Disastrous performance with rsync to mounted Gluster volume.

[Joe Julian]

Yes, the (kernel) nfs client caches a lot more so, as long as the 
destination hasn't changed, you should be good. If it has, it's possible 
to compare your source with stale target metadata which can, in theory, 
lead to unexpected results.

On 04/29/2015 03:20 PM, David Robinson wrote:
> Joe,
>
> I switched my mounts from FUSE to NFS and the rsync is traversing the 
> filesystem 5x faster.
>
> Previously I had:
>
> rsync -axv --numeric-ids --inplace -e 
> \"/root/hpnssh_for_backup/bin/ssh -p222 -ax -oNoneSwitch=yes 
> -oNoneEnabled=yes\" --delete --whole-file gfsib02b:/homegfs 
> /backup/backup.0
>
> where gfsib02b is one of the gluster nodes and homegfs is a FUSE 
> mounted volume.  /backup is a FUSE mount on my backup machine.
>
>
> I changed this to:
>
> rsync -axv --numeric-ids --inplace -e 
> \"/root/hpnssh_for_backup/bin/ssh -p222 -ax -oNoneSwitch=yes 
> -oNoneEnabled=yes\" --delete --whole-file gfsib02b:/homegfs_nfs 
> /backup/backup.0
> where gfsib02b:/homegfs_nfs is an NFS mount of the same volume.
> [root at gfs02b homegfs_nfs]# mount | grep homegfs
> gfsib02b.corvidtec.com:/homegfs.tcp on /homegfs type fuse.glusterfs 
> (rw,allow_other,max_read=131072)
> gfsib02b.corvidtec.com:/homegfs on /homegfs_nfs type nfs 
> (rw,vers=3,intr,bg,rsize=32768,wsize=32768,addr=10.200.71.2)
>
> Is a 5x speed difference expected between the FUSE and NFS mounts?  
> Note that very little data is being transferred.  And, when data is 
> being transferred, the transfer rate seems fine.  My issue with 
> gluster and rsync was that it seemed to be taking an extremely long 
> time to figure out what to transfer and this seems to largely be 
> mitigated by using NFS instead of FUSE on the server side (side where 
> data to be backed up resides).  I am still using gluster for the 
> backup machine as this seemed to have a very small affect on the timing.
>
> David
>
>
> ------ Original Message ------
> From: "Joe Julian" <joe at julianfamily.org>
> To: gluster-users at gluster.org
> Sent: 4/27/2015 6:56:04 PM
> Subject: Re: [Gluster-users] Disastrous performance with rsync to 
> mounted Gluster volume.
>
>>
>>
>> On 04/27/2015 03:00 PM, Ernie Dunbar wrote:
>>> On 2015-04-27 14:09, Joe Julian wrote:
>>>>
>>>> I've also noticed that if I increase the count of those writes, the
>>>> transfer speed increases as well:
>>>>
>>>> 2097152 bytes (2.1 MB) copied, 0.036291 s, 57.8 MB/s
>>>> root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
>>>> count=2048 bs=1024; sync
>>>> 2048+0 records in
>>>> 2048+0 records out
>>>> 2097152 bytes (2.1 MB) copied, 0.0362724 s, 57.8 MB/s
>>>> root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
>>>> count=2048 bs=1024; sync
>>>> 2048+0 records in
>>>> 2048+0 records out
>>>> 2097152 bytes (2.1 MB) copied, 0.0360319 s, 58.2 MB/s
>>>> root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
>>>> count=10240 bs=1024; sync
>>>> 10240+0 records in
>>>> 10240+0 records out
>>>> 10485760 bytes (10 MB) copied, 0.127219 s, 82.4 MB/s
>>>> root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile
>>>> count=10240 bs=1024; sync
>>>> 10240+0 records in
>>>> 10240+0 records out
>>>> 10485760 bytes (10 MB) copied, 0.128671 s, 81.5 MB/s
>>>>
>>>> This is correct, there is overhead that happens with small files and
>>>> the smaller the file the less throughput you get. That said, since
>>>> files are smaller you should get more files / second but less MB /
>>>> second. I have found that when you go under 16k changing files size
>>>> doesn't matter, you will get the same number of 16k files / second as
>>>> you do 1 k files.
>>>>
>>>>  The overhead happens regardless. You just notice it more when you're
>>>> doing it a lot more frequently.
>>>
>>>
>>> Well, it would be helpful to know what specifically rsync is trying 
>>> to do when it's sitting there making overhead, and whether it's 
>>> possible to tell rsync to avoid doing it, and just copy files 
>>> instead (which it does quite quickly).
>>>
>>> I suppose technically speaking, it's an rsync-specific question, but 
>>> it's all about making rsync and glusterfs play nice, and we pretty 
>>> much all need to know that!
>>
>> Yes, that's very rsync specific. Rsync not only checks the files 
>> metadata, but it also does a hash comparison.
>>
>> Each lookup() of each file requires a lookup from *each* replica 
>> server. Lookup's are triggered on open, or even fstat. Since rsync 
>> requests the stat of every file for comparison, this requires a 
>> little extra network time. The client queries all the replica in case 
>> one is out of date to ensure it's returning accurate results (and 
>> heal if a replica needs it).
>>
>> After bulding a list of files that differ between the source and the 
>> target, rsync copies the file to a temporary filename. After 
>> completing the temporary file, rsync then renames the temporary to 
>> the target filename. This has the disadvantage of putting the target 
>> file on the "wrong" dht subvolume because the hash for the temporary 
>> filename is different from the target filename.
>>
>> rsync's network transfer is over ssh by default, so you're also 
>> hitting encryption and buffer overhead.
>>
>> The optimum process for *initially* copying large numbers of files to 
>> gluster would be to blindly copy a list of files from the source to 
>> the target without reading the target. If copying across a network, 
>> maximizing the packet size is also advantageous. I've found tar ( + 
>> pv if you want to monitor throughput) + netcat (or socat) to be much 
>> faster than rsync.
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>