[Gluster-users] Strange performance issues on Debian/Lenny compared to Ubuntu

Raghavendra G raghavendra at gluster.com
Thu Apr 1 04:25:11 UTC 2010


Hi Oliver,

On Wed, Mar 31, 2010 at 12:23 PM, Olivier Le Cam <
Olivier.LeCam at crdp.ac-versailles.fr> wrote:

> Hi -
>
>
>  * Can you test dd performance with only write-behind translator (along
>> with
>> protocol/client) in client volume configuration?
>>
>
> That makes no difference with either only write-behind translator or all of
> default translator configurated by genvol. Without it, results are abt 10
> times worse.
>
>
>  * Also can you add "debug/trace" translator above and below write-behind,
>> start glusterfs with Log-level trace and send the log files?
>>
>
> * Debian/Lenny (kernel 2.6.32-trunk-amd64, same with 2.6.26)
>   ~# dd if=/dev/zero of=/mnt/test_file count=262144 bs=1024
>   262144+0 enregistrements lus
>   262144+0 enregistrements écrits
>   268435456 bytes (268 MB) copied, 117,14 s, 2,3 MB/s
>
>   Log file size is 283MB. Do you want to get it entirely?
>   First 10000 lines of /var/log/glusterfs/mnt.log uploaded to:
>   http://olecam.online.fr/glusterfs/lenny_log.txt
>
>   ~# cat /var/log/glusterfs/mnt.log |wc -l
>   1792172
>
>
> * Ubuntu/Hardy (kernel 2.6.24-23-server x86_64)
>    ~# dd if=/dev/zero of=/mnt/test_file count=262144 bs=1024
>
>    262144+0 records in
>    262144+0 records out
>    268435456 bytes (268 MB) copied, 17,1914 s, 15,6 MB/s
>
>    Log file size is 115MB. Do you want to get it entirely?
>    First 10000 lines of /var/log/glusterfs/mnt.log uploaded to:
>    http://olecam.online.fr/glusterfs/ubuntu_log.txt
>
>    ~# cat /var/log/glusterfs/mnt.log |wc -l
>    553600
>
> * Addition notes:
>
> - Using tcpdump, I have observed that dd'ing from the Debian/Lenny
> generates abt 10 times more frames on the network. Frame size sent by the
> Ubuntu client are generally larger, this can explain why the transfer rate
> is so terrible from my Lenny client.
>
>  - tcpdump on Lenny client shows: "278648 packets captured"
>  - tcpdump on Ubuntu client shows: "28120 packets captured" (abt 10x less
> frames traffic!!)
>
>  tcpdump -v outputs also show that packet size sent the Ubuntu client is
> usually abt 40-60 KBytes large. For example:
>  10:06:34.795101 IP (tos 0x0, ttl 64, id 38433, offset 0, flags [DF], proto
> TCP (6), length 50732)
>  This is never the case from the Debian client(always < 10KBytes).
>

I am seeing write-behind aggregating large number of writes into a single
large write on Ubuntu client, but not on lenny client. This is the cause of
large number of frames on network. This can also be the cause of slower
performance on lenny.

Can you set "option enable-trickling-writes false" in write-behind on lenny
and rerun tests. That will set write-behind to do more aggressive
aggregation. We are eagerly waiting for your results :).

However I am still investigating what is the reason behind write-behind on
Ubuntu doing large aggregation but not on lenny.


> As observed by Paul who has better results than mine with an Lenny client,
> this might be related to some kernel options or "optimizations" in recent
> versions. I haven't been able to give a test with 2.6.18, the only one I
> have under hand is a xen domU (under an Etch dom0) and I cannot get the fuse
> loading properly on it.
>
> Still available for anything else I can do.
>
> Thanks and best regards,
> --
> Olivier
>



-- 
Raghavendra G


More information about the Gluster-users mailing list