[Gluster-users] Strange performance issues on Debian/Lenny compared to Ubuntu

[Olivier Le Cam]

Hi -

> * Can you test dd performance with only write-behind translator (along with
> protocol/client) in client volume configuration?

That makes no difference with either only write-behind translator or all 
of default translator configurated by genvol. Without it, results are 
abt 10 times worse.

> * Also can you add "debug/trace" translator above and below write-behind,
> start glusterfs with Log-level trace and send the log files?

* Debian/Lenny (kernel 2.6.32-trunk-amd64, same with 2.6.26)
    ~# dd if=/dev/zero of=/mnt/test_file count=262144 bs=1024
    262144+0 enregistrements lus
    262144+0 enregistrements écrits
    268435456 bytes (268 MB) copied, 117,14 s, 2,3 MB/s

    Log file size is 283MB. Do you want to get it entirely?
    First 10000 lines of /var/log/glusterfs/mnt.log uploaded to:
    http://olecam.online.fr/glusterfs/lenny_log.txt

    ~# cat /var/log/glusterfs/mnt.log |wc -l
    1792172


* Ubuntu/Hardy (kernel 2.6.24-23-server x86_64)
     ~# dd if=/dev/zero of=/mnt/test_file count=262144 bs=1024
     262144+0 records in
     262144+0 records out
     268435456 bytes (268 MB) copied, 17,1914 s, 15,6 MB/s

     Log file size is 115MB. Do you want to get it entirely?
     First 10000 lines of /var/log/glusterfs/mnt.log uploaded to:
     http://olecam.online.fr/glusterfs/ubuntu_log.txt

     ~# cat /var/log/glusterfs/mnt.log |wc -l
     553600

* Addition notes:

- Using tcpdump, I have observed that dd'ing from the Debian/Lenny 
generates abt 10 times more frames on the network. Frame size sent by 
the Ubuntu client are generally larger, this can explain why the 
transfer rate is so terrible from my Lenny client.

  - tcpdump on Lenny client shows: "278648 packets captured"
  - tcpdump on Ubuntu client shows: "28120 packets captured" (abt 10x 
less frames traffic!!)

  tcpdump -v outputs also show that packet size sent the Ubuntu client 
is usually abt 40-60 KBytes large. For example:
  10:06:34.795101 IP (tos 0x0, ttl 64, id 38433, offset 0, flags [DF], 
proto TCP (6), length 50732)
  This is never the case from the Debian client(always < 10KBytes).

As observed by Paul who has better results than mine with an Lenny 
client, this might be related to some kernel options or "optimizations" 
in recent versions. I haven't been able to give a test with 2.6.18, the 
only one I have under hand is a xen domU (under an Etch dom0) and I 
cannot get the fuse loading properly on it.

Still available for anything else I can do.

Thanks and best regards,
-- 
Olivier