[Gluster-devel] Performance report and some issues

[Jordi Moles]

Hi,

I want to report back the performance issues i've had so far with 
glusterfs mainline 2.5, patch 690 and fuse-2.7.2glfs8.

I'm setting a mail system, which is all ran by Xen 3.2.0 and every 
"actual" piece of the mail system is a virtual machine from xen.

Anyway... the virtual machines accessing glusterfs are 6 dovecots and 4 
postfixs.  There are also 6 nodes, which share their own disk to the 
gluster filesystem. Two of the nodes, share 2 disks, one for the 
glusterfs, and the other for the namespace

these are the conf files:

****nodes with namespace****

volume esp
    type storage/posix
    option directory /mnt/compartit
end-volume

volume espa
    type features/posix-locks
    subvolumes esp
end-volume

volume espai
   type performance/io-threads
   option thread-count 15
   option cache-size 512MB
   subvolumes espa
end-volume

volume nm
    type storage/posix
    option directory /mnt/namespace
end-volume

volume ultim
    type protocol/server
    subvolumes espai nm
    option transport-type tcp/server
    option auth.ip.espai.allow *
    option auth.ip.nm.allow *
end-volume

*************


***nodes without namespace*****

volume esp
    type storage/posix
    option directory /mnt/compartit
end-volume

volume espa
    type features/posix-locks
    subvolumes esp
end-volume

volume espai
   type performance/io-threads
   option thread-count 15
   option cache-size 512MB
   subvolumes espa
end-volume

volume ultim
    type protocol/server
    subvolumes espai
    option transport-type tcp/server
    option auth.ip.espai.allow *
end-volume

*****************************


***clients****

volume espai1
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.1.204
    option remote-subvolume espai
end-volume

volume espai2
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.1.205
    option remote-subvolume espai
end-volume

volume espai3
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.1.206
    option remote-subvolume espai
end-volume

volume espai4
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.1.207
    option remote-subvolume espai
end-volume

volume espai5
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.1.213
    option remote-subvolume espai
end-volume

volume espai6
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.1.214
    option remote-subvolume espai
end-volume

volume namespace1
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.1.204
    option remote-subvolume nm
end-volume

volume namespace2
    type protocol/client
    option transport-type tcp/client
    option remote-host 192.168.1.205
    option remote-subvolume nm
end-volume

volume grup1
    type cluster/afr
    subvolumes espai1 espai2
end-volume

volume grup2
    type cluster/afr
    subvolumes espai3 espai4
end-volume

volume grup3
    type cluster/afr
    subvolumes espai5 espai6
end-volume

volume nm
    type cluster/afr
    subvolumes namespace1 namespace2
end-volume

volume ultim
    type cluster/unify
    subvolumes grup1 grup2 grup3
    option scheduler rr
    option namespace nm
end-volume

************

The thing is that in earlier patches, the whole system used to hang, 
with many different error messages.

Right now, it's been on for days without any hang at all, but i'm facing 
serious performance issues.

By only running an "ls" command, it can take like 3 seconds to show 
something when the system is "under load". It doesn't happen at all when 
there's no activity, so i don't thing has anything to do with xen. Well, 
actually, "under load" can mean 3 mails arriving per second. I'm 
monitoring everything, and no virtual machine is using more than 20% of 
cpu or so.

First, i had log level on both nodes and clients set to DEBUG, but now 
is just WARNING, and i've restarted everything so many times.

I was suggested to use "type performance/io-threads" on the node side. 
It actually worked, before that, it wasn't 3 seconds, but 5 or more. 
I've set the "thread-count" value to different values and also "cache-size"

The system is supposed to handle a big amount of traffic, far more than 
3 mails a second.

What do you think about the whole set up? Should i keep using namespace? 
Should i use new nodes for namespaces? Should i use different values for 
iothread?

One last thing... i'm using reiserfs on the "storage devices" that nodes 
share. Should i be using XFS or something else?

Logs don't show any kind of error now... i don't have a clue of what is 
failing now....

I would be pleased if you could give some ideas.

Thank you.