[Gluster-users] trying to figure out the best solution for vm and email storage

[Γιώργος Βασιλόπουλος]

Hello
I am trying to lay down my options regarding storage with glusterfs, vm 
storage and email storage

Hardware in my disposal is specific : I have 9 servers for running vm's 
under ovirt 4.2 and 3 servers for storage

The 3 storage machines are similar and each have 2xE5-2640v3 cpus 128GB 
RAM and 2x10G ethernet
each storage server has inside 2x300gb 10k drives which I intent to use 
as os install and maybe a litle volume for isos on nfs
also present are 6x200GB SSD drives which I think of using as tiering 
volume(s)
And the main storage is on an external JBOD box with 12x4TB drives 
connected via SAS to the server with RAID controller capable of varius 
raid levels.

So what I'm thinking is that I will create 3 replica 3 arbiter 1 volumes 
(high availability is the no 1 requirement) in a cyclic fashion, 2 data 
bricks and one arbiter on each storage server
I will implement raid6 with one spare drive on each server (we are in an 
island and getting a disk replacement can take days occasionaly) which 
will give me about 36T of usable storage per server.
So regarding the vm storage I am thinking that 2 volumes with 17TB each 
and an arbiter of 1TB.
What messes things up is that I was required to put the email storage in 
this installation. Our email is pretty big and busy with about 50000 
users curently at 13T of storage.
Currently it runs on a few vm's and uses storage from nfs given from 
another vm. It is runs postfix/dovecot and right now a single big vm 
does  mailbox delivery but this reaches it's limits. Mail storage now is 
on a EMC VNX5500
But it will be moved to glusterfs for various reasons.

I would like some advise regarding the email storage. I think my options 
are

1a. use a VM as NFS give it a huge disk (raw image on gluster vm 
optimized) and be done with it
1b use a VM as NFS give it a 2 or 3 disks unified under lvm vg->lv (raw 
images on gluster vm optimized) and maybe take some advantage of using 
2-3 io-threads in o virt to write to 2-3 disks simultaneously. Will this 
give extra performance ?

2.Give gluster as nfs straight to dovecot but I wonder if this will have 
performance drawback since it will be fuse mounted. I am also worried 
about the arbiter volume since there will be thousands of small files, 
practically arbiter
will probably have to be as large as the data bricks or half that size

3. Give gluster as glusterfs mount point which I think will have about 
the same issues as 2.

I have read about problems with dovecot indexes and glusterfs. Is this 
still an issue? or is it a problem that only shows when there is no 
dovecot director.
Personaly I am inclined on using solution 1 because I think that arbiter 
volumes will be smaller (Am I right?) though it may have some overhead 
regarding nfs on the vm. On the other hand this solution will use libgfapi
which might balance things a bit.
Will it help if in such a case use small (16mb) shard size and tiering ?

I'm afraid I have it a bit mixed up in my mind and I could really use 
some help.



-- 
Βασιλόπουλος Γιώργος
Ηλεκτρολόγος Μηχανικός Τ.Ε.
Διαχειριστής Υπολ. Συστημάτων

Πανεπιστήμιο Κρήτης
Κ.Υ.Υ.Τ.Π.Ε.
Τμήμα Επικοινωνιών και Δικτύων
Βούτες Ηρακλείου 70013
Τηλ   : 2810393310
email : g.vasilopoulos at uoc.gr
http://www.ucnet.uoc.gr