[Gluster-users] trying to figure out the best solution for vm and email storage
g.vasilopoulos at uoc.gr
Wed Jul 25 09:11:16 UTC 2018
I am trying to lay down my options regarding storage with glusterfs, vm
storage and email storage
Hardware in my disposal is specific : I have 9 servers for running vm's
under ovirt 4.2 and 3 servers for storage
The 3 storage machines are similar and each have 2xE5-2640v3 cpus 128GB
RAM and 2x10G ethernet
each storage server has inside 2x300gb 10k drives which I intent to use
as os install and maybe a litle volume for isos on nfs
also present are 6x200GB SSD drives which I think of using as tiering
And the main storage is on an external JBOD box with 12x4TB drives
connected via SAS to the server with RAID controller capable of varius
So what I'm thinking is that I will create 3 replica 3 arbiter 1 volumes
(high availability is the no 1 requirement) in a cyclic fashion, 2 data
bricks and one arbiter on each storage server
I will implement raid6 with one spare drive on each server (we are in an
island and getting a disk replacement can take days occasionaly) which
will give me about 36T of usable storage per server.
So regarding the vm storage I am thinking that 2 volumes with 17TB each
and an arbiter of 1TB.
What messes things up is that I was required to put the email storage in
this installation. Our email is pretty big and busy with about 50000
users curently at 13T of storage.
Currently it runs on a few vm's and uses storage from nfs given from
another vm. It is runs postfix/dovecot and right now a single big vm
does mailbox delivery but this reaches it's limits. Mail storage now is
on a EMC VNX5500
But it will be moved to glusterfs for various reasons.
I would like some advise regarding the email storage. I think my options
1a. use a VM as NFS give it a huge disk (raw image on gluster vm
optimized) and be done with it
1b use a VM as NFS give it a 2 or 3 disks unified under lvm vg->lv (raw
images on gluster vm optimized) and maybe take some advantage of using
2-3 io-threads in o virt to write to 2-3 disks simultaneously. Will this
give extra performance ?
2.Give gluster as nfs straight to dovecot but I wonder if this will have
performance drawback since it will be fuse mounted. I am also worried
about the arbiter volume since there will be thousands of small files,
will probably have to be as large as the data bricks or half that size
3. Give gluster as glusterfs mount point which I think will have about
the same issues as 2.
I have read about problems with dovecot indexes and glusterfs. Is this
still an issue? or is it a problem that only shows when there is no
Personaly I am inclined on using solution 1 because I think that arbiter
volumes will be smaller (Am I right?) though it may have some overhead
regarding nfs on the vm. On the other hand this solution will use libgfapi
which might balance things a bit.
Will it help if in such a case use small (16mb) shard size and tiering ?
I'm afraid I have it a bit mixed up in my mind and I could really use
Ηλεκτρολόγος Μηχανικός Τ.Ε.
Διαχειριστής Υπολ. Συστημάτων
Τμήμα Επικοινωνιών και Δικτύων
Βούτες Ηρακλείου 70013
Τηλ : 2810393310
email : g.vasilopoulos at uoc.gr
More information about the Gluster-users