<div dir="ltr"><div>Just create one 3 replica volume with 1 brick on each of 3 storage servers. Raid5 for servers will be more then enough - it is already replica 3.<br></div><div>Use ovirt to mount glusterfs to VM from hosts (as it uses libgfapi) rather then fuse mount from VM itself.</div><div>libgfapi is supposedly faster. Might depend on which mail storage type you use - if maildir, libgfapi to VM on he host should be better.</div><div>Also mid that you might need compression for mail storage and deduplication (of attachments)<br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 25, 2018 at 5:11 AM, Γιώργος Βασιλόπουλος <span dir="ltr"><<a href="mailto:g.vasilopoulos@uoc.gr" target="_blank">g.vasilopoulos@uoc.gr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello<br>
I am trying to lay down my options regarding storage with glusterfs, vm storage and email storage<br>
<br>
Hardware in my disposal is specific : I have 9 servers for running vm's under ovirt 4.2 and 3 servers for storage<br>
<br>
The 3 storage machines are similar and each have 2xE5-2640v3 cpus 128GB RAM and 2x10G ethernet<br>
each storage server has inside 2x300gb 10k drives which I intent to use as os install and maybe a litle volume for isos on nfs<br>
also present are 6x200GB SSD drives which I think of using as tiering volume(s)<br>
And the main storage is on an external JBOD box with 12x4TB drives connected via SAS to the server with RAID controller capable of varius raid levels.<br>
<br>
So what I'm thinking is that I will create 3 replica 3 arbiter 1 volumes (high availability is the no 1 requirement) in a cyclic fashion, 2 data bricks and one arbiter on each storage server<br>
I will implement raid6 with one spare drive on each server (we are in an island and getting a disk replacement can take days occasionaly) which will give me about 36T of usable storage per server.<br>
So regarding the vm storage I am thinking that 2 volumes with 17TB each and an arbiter of 1TB.<br>
What messes things up is that I was required to put the email storage in this installation. Our email is pretty big and busy with about 50000 users curently at 13T of storage.<br>
Currently it runs on a few vm's and uses storage from nfs given from another vm. It is runs postfix/dovecot and right now a single big vm does mailbox delivery but this reaches it's limits. Mail storage now is on a EMC VNX5500<br>
But it will be moved to glusterfs for various reasons.<br>
<br>
I would like some advise regarding the email storage. I think my options are<br>
<br>
1a. use a VM as NFS give it a huge disk (raw image on gluster vm optimized) and be done with it<br>
1b use a VM as NFS give it a 2 or 3 disks unified under lvm vg->lv (raw images on gluster vm optimized) and maybe take some advantage of using 2-3 io-threads in o virt to write to 2-3 disks simultaneously. Will this give extra performance ?<br>
<br>
2.Give gluster as nfs straight to dovecot but I wonder if this will have performance drawback since it will be fuse mounted. I am also worried about the arbiter volume since there will be thousands of small files, practically arbiter<br>
will probably have to be as large as the data bricks or half that size<br>
<br>
3. Give gluster as glusterfs mount point which I think will have about the same issues as 2.<br>
<br>
I have read about problems with dovecot indexes and glusterfs. Is this still an issue? or is it a problem that only shows when there is no dovecot director.<br>
Personaly I am inclined on using solution 1 because I think that arbiter volumes will be smaller (Am I right?) though it may have some overhead regarding nfs on the vm. On the other hand this solution will use libgfapi<br>
which might balance things a bit.<br>
Will it help if in such a case use small (16mb) shard size and tiering ?<br>
<br>
I'm afraid I have it a bit mixed up in my mind and I could really use some help.<br>
<br>
<br>
<br>
-- <br>
Βασιλόπουλος Γιώργος<br>
Ηλεκτρολόγος Μηχανικός Τ.Ε.<br>
Διαχειριστής Υπολ. Συστημάτων<br>
<br>
Πανεπιστήμιο Κρήτης<br>
Κ.Υ.Υ.Τ.Π.Ε.<br>
Τμήμα Επικοινωνιών και Δικτύων<br>
Βούτες Ηρακλείου 70013<br>
Τηλ : 2810393310<br>
email : <a href="mailto:g.vasilopoulos@uoc.gr" target="_blank">g.vasilopoulos@uoc.gr</a><br>
<a href="http://www.ucnet.uoc.gr" rel="noreferrer" target="_blank">http://www.ucnet.uoc.gr</a><br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a></blockquote></div><br></div>