[Gluster-users] Recommendations for busy static web server replacement

Wed Feb 8 01:11:01 UTC 2012

We run a similar setup here.  I use gluster 2.x to strip/replicate 8
30tb xfs bricks (24x1.5tb raid6) into a single 120tb array.  Each of
my objects is around 1-4megs in size, avg is around 2.  Here are a few
things we've done to get good performance from our setup.

a) We use XFS with mount options "noatime,nodiratime,inode64"

b) I use a hardware 3ware 9690/9750 raid controller with the latest
firmware and the battery backup option.  With such a large array as we
use we lose hard drives all the time so doing rebuilds in hardware is
great.  Make sure to run the latest 3ware firmware.  I've tried Dell
perc, a few different LSI cards, supermicro branded LSI cards, and a
few others and 3ware was by far the most reliable.  I wanted to use
cheap off the shelf desktop SATA drives so the controller has to be
smart about dealing with SATA timeouts.

c) Our gluster servers are Centos 5 or 6.  Disable smartd as it
interferes with the 3ware controller.  I set the blockdev
(/sbin/blockdev --setra 16384 /dev/sda) on each of the bricks - this
was a huge help!  Also make sure you run the latest drive available
from 3ware.

d) On the gluster side of things, we use a "raid10" type of setup.  We
replicate two sets of 4 bricks striped together (type =
cluster/distribute), so we have two complete copies of our data.  We
break this mirror on our public facing feed servers.   We have two
feed servers running apache with a custom in-house apache module to
handle the actual serving of data.  Each server only talks to one side
of gluster - so we intensionally break glusters replication on
feeding.  If one of our filers goes offline we have to disable that
feed server in our load balancer and then of course repair any data
that wasn't replicated with a "ls -alR".  We've found that disabling
gluster's replication on our feed side increased performance
dramatically because it wasn't having to do read-repairs checking.  Of
course the servers that are creating the content talk to both sides of
the cluster.  Make sure to have good monitoring (we use nagios with
lots of custom scripts).

e) I have a very small 2mb cache in our gluster clients.  We have such
a large volume/library that getting a cache hit almost never happens
so don't waste the memory.

f)  My apache module rewrites incoming URIs to load balance incoming
requests to two different gluster mounts on the filesystem.  Each
gluster mount is its own client talking to the same server over
different gigabit ethernet links to different glusterfsd daemons
running on different ports.  ie 192.168.1.50:6996 and
192.168.2.50:6997.  This not only doubled my bandwidth between my feed
clients and backend servers (which is good) it also reduces latency on
those links. I've also found that the linux thread scheduler does a
better job of distributing glusterfsd file requests with multiple
glusterfsd processes then a single gluster process and a higher thread
count.

g) My hardware is pretty simple.  As mentioned above, I use 3ware
hardware raid controllers.  Seagate 1.5tb and 2tb 7200rpm SATA desktop
drives.  Supermicro 4u 24 drive chassis with a builtin SAS expander
backplane.  The raid card connects to the backplane via 4 lane SAS
connector.  I use low end dual cpu quad-core xeon cpus and 8g of ram
(remember you can't cache anything).  Each head server has ~48 drives
(2 bricks).  My apache feed servers are currently two 5 year old Dell
1950 with 8g of ram.

An average day here I pushed on average about 1tb of data a day in
about 4 million requests or around 100-200mbit/sec.  However i've
sustained 6 times that with the same setup or around ~600-700mbits/sec
and nearly 20 million requests a day.

I haven't tried the newer version of gluster 3.x as everything just
sort of works for the most part on the 2.x code.  There are gotchas
and things that annoy me but for the most part everything works very
well.  I was able to replace my old Isilon storage for less then the
annual cost of the support contract and doubling the space in the
process!

liam

On Tue, Feb 7, 2012 at 11:26 AM, Brian Candler <B.Candler at pobox.com> wrote:
> On Tue, Feb 07, 2012 at 12:56:57PM -0500, John Mark Walker wrote:
>> Brian - thank you for sharing these configuration tips. I'd love to have that in a blog post :)
>>
>> As a close second, perhaps you could post a mini Q&A on community.gluster.org? This is the type of information that's very useful for google to index and make available.
>
> I'm just learning this stuff myself, accumulating tips from here and from
> the XFS mailing list :-)
>
> I'm doing internal documentation as I go, but I'll try to make a public
> summary too.
>
> Regards,
>
> Brian.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users