[Gluster-users] Best practices?

[Greg_Swift at aotx.uscourts.gov]

gluster-users-bounces at gluster.org wrote on 01/24/2012 10:10:15 AM:
>
> On Tue, Jan 24, 2012 at 09:11:01AM -0600, Greg_Swift at aotx.uscourts.gov
wrote:
> > We have to have large numbers of volumes (~200).  Quick run down to
give
> > context.
> >
> > Our nodes would have around 128TB of local storage from several 32TB
raid
> > sets.  We started with ext4, so had a 16TB maximum.
>
> Aside: http://blog.ronnyegner-consulting.de/2011/08/18/ext4-and-
> the-16-tb-limit-now-solved/

kewl

> > So we broke it down
> > into nice even chunks of 16TB, thus 8 file systems. Our first attempt
was
> > ~200 volumes all using the 8 bricks per node (thus 1600 process/ports)
> ...
> > We had issues, and Gluster recommended
> > reducing our process/port count.
>
> So just checking I understand, the original configuration was:
>
> /data1/vol1 .. /data1/vol200
> ...
> /data8/vol1 .. /data8/vol200

correct

>
> Terminology issue: isn't each serverN:/dirM considered a separate 'brick'
to
> Gluster?  I would have thought that configuration would count as 1600
bricks
> per node (but with groups of 200 bricks sharing 1 underlying filesystem)

You are right... i should have said 1600 bricks/process/ports

For the sake of the conversation I think this expansion to the brick
definition is

Brick: A unique file system on a single storage node that runs a single
glusterfsd process and opens a single listening TCP port. (1 brick = 1
process = 1 port).  A combination of 1 or more bricks compromise a volume.

>
> > First we dropped down to only using 1 brick per volume per node, but
this
> > left us in a scenario of managing growth
>
> Like this?
>
> /data1/vol1   .. /data1/vol25
> /data2/vol26  .. /data2/vol50
> ...
> /data8/vol175 .. /data8/vol200

yes


> I see, so you have to assign the right subset of volumes to each
filesystem.
> I guess you could shuffle them around using replace-brick, but it would
be a
> pain.

very much so

> > So we determined to move to XFS to reduce from 8 partitions
> > down to 2 LVs.  Each would be 64TB each
>
> /data1/vol1   .. /data1/vol200
> /data2/vol1   .. /data2/vol200
>
> i.e. 400 ports/processes/(bricks?) per server.

That was the plan

> > We then ran into some performance
> > issues and found we had not tuned the XFS enough, which also deterred
us
> > from pushing forward with the move.
>
> I don't have any experience with XFS, but the Gluster docs do recommend
it
> as the one most heavily tested.
>
> I saw an old note here about tuning XFS to include extended attributes in
> the inode:
> http://www.gluster.org/community/documentation/index.php/
> Guide_to_Optimizing_GlusterFS
> (although the values shown seem to be defaults to mkfs.xfs nowadays)
>
> Did you find any other tuning was required?

We didn't do much tuning at the front end, which seems to have potentially
been a problem.  After the fact we did add some tuning such as disabling
barriers, atime and diratime.

> This is all extremely helpful - many thanks for sharing your experiences.

>
> BTW I am just in the process of setting up two test systems here.
Somewhat
> smaller than yours, but they are based on this chassis:
> http://www.xcase.co.uk/24-bay-Hotswap-rackmount-chassis-norco-
> RPC-4224-p/case-xcase-rm424.htm
> with Hitachi low-power 3TB drives.
>

thats pretty kewl.. to bad it doesn't do SFF hard drives... that would be
awesome.

-greg