[Gluster-users] GlusterFS performance, concurrency and I/O blocking

[Joe Landman]

On 08/23/2011 02:17 PM, Ken Randall wrote:
> Hi everybody!  Love this community, and I love GlusterFS.

[...]

> Before I get to my suspicion of what's happening, keep in mind that we
> have 50+ million files (over hundreds of thousands of directories), most
> of them are small, and each page request will pull in upwards of 10-40
> supporting assets (images, Flash files, CSS, JS, etc.).  We also have
> people executing directory listings whenever they're editing their site,
> as they choose images, etc. to insert onto the page.  We're also
> exporting the volume to CIFS so our Windows servers can access the
> GlusterFS client on the Linux machines in the cluster.  The Samba
> settings on there were tweaked to the hilt as well, turning off
> case-insensitivity, bumping up caches and async IO, etc.
>
> It appears as if GlusterFS has some kind of I/O blocking going on.
> Whenever a directory listing is being pieced together, it noticeably
> slows down (or stops?) other operations through the same client.  For a
> high-concurrency app like ours where the storage backend needs to be
> able to pull off 10 to 100 directory listings a second, and 5,000 to
> 10,000 IOPS overall, it's easy to see how perf would degrade if my
> blocking suspicion is correct.  The biggest culprit, in my guess, is the
> directory listing.  Executing one makes things drag.  I've been able to
> demonstrate that through a simple script.  And we're running some pretty
> monster machines with 24 cores, 24 GB RAM, etc.

[...]


> Am I way off?  Does GlusterFS block on directory listings (getdents) or
> any other operations?  If so, is there a way to enable the database
> equivalent of "dirty reads" so it doesn't block?

What was the back end file system and how did you construct it?

Many folks use the ext* series based upon recommendations from the 
company.  ext* is highly contra-indicated for highly parallel 
operations.  It simply cannot keep up with file systems designed for this.

Another area are in the extended attributes.  If you size the system or 
the extended attributes wrong, you will lose performance.

And (note we are biased given what we build/sell), hardware matters. 
The vast majority of hardware we've seen people stand up themselves for 
this, has been badly underpowered for massive scale IOs.  It usually 
starts with someone spec'ing a 6G backplane, and rapidly goes south from 
there.  It doesn't matter how fast the software layers are, if the 
hardware simply cannot keep up.  In the vast majority of cases where 
we've been called in to look at someones setup, yeah, the hardware 
played a huge role in the (lack of) performance.

>
> Ken
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615