[Gluster-users] Very slow directory listing and high CPU usage on replicated volume

[Jonathan Lefman]

I take it back. Things deteriorated pretty quickly after I began dumping
data onto my volume from multiple clients. Initially my transfer rates were
okay, not fast, but livable. However after about an hour of copying several
terabytes from 3-4 client machines, the rates of transfer often dropped to
lb/s. Sometimes I would see a couple second burst of good transfer rates.

Anyone have ideas on how to address this effectively? I'm at a loss.

-Jon
On Nov 2, 2012 1:21 PM, "Jonathan Lefman" <jonathan.lefman at essess.com>
wrote:

> I should have also said that my volume is working well now and all is well.
>
> -Jon
>
>
> On Fri, Nov 2, 2012 at 1:21 PM, Jonathan Lefman <
> jonathan.lefman at essess.com> wrote:
>
>> Thank you Brian. I'm happy to hear that this behavior is not typical. I
>> am now using xfs on all of my drives.  I also wiped out the entire
>>  /etc/glusterd directory for good measure.  I bet that there was residual
>> information from a previous attempt at a gluster volume that must have
>> caused problems.  Or moving to xfs from ext4 is an amazing fix, but I think
>> this is less likely.
>>
>> I appreciate your time responding to me.
>>
>> -Jon
>> On Nov 2, 2012 4:44 AM, "Brian Candler" <B.Candler at pobox.com> wrote:
>>
>>> On Thu, Nov 01, 2012 at 08:03:21PM -0400, Jonathan Lefman wrote:
>>> >    Soon after loading up about 100 MB of small files (about 300kb
>>> each),
>>> >    the drive usage is at 1.1T.
>>>
>>> That is very odd. What do you get if you run du and df on the individual
>>> bricks themselves? 100MB is only ~330 files of 300KB each.
>>>
>>> Did you specify any special options to mkfs.ext4? Maybe -l 512 would
>>> help,
>>> as the xattrs are more likely to sit within the indoes themselves.
>>>
>>> If you start everything from scratch, it would be interesting to see df
>>> stats when the filesystem is empty.  It may be that a huge amount of
>>> space
>>> has been allocated to inodes.  If you expect most of your files >16KB
>>> then
>>> you could add -i 16384 to mkfs.ext4 to reduce the space reserved for
>>> inodes.
>>> But using xfs would be better, as it doesn't reserve any space for
>>> inodes,
>>> it allocates it dynamically.
>>>
>>> Ignore the comment that glusterfs is "not designed for handling large
>>> count
>>> small files" - 300KB is not small.
>>>
>>> Regards,
>>>
>>> Brian.
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121105/64dd239b/attachment.html>