[Gluster-users] Very slow directory listing and high CPU usage on replicated volume

[Jonathan Lefman]

I am attaching some extra information to help diagnose this issue.
 Hopefully this is useful.

Volume Name: my_gluster_data
Type: Distributed-Replicate
Volume ID: e8865f37-6e22-476d-956e-29280bb07e75
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: server1:/media/data1/my_gluster_data
Brick2: server2:/media/data1/my_gluster_data
Brick3: server1:/media/data2/my_gluster_data
Brick4: server2:/media/data2/my_gluster_data
Brick5: server1:/media/data3/my_gluster_data
Brick6: hserver2:/media/data3/my_gluster_data


On Thu, Nov 1, 2012 at 8:03 PM, Jonathan Lefman
<jonathan.lefman at essess.com>wrote:

> Hi all,
>
> I am having problems with painfully slow directory listings on a freshly
> created replicated volume.  The configuration is as follows:   2 nodes with
> 3 replicated drives each.  The total volume capacity is 5.6T.  We would
> like to expand the storage capacity much more, but first we need to figure
> this problem out.
>
> Soon after loading up about 100 MB of small files (about 300kb each), the
> drive usage is at 1.1T.  I am not sure if this to be expected.  The main
> problem is that directory listing (ls or find) takes a very long time.  The
> CPU usage on the nodes is high for each of the glusterfsd processes - 3 on
> each machine 54%, 43%, and 25% per core is an example of the usage.  Memory
> is very low for each process.  It is incredibly difficult to diagnose this
> issue.  We have wiped previous gluster installs, all directories, and mount
> points as well as reformatting the disks.  Each drive is formatted with
> ext4.
>
> Has anyone had a similar result?  Any ideas on how to debug this one?
>
> Thank you,
>
> Jon
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121101/58e1bf52/attachment.html>