[Gluster-users] Very slow directory listing and high CPU usage on replicated volume

[Jonathan Lefman]

Thanks Pranith. We have tried two organizations with the same results.
The levels we use currently are like this (indicates number of dirs at this
level):
-dir-top
--runs (1)
---id (1)
----cam (4)
-----date (1)
------h (6)
-------m (60)
--------s (60)

We have also used a flatter structure too. I find that the client machine's
glusterfs process uses 100% of the core processing. When I kill the ls (or
find, mv, etc) command, usage returns to baseline on the client and daemons.

-Jon
On Nov 1, 2012 10:52 PM, "Pranith Kumar Karampuri" <pkarampu at redhat.com>
wrote:

> Jonathan,
>    Could you give us the directory structure details, so that we can try
> to re-create the issue. I am assuming each file is about 300kb. Please give
> us the depth of the directory structure and how many directories in each
> level.
>
> Thanks
> Pranith
> ----- Original Message -----
> From: "Jonathan Lefman" <jonathan.lefman at essess.com>
> To: gluster-users at gluster.org
> Sent: Friday, November 2, 2012 5:33:21 AM
> Subject: [Gluster-users] Very slow directory listing and high CPU usage on
>      replicated volume
>
>
> Hi all,
>
>
> I am having problems with painfully slow directory listings on a freshly
> created replicated volume. The configuration is as follows: 2 nodes with 3
> replicated drives each. The total volume capacity is 5.6T. We would like to
> expand the storage capacity much more, but first we need to figure this
> problem out.
>
>
> Soon after loading up about 100 MB of small files (about 300kb each), the
> drive usage is at 1.1T. I am not sure if this to be expected. The main
> problem is that directory listing (ls or find) takes a very long time. The
> CPU usage on the nodes is high for each of the glusterfsd processes - 3 on
> each machine 54%, 43%, and 25% per core is an example of the usage. Memory
> is very low for each process. It is incredibly difficult to diagnose this
> issue. We have wiped previous gluster installs, all directories, and mount
> points as well as reformatting the disks. Each drive is formatted with ext4.
>
>
> Has anyone had a similar result? Any ideas on how to debug this one?
>
>
> Thank you,
>
>
> Jon
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121101/7bde7310/attachment.html>