[Gluster-devel] successive bonnie++ tests taking longer and longer to run (system load steadily increasing)
Raghavendra G
raghavendra at gluster.com
Tue Feb 23 04:29:36 UTC 2010
* What does the top output corresponding to glusterfs say? what is the
memory usage and cpu usage?
* Do you find anything interesting in glusterfs client log files? Can we get
the log files?
regards,
On Mon, Feb 22, 2010 at 1:29 PM, Daniel Maher
<dma+gluster at witbe.net<dma%2Bgluster at witbe.net>
> wrote:
> Hello all,
>
> Four-node client-side replication setup running 3.0.0 throughout.
> Config generated via :
> $ /usr/bin/glusterfs-volgen --name replicated --raid 1 s01:/opt/gluster
> s02:/opt/gluster
>
> Connected via private gigabit LAN. The hardware in each of the four
> nodes is identical.
>
> Running the following script on one of the clients :
>
> ---
> #!/bin/bash
>
> loop=0
>
> while [ $loop -lt 100 ]
> do
> (time /usr/sbin/bonnie++ -d /opt/gluster/bonnie -u root) 3>&1 2>&3 |
> tee -a bonnie-single-c01.log
> done
> ---
>
> [root at c01 ~]# # bonnie++ 2>&1 | grep Version
> Version: 1.03
>
> The time to completion has been slowly creeping upwards. Granted, the
> full run hasn't completed yet, but after 89 iterations, a clear trend
> appears to be forming :
>
> [root at c01 ~]# grep real bonnie-single-c01.log
> real 15m4.444s
> real 15m49.245s
> real 16m44.263s
> real 16m55.868s
> real 17m16.479s
> real 18m4.800s
> real 19m18.089s
> real 20m59.218s
> real 22m58.493s
> real 22m25.134s
> real 23m36.715s
> real 25m4.022s
> real 29m36.527s
> real 26m26.325s
> real 28m37.396s
> real 35m4.289s
> real 32m41.834s
> real 34m33.418s
> real 37m1.098s
> real 38m10.750s
> real 46m14.002s
> real 40m15.416s
> real 36m53.414s
> real 44m46.302s
> real 38m57.842s
> real 41m55.745s
> real 46m41.493s
> real 54m12.818s
> real 52m53.408s
> real 36m20.212s
> real 40m1.991s
> real 54m54.110s
> real 57m57.162s
> real 57m53.896s
> real 62m56.905s
> real 65m53.524s
> real 67m54.105s
> real 74m40.195s
> real 80m30.843s
> real 81m15.571s
> real 77m50.950s
> real 91m46.081s
> real 87m50.726s
> real 94m7.863s
> real 94m37.050s
> real 86m43.104s
> real 96m22.384s
> real 98m42.054s
> real 103m56.266s
> real 109m33.518s
> real 101m49.557s
> real 93m25.883s
> real 87m26.209s
> real 97m36.759s
> real 111m18.548s
> real 97m53.249s
> real 97m44.629s
> real 100m36.912s
> real 102m15.858s
> real 97m16.631s
> real 116m44.927s
> real 102m11.010s
> real 129m3.242s
> real 99m32.759s
> real 117m23.075s
> real 112m1.927s
> real 116m45.873s
> real 125m22.549s
> real 129m7.776s
> real 114m19.009s
> real 120m21.973s
> real 111m15.333s
> real 113m4.147s
> real 96m20.848s
> real 107m0.043s
> real 128m11.131s
> real 118m58.662s
> real 116m40.381s
> real 118m50.392s
> real 118m17.404s
> real 117m29.695s
> real 94m45.270s
> real 104m35.469s
> real 125m56.206s
> real 108m40.728s
> real 133m24.056s
> real 103m16.188s
> real 146m42.629s
> real 101m5.182s
>
> As a control, i ran the same script with a local filesystem target on
> the other client :
>
> [root at c02 ~]# grep real bonnie-local-c02.log
> real 3m42.788s
> real 3m43.251s
> real 3m42.568s
> real 3m42.533s
> real 3m41.764s
> real 3m43.305s
> real 3m42.536s
> real 3m43.052s
> real 3m44.406s
> real 3m42.894s
> real 3m43.639s
> (cut for brevity - all 100 runs are around this time mark.)
>
> It is perhaps worth noting that the load on c01 (running the gluster
> tests) has been creeping up as well. It started at basically 0, and is
> now at 4.03, 3.94, 3.27 .
>
> Any ideas what's going on ?
>
>
>
> --
> Daniel Maher <dma+gluster AT witbe DOT net>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
--
Raghavendra G
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20100223/23536725/attachment-0003.html>
More information about the Gluster-devel
mailing list