[Gluster-users] Slow write times to gluster disk
Pat Haley
phaley at mit.edu
Fri Jun 29 14:54:46 UTC 2018
Hi Raghavendra,
Our technician was able to try the manual setting today. He found that
our upper limit for performance.md-cache-timeout was 60 not 600, so he
used that value, along with the network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. Does this suggest
some addition tests/changes we could try?
Thanks
Pat
On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote:
>
>
> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu
> <mailto:phaley at mit.edu>> wrote:
>
>
> Hi Raghavendra,
>
> Setting the performance.write-behind off had a small improvement
> on the write speed (~3%),
>
> We were unable to turn on "group metadata-cache". When we try get
> errors like
>
> # gluster volume set data-volume group metadata-cache
> '/var/lib/glusterd/groups/metadata-cache' file format not valid.
>
> Was metadata-cache available for gluster 3.7.11? We ask because
> the release notes for 3.11 mentions “Feature for
> metadata-caching/small file performance is production ready.”
> (https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/
> <https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>).
>
> Do any of these results suggest anything? If not, what further
> tests would be useful?
>
>
> Group metadata-cache is just a bunch of options one sets on a volume.
> So, You can set them manually using gluster cli. Following are the
> options and their values:
>
> performance.md-cache-timeout=600
> network.inode-lru-limit=50000
>
>
>
> Thanks
>
> Pat
>
>
>
>
> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:
>>
>>
>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <phaley at mit.edu
>> <mailto:phaley at mit.edu>> wrote:
>>
>>
>> Hi Raghavendra,
>>
>> Thanks for the suggestions. Our technician will be in on
>> Monday. We'll test then and let you know the results.
>>
>> One question I have, is the "group metadata-cache" option
>> supposed to directly impact the performance or is it to help
>> collect data? If the latter, where will the data be located?
>>
>>
>> It impacts performance.
>>
>>
>> Thanks again.
>>
>> Pat
>>
>>
>>
>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:
>>>
>>>
>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa
>>> <rgowdapp at redhat.com <mailto:rgowdapp at redhat.com>> wrote:
>>>
>>> For the case of writes to glusterfs mount,
>>>
>>> I saw in earlier conversations that there are too many
>>> lookups, but small number of writes. Since writes cached
>>> in write-behind would invalidate metadata cache, lookups
>>> won't be absorbed by md-cache. I am wondering what would
>>> results look like if we turn off performance.write-behind.
>>>
>>> @Pat,
>>>
>>> Can you set,
>>>
>>> # gluster volume set <volname> performance.write-behind off
>>>
>>>
>>> Please turn on "group metadata-cache" for write tests too.
>>>
>>>
>>> and redo the tests writing to glusterfs mount? Let us
>>> know about the results you see.
>>>
>>> regards,
>>> Raghavendra
>>>
>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa
>>> <rgowdapp at redhat.com <mailto:rgowdapp at redhat.com>> wrote:
>>>
>>>
>>>
>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra
>>> Gowdappa <rgowdapp at redhat.com
>>> <mailto:rgowdapp at redhat.com>> wrote:
>>>
>>> For the case of reading from Glusterfs mount,
>>> read-ahead should help. However, we've known
>>> issues with read-ahead[1][2]. To work around
>>> these, can you try with,
>>>
>>> 1. Turn off performance.open-behind
>>> #gluster volume set <volname>
>>> performance.open-behind off
>>>
>>> 2. enable group meta metadata-cache
>>> # gluster volume set <volname> group metadata-cache
>>>
>>>
>>> [1]
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1084508
>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
>>> [2]
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
>>>
>>>
>>>
>>>
>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley
>>> <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> We were recently revisiting our problems
>>> with the slowness of gluster writes
>>> (http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
>>> <http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
>>> Specifically we were testing the suggestions
>>> in a recent post
>>> (http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
>>> <http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
>>> The first two suggestions (specifying a
>>> negative-timeout in the mount settings or
>>> adding rpc-auth-allow-insecure to
>>> glusterd.vol) did not improve our
>>> performance, while setting
>>> "disperse.eager-lock off" provided a tiny
>>> (5%) speed-up.
>>>
>>> Some of the various tests we have tried
>>> earlier can be seen in the links below. Do
>>> any of the above observations suggest what
>>> we could try next to either improve the
>>> speed or debug the issue? Thanks
>>>
>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
>>> <http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
>>> <http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
>>>
>>> Pat
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> <mailto:phaley at mit.edu>
>>> Center for Ocean Engineering Phone: (617)
>>> 253-6824
>>> Dept. of Mechanical Engineering Fax:
>>> (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> <mailto:Gluster-users at gluster.org>
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>> <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>> <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>
>>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180629/4fb5e7ea/attachment.html>
More information about the Gluster-users
mailing list