[Gluster-users] Slow write times to gluster disk

Tue Jun 26 01:39:15 UTC 2018

On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu> wrote:

>
> Hi Raghavendra,
>
> Setting the performance.write-behind off had a small improvement on the
> write speed (~3%),
>
> We were unable to turn on "group metadata-cache".  When we try get errors
> like
>
> # gluster volume set data-volume group metadata-cache
> '/var/lib/glusterd/groups/metadata-cache' file format not valid.
>
> Was metadata-cache available for gluster 3.7.11? We ask because the
> release notes for 3.11 mentions “Feature for metadata-caching/small file
> performance is production ready.” (https://gluster.readthedocs.
> io/en/latest/release-notes/3.11.0/).
>
> Do any of these results suggest anything?  If not, what further tests
> would be useful?
>

Group metadata-cache is just a bunch of options one sets on a volume. So,
You can set them manually using gluster cli. Following are the options and
their values:

performance.md-cache-timeout=600
network.inode-lru-limit=50000

> Thanks
>
> Pat
>
>
>
>
> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:
>
>
>
> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <phaley at mit.edu> wrote:
>
>>
>> Hi Raghavendra,
>>
>> Thanks for the suggestions.  Our technician will be in on Monday.  We'll
>> test then and let you know the results.
>>
>> One question I have, is the "group metadata-cache" option supposed to
>> directly impact the performance or is it to help collect data?  If the
>> latter, where will the data be located?
>>
>
> It impacts performance.
>
>
>> Thanks again.
>>
>> Pat
>>
>>
>>
>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:
>>
>>
>>
>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
>> rgowdapp at redhat.com> wrote:
>>
>>> For the case of writes to glusterfs mount,
>>>
>>> I saw in earlier conversations that there are too many lookups, but
>>> small number of writes. Since writes cached in write-behind would
>>> invalidate metadata cache, lookups won't be absorbed by md-cache. I am
>>> wondering what would results look like if we turn off
>>> performance.write-behind.
>>>
>>> @Pat,
>>>
>>> Can you set,
>>>
>>> # gluster volume set <volname> performance.write-behind off
>>>
>>
>> Please turn on "group metadata-cache" for write tests too.
>>
>>
>>> and redo the tests writing to glusterfs mount? Let us know about the
>>> results you see.
>>>
>>> regards,
>>> Raghavendra
>>>
>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
>>> rgowdapp at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
>>>> rgowdapp at redhat.com> wrote:
>>>>
>>>>> For the case of reading from Glusterfs mount, read-ahead should help.
>>>>> However, we've known issues with read-ahead[1][2]. To work around these,
>>>>> can you try with,
>>>>>
>>>>> 1. Turn off performance.open-behind
>>>>> #gluster volume set <volname> performance.open-behind off
>>>>>
>>>>> 2. enable group meta metadata-cache
>>>>> # gluster volume set <volname> group metadata-cache
>>>>>
>>>>
>>>> [1]  https://bugzilla.redhat.com/show_bug.cgi?id=1084508
>>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>>>>
>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley <phaley at mit.edu> wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We were recently revisiting our problems with the slowness of gluster
>>>>>> writes (http://lists.gluster.org/pipermail/gluster-users/2017-April
>>>>>> /030529.html). Specifically we were testing the suggestions in a
>>>>>> recent post (http://lists.gluster.org/pipe
>>>>>> rmail/gluster-users/2018-March/033699.html). The first two
>>>>>> suggestions (specifying a negative-timeout in the mount settings or adding
>>>>>> rpc-auth-allow-insecure to glusterd.vol) did not improve our performance,
>>>>>> while setting "disperse.eager-lock off" provided a tiny (5%) speed-up.
>>>>>>
>>>>>> Some of the various tests we have tried earlier can be seen in the
>>>>>> links below.  Do any of the above observations suggest what we could try
>>>>>> next to either improve the speed or debug the issue?  Thanks
>>>>>>
>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/0
>>>>>> 31565.html
>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
>>>>>>
>>>>>> Pat
>>>>>>
>>>>>> --
>>>>>>
>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>> Pat Haley                          Email:  phaley at mit.edu
>>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>>>>>> 77 Massachusetts Avenue
>>>>>> Cambridge, MA  02139-4301
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley                          Email:  phaley at mit.edu
>> Center for Ocean Engineering       Phone:  (617) 253-6824
>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA  02139-4301
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley                          Email:  phaley at mit.edu
> Center for Ocean Engineering       Phone:  (617) 253-6824
> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180626/ab004c50/attachment.html>