[Gluster-users] Slow write times to gluster disk
Raghavendra Gowdappa
rgowdapp at redhat.com
Fri Jun 29 16:28:35 UTC 2018
On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley at mit.edu> wrote:
>
> Hi Raghavendra,
>
> Our technician was able to try the manual setting today. He found that
> our upper limit for performance.md-cache-timeout was 60 not 600, so he
> used that value, along with the network.inode-lru-limit=50000.
>
> The result was another small (~1%) increase in speed. Does this suggest
> some addition tests/changes we could try?
>
Can you set gluster option diagnostics.client-log-level to TRACE and run
sequential read tests again (with md-cache-timeout value of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you give the output
of,
# gluster volume info <volname>
> Thanks
>
> Pat
>
>
>
>
> On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote:
>
>
>
> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu> wrote:
>
>>
>> Hi Raghavendra,
>>
>> Setting the performance.write-behind off had a small improvement on the
>> write speed (~3%),
>>
>> We were unable to turn on "group metadata-cache". When we try get errors
>> like
>>
>> # gluster volume set data-volume group metadata-cache
>> '/var/lib/glusterd/groups/metadata-cache' file format not valid.
>>
>> Was metadata-cache available for gluster 3.7.11? We ask because the
>> release notes for 3.11 mentions “Feature for metadata-caching/small file
>> performance is production ready.” (https://gluster.readthedocs.i
>> o/en/latest/release-notes/3.11.0/).
>>
>> Do any of these results suggest anything? If not, what further tests
>> would be useful?
>>
>
> Group metadata-cache is just a bunch of options one sets on a volume. So,
> You can set them manually using gluster cli. Following are the options and
> their values:
>
> performance.md-cache-timeout=600
> network.inode-lru-limit=50000
>
>
>
>> Thanks
>>
>> Pat
>>
>>
>>
>>
>> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:
>>
>>
>>
>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <phaley at mit.edu> wrote:
>>
>>>
>>> Hi Raghavendra,
>>>
>>> Thanks for the suggestions. Our technician will be in on Monday. We'll
>>> test then and let you know the results.
>>>
>>> One question I have, is the "group metadata-cache" option supposed to
>>> directly impact the performance or is it to help collect data? If the
>>> latter, where will the data be located?
>>>
>>
>> It impacts performance.
>>
>>
>>> Thanks again.
>>>
>>> Pat
>>>
>>>
>>>
>>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:
>>>
>>>
>>>
>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
>>> rgowdapp at redhat.com> wrote:
>>>
>>>> For the case of writes to glusterfs mount,
>>>>
>>>> I saw in earlier conversations that there are too many lookups, but
>>>> small number of writes. Since writes cached in write-behind would
>>>> invalidate metadata cache, lookups won't be absorbed by md-cache. I am
>>>> wondering what would results look like if we turn off
>>>> performance.write-behind.
>>>>
>>>> @Pat,
>>>>
>>>> Can you set,
>>>>
>>>> # gluster volume set <volname> performance.write-behind off
>>>>
>>>
>>> Please turn on "group metadata-cache" for write tests too.
>>>
>>>
>>>> and redo the tests writing to glusterfs mount? Let us know about the
>>>> results you see.
>>>>
>>>> regards,
>>>> Raghavendra
>>>>
>>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
>>>> rgowdapp at redhat.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
>>>>> rgowdapp at redhat.com> wrote:
>>>>>
>>>>>> For the case of reading from Glusterfs mount, read-ahead should help.
>>>>>> However, we've known issues with read-ahead[1][2]. To work around these,
>>>>>> can you try with,
>>>>>>
>>>>>> 1. Turn off performance.open-behind
>>>>>> #gluster volume set <volname> performance.open-behind off
>>>>>>
>>>>>> 2. enable group meta metadata-cache
>>>>>> # gluster volume set <volname> group metadata-cache
>>>>>>
>>>>>
>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
>>>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley <phaley at mit.edu> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We were recently revisiting our problems with the slowness of
>>>>>>> gluster writes (http://lists.gluster.org/pipe
>>>>>>> rmail/gluster-users/2017-April/030529.html). Specifically we were
>>>>>>> testing the suggestions in a recent post (
>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2018-March
>>>>>>> /033699.html). The first two suggestions (specifying a
>>>>>>> negative-timeout in the mount settings or adding rpc-auth-allow-insecure to
>>>>>>> glusterd.vol) did not improve our performance, while setting
>>>>>>> "disperse.eager-lock off" provided a tiny (5%) speed-up.
>>>>>>>
>>>>>>> Some of the various tests we have tried earlier can be seen in the
>>>>>>> links below. Do any of the above observations suggest what we could try
>>>>>>> next to either improve the speed or debug the issue? Thanks
>>>>>>>
>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/0
>>>>>>> 31565.html
>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/03
>>>>>>> 0937.html
>>>>>>>
>>>>>>> Pat
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>>> 77 Massachusetts Avenue
>>>>>>> Cambridge, MA 02139-4301
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email: phaley at mit.edu
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>>
>>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email: phaley at mit.edu
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213 http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180629/b428c079/attachment.html>
More information about the Gluster-users
mailing list