[Gluster-users] Slow write times to gluster disk
Raghavendra Gowdappa
rgowdapp at redhat.com
Sat Jun 30 03:25:22 UTC 2018
On Fri, Jun 29, 2018 at 10:38 PM, Pat Haley <phaley at mit.edu> wrote:
>
> Hi Raghavendra,
>
> We ran the tests (write tests) and I copied the log files for both the
> server and the client to http://mseas.mit.edu/download/
> phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace
> information you need? (If so, where should I look for it?)
>
Nothing for now. I can see from logs that workaround is not helping. fstat
requests are not absorbed by md-cache and read-ahead is witnessing them and
flushing its read-ahead cache. I am investigating more on md-cache (It also
seems to be invalidating inodes quite frequently which actually might be
the root cause of seeing so many fstat requests from kernel). Will post
when I find anything relevant.
> Also the volume information you requested
>
> [root at mseas-data2 ~]# gluster volume info data-volume
>
> Volume Name: data-volume
> Type: Distribute
> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: mseas-data2:/mnt/brick1
> Brick2: mseas-data2:/mnt/brick2
> Options Reconfigured:
> diagnostics.client-log-level: TRACE
> network.inode-lru-limit: 50000
> performance.md-cache-timeout: 60
> performance.open-behind: off
> disperse.eager-lock: off
> auth.allow: *
> server.allow-insecure: on
> nfs.exports-auth-enable: on
> diagnostics.brick-sys-log-level: WARNING
> performance.readdir-ahead: on
> nfs.disable: on
> nfs.export-volumes: off
> [root at mseas-data2 ~]#
>
>
> On 06/29/2018 12:28 PM, Raghavendra Gowdappa wrote:
>
>
>
> On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley at mit.edu> wrote:
>
>>
>> Hi Raghavendra,
>>
>> Our technician was able to try the manual setting today. He found that
>> our upper limit for performance.md-cache-timeout was 60 not 600, so he
>> used that value, along with the network.inode-lru-limit=50000.
>>
>> The result was another small (~1%) increase in speed. Does this suggest
>> some addition tests/changes we could try?
>>
>
> Can you set gluster option diagnostics.client-log-level to TRACE and run
> sequential read tests again (with md-cache-timeout value of 60)?
>
> #gluster volume set <volname> diagnostics.client-log-level TRACE
>
> Also are you sure that open-behind was turned off? Can you give the output
> of,
>
> # gluster volume info <volname>
>
>
>> Thanks
>>
>> Pat
>>
>>
>>
>>
>> On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote:
>>
>>
>>
>> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu> wrote:
>>
>>>
>>> Hi Raghavendra,
>>>
>>> Setting the performance.write-behind off had a small improvement on the
>>> write speed (~3%),
>>>
>>> We were unable to turn on "group metadata-cache". When we try get
>>> errors like
>>>
>>> # gluster volume set data-volume group metadata-cache
>>> '/var/lib/glusterd/groups/metadata-cache' file format not valid.
>>>
>>> Was metadata-cache available for gluster 3.7.11? We ask because the
>>> release notes for 3.11 mentions “Feature for metadata-caching/small file
>>> performance is production ready.” (https://gluster.readthedocs.i
>>> o/en/latest/release-notes/3.11.0/).
>>>
>>> Do any of these results suggest anything? If not, what further tests
>>> would be useful?
>>>
>>
>> Group metadata-cache is just a bunch of options one sets on a volume. So,
>> You can set them manually using gluster cli. Following are the options and
>> their values:
>>
>> performance.md-cache-timeout=600
>> network.inode-lru-limit=50000
>>
>>
>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>>
>>>
>>> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:
>>>
>>>
>>>
>>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <phaley at mit.edu> wrote:
>>>
>>>>
>>>> Hi Raghavendra,
>>>>
>>>> Thanks for the suggestions. Our technician will be in on Monday.
>>>> We'll test then and let you know the results.
>>>>
>>>> One question I have, is the "group metadata-cache" option supposed to
>>>> directly impact the performance or is it to help collect data? If the
>>>> latter, where will the data be located?
>>>>
>>>
>>> It impacts performance.
>>>
>>>
>>>> Thanks again.
>>>>
>>>> Pat
>>>>
>>>>
>>>>
>>>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:
>>>>
>>>>
>>>>
>>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
>>>> rgowdapp at redhat.com> wrote:
>>>>
>>>>> For the case of writes to glusterfs mount,
>>>>>
>>>>> I saw in earlier conversations that there are too many lookups, but
>>>>> small number of writes. Since writes cached in write-behind would
>>>>> invalidate metadata cache, lookups won't be absorbed by md-cache. I am
>>>>> wondering what would results look like if we turn off
>>>>> performance.write-behind.
>>>>>
>>>>> @Pat,
>>>>>
>>>>> Can you set,
>>>>>
>>>>> # gluster volume set <volname> performance.write-behind off
>>>>>
>>>>
>>>> Please turn on "group metadata-cache" for write tests too.
>>>>
>>>>
>>>>> and redo the tests writing to glusterfs mount? Let us know about the
>>>>> results you see.
>>>>>
>>>>> regards,
>>>>> Raghavendra
>>>>>
>>>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
>>>>> rgowdapp at redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
>>>>>> rgowdapp at redhat.com> wrote:
>>>>>>
>>>>>>> For the case of reading from Glusterfs mount, read-ahead should
>>>>>>> help. However, we've known issues with read-ahead[1][2]. To work around
>>>>>>> these, can you try with,
>>>>>>>
>>>>>>> 1. Turn off performance.open-behind
>>>>>>> #gluster volume set <volname> performance.open-behind off
>>>>>>>
>>>>>>> 2. enable group meta metadata-cache
>>>>>>> # gluster volume set <volname> group metadata-cache
>>>>>>>
>>>>>>
>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
>>>>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley <phaley at mit.edu> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We were recently revisiting our problems with the slowness of
>>>>>>>> gluster writes (http://lists.gluster.org/pipe
>>>>>>>> rmail/gluster-users/2017-April/030529.html). Specifically we were
>>>>>>>> testing the suggestions in a recent post (
>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2018-March
>>>>>>>> /033699.html). The first two suggestions (specifying a
>>>>>>>> negative-timeout in the mount settings or adding rpc-auth-allow-insecure to
>>>>>>>> glusterd.vol) did not improve our performance, while setting
>>>>>>>> "disperse.eager-lock off" provided a tiny (5%) speed-up.
>>>>>>>>
>>>>>>>> Some of the various tests we have tried earlier can be seen in the
>>>>>>>> links below. Do any of the above observations suggest what we could try
>>>>>>>> next to either improve the speed or debug the issue? Thanks
>>>>>>>>
>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/0
>>>>>>>> 31565.html
>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/03
>>>>>>>> 0937.html
>>>>>>>>
>>>>>>>> Pat
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>>>> 77 Massachusetts Avenue
>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>> Pat Haley Email: phaley at mit.edu
>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>> 77 Massachusetts Avenue
>>>> Cambridge, MA 02139-4301
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email: phaley at mit.edu
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>>
>>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email: phaley at mit.edu
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213 http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180630/b79394f1/attachment.html>
More information about the Gluster-users
mailing list