[Gluster-users] Slow write times to gluster disk
Raghavendra Gowdappa
rgowdapp at redhat.com
Sat Jul 14 02:07:20 UTC 2018
On Fri, Jul 13, 2018 at 5:00 AM, Pat Haley <phaley at mit.edu> wrote:
>
> Hi Raghavendra,
>
> We were wondering if you have had a chance to look at this again, and if
> so, did you have any further suggestions?
>
Sorry Pat. Too much of work :). I'll be working on a patch today to make
sure read-ahead doesn't flush its cache due to fstats making this behavior
optional. You can try this patch and let us know about results.
Will let you know when patch is ready.
> Thanks
>
> Pat
>
>
>
> On 07/06/2018 01:27 AM, Raghavendra Gowdappa wrote:
>
>
>
> On Fri, Jul 6, 2018 at 5:29 AM, Pat Haley <phaley at mit.edu> wrote:
>
>>
>> Hi Raghavendra,
>>
>> Our technician may have some time to look at this issue tomorrow. Are
>> there any tests that you'd like to see?
>>
>
> Sorry. I've been busy with other things and was away from work for couple
> of days. It'll take me another 2 days to work on this issue again. So, most
> likely you'll have an update on this next week.
>
>
>> Thanks
>>
>> Pat
>>
>>
>>
>> On 06/29/2018 11:25 PM, Raghavendra Gowdappa wrote:
>>
>>
>>
>> On Fri, Jun 29, 2018 at 10:38 PM, Pat Haley <phaley at mit.edu> wrote:
>>
>>>
>>> Hi Raghavendra,
>>>
>>> We ran the tests (write tests) and I copied the log files for both the
>>> server and the client to http://mseas.mit.edu/download/
>>> phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace
>>> information you need? (If so, where should I look for it?)
>>>
>>
>> Nothing for now. I can see from logs that workaround is not helping.
>> fstat requests are not absorbed by md-cache and read-ahead is witnessing
>> them and flushing its read-ahead cache. I am investigating more on md-cache
>> (It also seems to be invalidating inodes quite frequently which actually
>> might be the root cause of seeing so many fstat requests from kernel). Will
>> post when I find anything relevant.
>>
>>
>>> Also the volume information you requested
>>>
>>> [root at mseas-data2 ~]# gluster volume info data-volume
>>>
>>> Volume Name: data-volume
>>> Type: Distribute
>>> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>>> Status: Started
>>> Number of Bricks: 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: mseas-data2:/mnt/brick1
>>> Brick2: mseas-data2:/mnt/brick2
>>> Options Reconfigured:
>>> diagnostics.client-log-level: TRACE
>>> network.inode-lru-limit: 50000
>>> performance.md-cache-timeout: 60
>>> performance.open-behind: off
>>> disperse.eager-lock: off
>>> auth.allow: *
>>> server.allow-insecure: on
>>> nfs.exports-auth-enable: on
>>> diagnostics.brick-sys-log-level: WARNING
>>> performance.readdir-ahead: on
>>> nfs.disable: on
>>> nfs.export-volumes: off
>>> [root at mseas-data2 ~]#
>>>
>>>
>>> On 06/29/2018 12:28 PM, Raghavendra Gowdappa wrote:
>>>
>>>
>>>
>>> On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley at mit.edu> wrote:
>>>
>>>>
>>>> Hi Raghavendra,
>>>>
>>>> Our technician was able to try the manual setting today. He found that
>>>> our upper limit for performance.md-cache-timeout was 60 not 600, so he
>>>> used that value, along with the network.inode-lru-limit=50000.
>>>>
>>>> The result was another small (~1%) increase in speed. Does this
>>>> suggest some addition tests/changes we could try?
>>>>
>>>
>>> Can you set gluster option diagnostics.client-log-level to TRACE and
>>> run sequential read tests again (with md-cache-timeout value of 60)?
>>>
>>> #gluster volume set <volname> diagnostics.client-log-level TRACE
>>>
>>> Also are you sure that open-behind was turned off? Can you give the
>>> output of,
>>>
>>> # gluster volume info <volname>
>>>
>>>
>>>> Thanks
>>>>
>>>> Pat
>>>>
>>>>
>>>>
>>>>
>>>> On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote:
>>>>
>>>>
>>>>
>>>> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu> wrote:
>>>>
>>>>>
>>>>> Hi Raghavendra,
>>>>>
>>>>> Setting the performance.write-behind off had a small improvement on
>>>>> the write speed (~3%),
>>>>>
>>>>> We were unable to turn on "group metadata-cache". When we try get
>>>>> errors like
>>>>>
>>>>> # gluster volume set data-volume group metadata-cache
>>>>> '/var/lib/glusterd/groups/metadata-cache' file format not valid.
>>>>>
>>>>> Was metadata-cache available for gluster 3.7.11? We ask because the
>>>>> release notes for 3.11 mentions “Feature for metadata-caching/small file
>>>>> performance is production ready.” (https://gluster.readthedocs.i
>>>>> o/en/latest/release-notes/3.11.0/).
>>>>>
>>>>> Do any of these results suggest anything? If not, what further tests
>>>>> would be useful?
>>>>>
>>>>
>>>> Group metadata-cache is just a bunch of options one sets on a volume.
>>>> So, You can set them manually using gluster cli. Following are the options
>>>> and their values:
>>>>
>>>> performance.md-cache-timeout=600
>>>> network.inode-lru-limit=50000
>>>>
>>>>
>>>>
>>>>> Thanks
>>>>>
>>>>> Pat
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <phaley at mit.edu> wrote:
>>>>>
>>>>>>
>>>>>> Hi Raghavendra,
>>>>>>
>>>>>> Thanks for the suggestions. Our technician will be in on Monday.
>>>>>> We'll test then and let you know the results.
>>>>>>
>>>>>> One question I have, is the "group metadata-cache" option supposed to
>>>>>> directly impact the performance or is it to help collect data? If the
>>>>>> latter, where will the data be located?
>>>>>>
>>>>>
>>>>> It impacts performance.
>>>>>
>>>>>
>>>>>> Thanks again.
>>>>>>
>>>>>> Pat
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
>>>>>> rgowdapp at redhat.com> wrote:
>>>>>>
>>>>>>> For the case of writes to glusterfs mount,
>>>>>>>
>>>>>>> I saw in earlier conversations that there are too many lookups, but
>>>>>>> small number of writes. Since writes cached in write-behind would
>>>>>>> invalidate metadata cache, lookups won't be absorbed by md-cache. I am
>>>>>>> wondering what would results look like if we turn off
>>>>>>> performance.write-behind.
>>>>>>>
>>>>>>> @Pat,
>>>>>>>
>>>>>>> Can you set,
>>>>>>>
>>>>>>> # gluster volume set <volname> performance.write-behind off
>>>>>>>
>>>>>>
>>>>>> Please turn on "group metadata-cache" for write tests too.
>>>>>>
>>>>>>
>>>>>>> and redo the tests writing to glusterfs mount? Let us know about the
>>>>>>> results you see.
>>>>>>>
>>>>>>> regards,
>>>>>>> Raghavendra
>>>>>>>
>>>>>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
>>>>>>> rgowdapp at redhat.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
>>>>>>>> rgowdapp at redhat.com> wrote:
>>>>>>>>
>>>>>>>>> For the case of reading from Glusterfs mount, read-ahead should
>>>>>>>>> help. However, we've known issues with read-ahead[1][2]. To work around
>>>>>>>>> these, can you try with,
>>>>>>>>>
>>>>>>>>> 1. Turn off performance.open-behind
>>>>>>>>> #gluster volume set <volname> performance.open-behind off
>>>>>>>>>
>>>>>>>>> 2. enable group meta metadata-cache
>>>>>>>>> # gluster volume set <volname> group metadata-cache
>>>>>>>>>
>>>>>>>>
>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
>>>>>>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley <phaley at mit.edu> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> We were recently revisiting our problems with the slowness of
>>>>>>>>>> gluster writes (http://lists.gluster.org/pipe
>>>>>>>>>> rmail/gluster-users/2017-April/030529.html). Specifically we
>>>>>>>>>> were testing the suggestions in a recent post (
>>>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2018-March
>>>>>>>>>> /033699.html). The first two suggestions (specifying a
>>>>>>>>>> negative-timeout in the mount settings or adding rpc-auth-allow-insecure to
>>>>>>>>>> glusterd.vol) did not improve our performance, while setting
>>>>>>>>>> "disperse.eager-lock off" provided a tiny (5%) speed-up.
>>>>>>>>>>
>>>>>>>>>> Some of the various tests we have tried earlier can be seen in
>>>>>>>>>> the links below. Do any of the above observations suggest what we could
>>>>>>>>>> try next to either improve the speed or debug the issue? Thanks
>>>>>>>>>>
>>>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/0
>>>>>>>>>> 31565.html
>>>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/03
>>>>>>>>>> 0937.html
>>>>>>>>>>
>>>>>>>>>> Pat
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>> 77 Massachusetts Avenue
>>>>>> Cambridge, MA 02139-4301
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>> Pat Haley Email: phaley at mit.edu
>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>> 77 Massachusetts Avenue
>>>>> Cambridge, MA 02139-4301
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>> Pat Haley Email: phaley at mit.edu
>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>> 77 Massachusetts Avenue
>>>> Cambridge, MA 02139-4301
>>>>
>>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email: phaley at mit.edu
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>>
>>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email: phaley at mit.edu
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213 http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180714/9f1eba38/attachment.html>
More information about the Gluster-users
mailing list