[Gluster-users] Slow write times to gluster disk
Pat Haley
phaley at mit.edu
Thu Jul 5 23:59:58 UTC 2018
Hi Raghavendra,
Our technician may have some time to look at this issue tomorrow. Are
there any tests that you'd like to see?
Thanks
Pat
On 06/29/2018 11:25 PM, Raghavendra Gowdappa wrote:
>
>
> On Fri, Jun 29, 2018 at 10:38 PM, Pat Haley <phaley at mit.edu
> <mailto:phaley at mit.edu>> wrote:
>
>
> Hi Raghavendra,
>
> We ran the tests (write tests) and I copied the log files for both
> the server and the client to
> http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/
> <http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/> .
> Is there any additional trace information you need? (If so, where
> should I look for it?)
>
>
> Nothing for now. I can see from logs that workaround is not helping.
> fstat requests are not absorbed by md-cache and read-ahead is
> witnessing them and flushing its read-ahead cache. I am investigating
> more on md-cache (It also seems to be invalidating inodes quite
> frequently which actually might be the root cause of seeing so many
> fstat requests from kernel). Will post when I find anything relevant.
>
>
> Also the volume information you requested
>
> [root at mseas-data2 ~]# gluster volume info data-volume
>
> Volume Name: data-volume
> Type: Distribute
> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: mseas-data2:/mnt/brick1
> Brick2: mseas-data2:/mnt/brick2
> Options Reconfigured:
> diagnostics.client-log-level: TRACE
> network.inode-lru-limit: 50000
> performance.md-cache-timeout: 60
> performance.open-behind: off
> disperse.eager-lock: off
> auth.allow: *
> server.allow-insecure: on
> nfs.exports-auth-enable: on
> diagnostics.brick-sys-log-level: WARNING
> performance.readdir-ahead: on
> nfs.disable: on
> nfs.export-volumes: off
> [root at mseas-data2 ~]#
>
>
> On 06/29/2018 12:28 PM, Raghavendra Gowdappa wrote:
>>
>>
>> On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley at mit.edu
>> <mailto:phaley at mit.edu>> wrote:
>>
>>
>> Hi Raghavendra,
>>
>> Our technician was able to try the manual setting today. He
>> found that our upper limit for performance.md-cache-timeout
>> was 60 not 600, so he used that value, along with the
>> network.inode-lru-limit=50000.
>>
>> The result was another small (~1%) increase in speed. Does
>> this suggest some addition tests/changes we could try?
>>
>>
>> Can you set gluster option diagnostics.client-log-level to TRACE
>> and run sequential read tests again (with md-cache-timeout value
>> of 60)?
>>
>> #gluster volume set <volname> diagnostics.client-log-level TRACE
>>
>> Also are you sure that open-behind was turned off? Can you give
>> the output of,
>>
>> # gluster volume info <volname>
>>
>>
>> Thanks
>>
>> Pat
>>
>>
>>
>>
>> On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote:
>>>
>>>
>>> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu
>>> <mailto:phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi Raghavendra,
>>>
>>> Setting the performance.write-behind off had a small
>>> improvement on the write speed (~3%),
>>>
>>> We were unable to turn on "group metadata-cache". When
>>> we try get errors like
>>>
>>> # gluster volume set data-volume group metadata-cache
>>> '/var/lib/glusterd/groups/metadata-cache' file format
>>> not valid.
>>>
>>> Was metadata-cache available for gluster 3.7.11? We ask
>>> because the release notes for 3.11 mentions “Feature for
>>> metadata-caching/small file performance is production
>>> ready.”
>>> (https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/
>>> <https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>).
>>>
>>> Do any of these results suggest anything? If not, what
>>> further tests would be useful?
>>>
>>>
>>> Group metadata-cache is just a bunch of options one sets on
>>> a volume. So, You can set them manually using gluster cli.
>>> Following are the options and their values:
>>>
>>> performance.md-cache-timeout=600
>>> network.inode-lru-limit=50000
>>>
>>>
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>>
>>>
>>> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:
>>>>
>>>>
>>>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley
>>>> <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
>>>>
>>>>
>>>> Hi Raghavendra,
>>>>
>>>> Thanks for the suggestions. Our technician will be
>>>> in on Monday. We'll test then and let you know the
>>>> results.
>>>>
>>>> One question I have, is the "group metadata-cache"
>>>> option supposed to directly impact the performance
>>>> or is it to help collect data? If the latter, where
>>>> will the data be located?
>>>>
>>>>
>>>> It impacts performance.
>>>>
>>>>
>>>> Thanks again.
>>>>
>>>> Pat
>>>>
>>>>
>>>>
>>>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:
>>>>>
>>>>>
>>>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra
>>>>> Gowdappa <rgowdapp at redhat.com
>>>>> <mailto:rgowdapp at redhat.com>> wrote:
>>>>>
>>>>> For the case of writes to glusterfs mount,
>>>>>
>>>>> I saw in earlier conversations that there are
>>>>> too many lookups, but small number of writes.
>>>>> Since writes cached in write-behind would
>>>>> invalidate metadata cache, lookups won't be
>>>>> absorbed by md-cache. I am wondering what
>>>>> would results look like if we turn off
>>>>> performance.write-behind.
>>>>>
>>>>> @Pat,
>>>>>
>>>>> Can you set,
>>>>>
>>>>> # gluster volume set <volname>
>>>>> performance.write-behind off
>>>>>
>>>>>
>>>>> Please turn on "group metadata-cache" for write
>>>>> tests too.
>>>>>
>>>>>
>>>>> and redo the tests writing to glusterfs mount?
>>>>> Let us know about the results you see.
>>>>>
>>>>> regards,
>>>>> Raghavendra
>>>>>
>>>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra
>>>>> Gowdappa <rgowdapp at redhat.com
>>>>> <mailto:rgowdapp at redhat.com>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 21, 2018 at 8:32 AM,
>>>>> Raghavendra Gowdappa <rgowdapp at redhat.com
>>>>> <mailto:rgowdapp at redhat.com>> wrote:
>>>>>
>>>>> For the case of reading from Glusterfs
>>>>> mount, read-ahead should help.
>>>>> However, we've known issues with
>>>>> read-ahead[1][2]. To work around
>>>>> these, can you try with,
>>>>>
>>>>> 1. Turn off performance.open-behind
>>>>> #gluster volume set <volname>
>>>>> performance.open-behind off
>>>>>
>>>>> 2. enable group meta metadata-cache
>>>>> # gluster volume set <volname> group
>>>>> metadata-cache
>>>>>
>>>>>
>>>>> [1]
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1084508
>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
>>>>> [2]
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat
>>>>> Haley <phaley at mit.edu
>>>>> <mailto:phaley at mit.edu>> wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> We were recently revisiting our
>>>>> problems with the slowness of
>>>>> gluster writes
>>>>> (http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
>>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
>>>>> Specifically we were testing the
>>>>> suggestions in a recent post
>>>>> (http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
>>>>> <http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
>>>>> The first two suggestions
>>>>> (specifying a negative-timeout in
>>>>> the mount settings or adding
>>>>> rpc-auth-allow-insecure to
>>>>> glusterd.vol) did not improve our
>>>>> performance, while setting
>>>>> "disperse.eager-lock off" provided
>>>>> a tiny (5%) speed-up.
>>>>>
>>>>> Some of the various tests we have
>>>>> tried earlier can be seen in the
>>>>> links below. Do any of the above
>>>>> observations suggest what we could
>>>>> try next to either improve the
>>>>> speed or debug the issue? Thanks
>>>>>
>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
>>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
>>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
>>>>>
>>>>> Pat
>>>>>
>>>>> --
>>>>>
>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>> Pat Haley Email:
>>>>> phaley at mit.edu <mailto:phaley at mit.edu>
>>>>> Center for Ocean Engineering
>>>>> Phone: (617) 253-6824
>>>>> Dept. of Mechanical Engineering
>>>>> Fax: (617) 253-8125
>>>>> MIT, Room 5-213
>>>>> http://web.mit.edu/phaley/www/
>>>>> 77 Massachusetts Avenue
>>>>> Cambridge, MA 02139-4301
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> <mailto:Gluster-users at gluster.org>
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>> 77 Massachusetts Avenue
>>>> Cambridge, MA 02139-4301
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> <mailto:Gluster-users at gluster.org>
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>
>>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>>
>>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180705/1a20b61b/attachment.html>
More information about the Gluster-users
mailing list