[Gluster-users] Slow write times to gluster disk
Pat Haley
phaley at mit.edu
Fri Jun 29 17:08:41 UTC 2018
Hi Raghavendra,
We ran the tests (write tests) and I copied the log files for both the
server and the client to
http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/ . Is
there any additional trace information you need? (If so, where should I
look for it?)
Also the volume information you requested
[root at mseas-data2 ~]# gluster volume info data-volume
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Options Reconfigured:
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
[root at mseas-data2 ~]#
On 06/29/2018 12:28 PM, Raghavendra Gowdappa wrote:
>
>
> On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley at mit.edu
> <mailto:phaley at mit.edu>> wrote:
>
>
> Hi Raghavendra,
>
> Our technician was able to try the manual setting today. He found
> that our upper limit for performance.md-cache-timeout was 60 not
> 600, so he used that value, along with the
> network.inode-lru-limit=50000.
>
> The result was another small (~1%) increase in speed. Does this
> suggest some addition tests/changes we could try?
>
>
> Can you set gluster option diagnostics.client-log-level to TRACE and
> run sequential read tests again (with md-cache-timeout value of 60)?
>
> #gluster volume set <volname> diagnostics.client-log-level TRACE
>
> Also are you sure that open-behind was turned off? Can you give the
> output of,
>
> # gluster volume info <volname>
>
>
> Thanks
>
> Pat
>
>
>
>
> On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote:
>>
>>
>> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu
>> <mailto:phaley at mit.edu>> wrote:
>>
>>
>> Hi Raghavendra,
>>
>> Setting the performance.write-behind off had a small
>> improvement on the write speed (~3%),
>>
>> We were unable to turn on "group metadata-cache". When we
>> try get errors like
>>
>> # gluster volume set data-volume group metadata-cache
>> '/var/lib/glusterd/groups/metadata-cache' file format not valid.
>>
>> Was metadata-cache available for gluster 3.7.11? We ask
>> because the release notes for 3.11 mentions “Feature for
>> metadata-caching/small file performance is production ready.”
>> (https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/
>> <https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>).
>>
>> Do any of these results suggest anything? If not, what
>> further tests would be useful?
>>
>>
>> Group metadata-cache is just a bunch of options one sets on a
>> volume. So, You can set them manually using gluster cli.
>> Following are the options and their values:
>>
>> performance.md-cache-timeout=600
>> network.inode-lru-limit=50000
>>
>>
>>
>> Thanks
>>
>> Pat
>>
>>
>>
>>
>> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:
>>>
>>>
>>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <phaley at mit.edu
>>> <mailto:phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi Raghavendra,
>>>
>>> Thanks for the suggestions. Our technician will be in
>>> on Monday. We'll test then and let you know the results.
>>>
>>> One question I have, is the "group metadata-cache"
>>> option supposed to directly impact the performance or is
>>> it to help collect data? If the latter, where will the
>>> data be located?
>>>
>>>
>>> It impacts performance.
>>>
>>>
>>> Thanks again.
>>>
>>> Pat
>>>
>>>
>>>
>>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:
>>>>
>>>>
>>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa
>>>> <rgowdapp at redhat.com <mailto:rgowdapp at redhat.com>> wrote:
>>>>
>>>> For the case of writes to glusterfs mount,
>>>>
>>>> I saw in earlier conversations that there are too
>>>> many lookups, but small number of writes. Since
>>>> writes cached in write-behind would invalidate
>>>> metadata cache, lookups won't be absorbed by
>>>> md-cache. I am wondering what would results look
>>>> like if we turn off performance.write-behind.
>>>>
>>>> @Pat,
>>>>
>>>> Can you set,
>>>>
>>>> # gluster volume set <volname>
>>>> performance.write-behind off
>>>>
>>>>
>>>> Please turn on "group metadata-cache" for write tests too.
>>>>
>>>>
>>>> and redo the tests writing to glusterfs mount? Let
>>>> us know about the results you see.
>>>>
>>>> regards,
>>>> Raghavendra
>>>>
>>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra
>>>> Gowdappa <rgowdapp at redhat.com
>>>> <mailto:rgowdapp at redhat.com>> wrote:
>>>>
>>>>
>>>>
>>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra
>>>> Gowdappa <rgowdapp at redhat.com
>>>> <mailto:rgowdapp at redhat.com>> wrote:
>>>>
>>>> For the case of reading from Glusterfs
>>>> mount, read-ahead should help. However,
>>>> we've known issues with read-ahead[1][2].
>>>> To work around these, can you try with,
>>>>
>>>> 1. Turn off performance.open-behind
>>>> #gluster volume set <volname>
>>>> performance.open-behind off
>>>>
>>>> 2. enable group meta metadata-cache
>>>> # gluster volume set <volname> group
>>>> metadata-cache
>>>>
>>>>
>>>> [1]
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1084508
>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
>>>> [2]
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley
>>>> <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> We were recently revisiting our
>>>> problems with the slowness of gluster
>>>> writes
>>>> (http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
>>>> Specifically we were testing the
>>>> suggestions in a recent post
>>>> (http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
>>>> <http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
>>>> The first two suggestions (specifying a
>>>> negative-timeout in the mount settings
>>>> or adding rpc-auth-allow-insecure to
>>>> glusterd.vol) did not improve our
>>>> performance, while setting
>>>> "disperse.eager-lock off" provided a
>>>> tiny (5%) speed-up.
>>>>
>>>> Some of the various tests we have tried
>>>> earlier can be seen in the links below.
>>>> Do any of the above observations
>>>> suggest what we could try next to
>>>> either improve the speed or debug the
>>>> issue? Thanks
>>>>
>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
>>>>
>>>> Pat
>>>>
>>>> --
>>>>
>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>> Pat Haley Email: phaley at mit.edu
>>>> <mailto:phaley at mit.edu>
>>>> Center for Ocean Engineering Phone:
>>>> (617) 253-6824
>>>> Dept. of Mechanical Engineering Fax:
>>>> (617) 253-8125
>>>> MIT, Room 5-213
>>>> http://web.mit.edu/phaley/www/
>>>> 77 Massachusetts Avenue
>>>> Cambridge, MA 02139-4301
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> <mailto:Gluster-users at gluster.org>
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>> <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>
>>>
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>>
>>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180629/58aefc6a/attachment.html>
More information about the Gluster-users
mailing list