[Gluster-users] Slow write times to gluster disk

Pat Haley phaley at mit.edu
Thu Jul 5 23:59:58 UTC 2018


Hi Raghavendra,

Our technician may have some time to look at this issue tomorrow. Are 
there any tests that you'd like to see?

Thanks

Pat


On 06/29/2018 11:25 PM, Raghavendra Gowdappa wrote:
>
>
> On Fri, Jun 29, 2018 at 10:38 PM, Pat Haley <phaley at mit.edu 
> <mailto:phaley at mit.edu>> wrote:
>
>
>     Hi Raghavendra,
>
>     We ran the tests (write tests) and I copied the log files for both
>     the server and the client to
>     http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/
>     <http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/> . 
>     Is there any additional trace information you need? (If so, where
>     should I look for it?)
>
>
> Nothing for now. I can see from logs that workaround is not helping. 
> fstat requests are not absorbed by md-cache and read-ahead is 
> witnessing them and flushing its read-ahead cache. I am investigating 
> more on md-cache (It also seems to be invalidating inodes quite 
> frequently which actually might be the root cause of seeing so many 
> fstat requests from kernel). Will post when I find anything relevant.
>
>
>     Also the volume information you requested
>
>     [root at mseas-data2 ~]# gluster volume info data-volume
>
>     Volume Name: data-volume
>     Type: Distribute
>     Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>     Status: Started
>     Number of Bricks: 2
>     Transport-type: tcp
>     Bricks:
>     Brick1: mseas-data2:/mnt/brick1
>     Brick2: mseas-data2:/mnt/brick2
>     Options Reconfigured:
>     diagnostics.client-log-level: TRACE
>     network.inode-lru-limit: 50000
>     performance.md-cache-timeout: 60
>     performance.open-behind: off
>     disperse.eager-lock: off
>     auth.allow: *
>     server.allow-insecure: on
>     nfs.exports-auth-enable: on
>     diagnostics.brick-sys-log-level: WARNING
>     performance.readdir-ahead: on
>     nfs.disable: on
>     nfs.export-volumes: off
>     [root at mseas-data2 ~]#
>
>
>     On 06/29/2018 12:28 PM, Raghavendra Gowdappa wrote:
>>
>>
>>     On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley at mit.edu
>>     <mailto:phaley at mit.edu>> wrote:
>>
>>
>>         Hi Raghavendra,
>>
>>         Our technician was able to try the manual setting today.  He
>>         found that our upper limit for performance.md-cache-timeout
>>         was 60 not 600, so he used that value, along with the
>>         network.inode-lru-limit=50000.
>>
>>         The result was another small (~1%) increase in speed.  Does
>>         this suggest some addition tests/changes we could try?
>>
>>
>>     Can you set gluster option diagnostics.client-log-level to TRACE 
>>     and run sequential read tests again (with md-cache-timeout value
>>     of 60)?
>>
>>     #gluster volume set <volname> diagnostics.client-log-level TRACE
>>
>>     Also are you sure that open-behind was turned off? Can you give
>>     the output of,
>>
>>     # gluster volume info <volname>
>>
>>
>>         Thanks
>>
>>         Pat
>>
>>
>>
>>
>>         On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote:
>>>
>>>
>>>         On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu
>>>         <mailto:phaley at mit.edu>> wrote:
>>>
>>>
>>>             Hi Raghavendra,
>>>
>>>             Setting the performance.write-behind off had a small
>>>             improvement on the write speed (~3%),
>>>
>>>             We were unable to turn on "group metadata-cache".  When
>>>             we try get errors like
>>>
>>>             # gluster volume set data-volume group metadata-cache
>>>             '/var/lib/glusterd/groups/metadata-cache' file format
>>>             not valid.
>>>
>>>             Was metadata-cache available for gluster 3.7.11? We ask
>>>             because the release notes for 3.11 mentions “Feature for
>>>             metadata-caching/small file performance is production
>>>             ready.”
>>>             (https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/
>>>             <https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>).
>>>
>>>             Do any of these results suggest anything?  If not, what
>>>             further tests would be useful?
>>>
>>>
>>>         Group metadata-cache is just a bunch of options one sets on
>>>         a volume. So, You can set them manually using gluster cli.
>>>         Following are the options and their values:
>>>
>>>         performance.md-cache-timeout=600
>>>         network.inode-lru-limit=50000
>>>
>>>
>>>
>>>             Thanks
>>>
>>>             Pat
>>>
>>>
>>>
>>>
>>>             On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:
>>>>
>>>>
>>>>             On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley
>>>>             <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
>>>>
>>>>
>>>>                 Hi Raghavendra,
>>>>
>>>>                 Thanks for the suggestions. Our technician will be
>>>>                 in on Monday.  We'll test then and let you know the
>>>>                 results.
>>>>
>>>>                 One question I have, is the "group metadata-cache"
>>>>                 option supposed to directly impact the performance
>>>>                 or is it to help collect data? If the latter, where
>>>>                 will the data be located?
>>>>
>>>>
>>>>             It impacts performance.
>>>>
>>>>
>>>>                 Thanks again.
>>>>
>>>>                 Pat
>>>>
>>>>
>>>>
>>>>                 On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:
>>>>>
>>>>>
>>>>>                 On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra
>>>>>                 Gowdappa <rgowdapp at redhat.com
>>>>>                 <mailto:rgowdapp at redhat.com>> wrote:
>>>>>
>>>>>                     For the case of writes to glusterfs mount,
>>>>>
>>>>>                     I saw in earlier conversations that there are
>>>>>                     too many lookups, but small number of writes.
>>>>>                     Since writes cached in write-behind would
>>>>>                     invalidate metadata cache, lookups won't be
>>>>>                     absorbed by md-cache. I am wondering what
>>>>>                     would results look like if we turn off
>>>>>                     performance.write-behind.
>>>>>
>>>>>                     @Pat,
>>>>>
>>>>>                     Can you set,
>>>>>
>>>>>                     # gluster volume set <volname>
>>>>>                     performance.write-behind off
>>>>>
>>>>>
>>>>>                 Please turn on "group metadata-cache" for write
>>>>>                 tests too.
>>>>>
>>>>>
>>>>>                     and redo the tests writing to glusterfs mount?
>>>>>                     Let us know about the results you see.
>>>>>
>>>>>                     regards,
>>>>>                     Raghavendra
>>>>>
>>>>>                     On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra
>>>>>                     Gowdappa <rgowdapp at redhat.com
>>>>>                     <mailto:rgowdapp at redhat.com>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>                         On Thu, Jun 21, 2018 at 8:32 AM,
>>>>>                         Raghavendra Gowdappa <rgowdapp at redhat.com
>>>>>                         <mailto:rgowdapp at redhat.com>> wrote:
>>>>>
>>>>>                             For the case of reading from Glusterfs
>>>>>                             mount, read-ahead should help.
>>>>>                             However, we've known issues with
>>>>>                             read-ahead[1][2]. To work around
>>>>>                             these, can you try with,
>>>>>
>>>>>                             1. Turn off performance.open-behind
>>>>>                             #gluster volume set <volname>
>>>>>                             performance.open-behind off
>>>>>
>>>>>                             2. enable group meta metadata-cache
>>>>>                             # gluster volume set <volname> group
>>>>>                             metadata-cache
>>>>>
>>>>>
>>>>>                         [1]
>>>>>                         https://bugzilla.redhat.com/show_bug.cgi?id=1084508
>>>>>                         <https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
>>>>>                         [2]
>>>>>                         https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>>>>>                         <https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                             On Thu, Jun 21, 2018 at 5:00 AM, Pat
>>>>>                             Haley <phaley at mit.edu
>>>>>                             <mailto:phaley at mit.edu>> wrote:
>>>>>
>>>>>
>>>>>                                 Hi,
>>>>>
>>>>>                                 We were recently revisiting our
>>>>>                                 problems with the slowness of
>>>>>                                 gluster writes
>>>>>                                 (http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
>>>>>                                 <http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
>>>>>                                 Specifically we were testing the
>>>>>                                 suggestions in a recent post
>>>>>                                 (http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
>>>>>                                 <http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
>>>>>                                 The first two suggestions
>>>>>                                 (specifying a negative-timeout in
>>>>>                                 the mount settings or adding
>>>>>                                 rpc-auth-allow-insecure to
>>>>>                                 glusterd.vol) did not improve our
>>>>>                                 performance, while setting
>>>>>                                 "disperse.eager-lock off" provided
>>>>>                                 a tiny (5%) speed-up.
>>>>>
>>>>>                                 Some of the various tests we have
>>>>>                                 tried earlier can be seen in the
>>>>>                                 links below. Do any of the above
>>>>>                                 observations suggest what we could
>>>>>                                 try next to either improve the
>>>>>                                 speed or debug the issue? Thanks
>>>>>
>>>>>                                 http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
>>>>>                                 <http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
>>>>>                                 http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
>>>>>                                 <http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
>>>>>
>>>>>                                 Pat
>>>>>
>>>>>                                 -- 
>>>>>
>>>>>                                 -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>                                 Pat Haley       Email:
>>>>>                                 phaley at mit.edu <mailto:phaley at mit.edu>
>>>>>                                 Center for Ocean Engineering  
>>>>>                                  Phone: (617) 253-6824
>>>>>                                 Dept. of Mechanical Engineering
>>>>>                                 Fax:    (617) 253-8125
>>>>>                                 MIT, Room 5-213
>>>>>                                 http://web.mit.edu/phaley/www/
>>>>>                                 77 Massachusetts Avenue
>>>>>                                 Cambridge, MA 02139-4301
>>>>>
>>>>>                                 _______________________________________________
>>>>>                                 Gluster-users mailing list
>>>>>                                 Gluster-users at gluster.org
>>>>>                                 <mailto:Gluster-users at gluster.org>
>>>>>                                 http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>                                 <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>                 -- 
>>>>
>>>>                 -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>                 Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>>>                 Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>                 Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>                 MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>                 77 Massachusetts Avenue
>>>>                 Cambridge, MA  02139-4301
>>>>
>>>>
>>>>                 _______________________________________________
>>>>                 Gluster-users mailing list
>>>>                 Gluster-users at gluster.org
>>>>                 <mailto:Gluster-users at gluster.org>
>>>>                 http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>                 <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>
>>>>
>>>
>>>             -- 
>>>
>>>             -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>             Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>>             Center for Ocean Engineering       Phone:  (617) 253-6824
>>>             Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>             MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>             77 Massachusetts Avenue
>>>             Cambridge, MA  02139-4301
>>>
>>>
>>
>>         -- 
>>
>>         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>         Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>         Center for Ocean Engineering       Phone:  (617) 253-6824
>>         Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>         MIT, Room 5-213http://web.mit.edu/phaley/www/
>>         77 Massachusetts Avenue
>>         Cambridge, MA  02139-4301
>>
>>
>
>     -- 
>
>     -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>     Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>     Center for Ocean Engineering       Phone:  (617) 253-6824
>     Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>     MIT, Room 5-213http://web.mit.edu/phaley/www/
>     77 Massachusetts Avenue
>     Cambridge, MA  02139-4301
>
>

-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180705/1a20b61b/attachment.html>


More information about the Gluster-users mailing list