[Gluster-users] Speed up heal performance

Wed Oct 14 17:13:50 UTC 2015

Hi Pranith,

Will this patch improve the heal performance on distributed disperse
volume?. Currently we are getting 10MB/s heal performance on 10G backed
network. SHD daemon takes 5 days to complete the heal operation for single
4TB( 3.5 TB data) disk failure.

Regards,
Backer

On Wed, Oct 14, 2015 at 9:08 PM, Ben Turner <bturner at redhat.com> wrote:

> ----- Original Message -----
> > From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> > To: "Ben Turner" <bturner at redhat.com>, "Humble Devassy Chirammal" <
> humble.devassy at gmail.com>, "Atin Mukherjee"
> > <atin.mukherjee83 at gmail.com>
> > Cc: "gluster-users" <gluster-users at gluster.org>
> > Sent: Wednesday, October 14, 2015 1:39:14 AM
> > Subject: Re: [Gluster-users] Speed up heal performance
> >
> >
> >
> > On 10/13/2015 07:11 PM, Ben Turner wrote:
> > > ----- Original Message -----
> > >> From: "Humble Devassy Chirammal" <humble.devassy at gmail.com>
> > >> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> > >> Cc: "Ben Turner" <bturner at redhat.com>, "gluster-users"
> > >> <gluster-users at gluster.org>
> > >> Sent: Tuesday, October 13, 2015 6:14:46 AM
> > >> Subject: Re: [Gluster-users] Speed up heal performance
> > >>
> > >>> Good news is we already have a WIP patch review.glusterd.org/10851
> to
> > >> introduce multi threaded shd. Credits to Richard/Shreyas from
> facebook for
> > >> this. IIRC, we also have a BZ for the same
> > >> Isnt it the same bugzilla (
> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
> > >> commit log?
> > > @Lindsay - No need for a BZ, the above BZ should suffice.
> > >
> > > @Anyone - In the commit I see:
> > >
> > >          { .key        = "cluster.shd-max-threads",
> > >            .voltype    = "cluster/replicate",
> > >            .option     = "shd-max-threads",
> > >            .op_version = 1,
> > >            .flags      = OPT_FLAG_CLIENT_OPT
> > >          },
> > >          { .key        = "cluster.shd-thread-batch-size",
> > >            .voltype    = "cluster/replicate",
> > >            .option     = "shd-thread-batch-size",
> > >            .op_version = 1,
> > >            .flags      = OPT_FLAG_CLIENT_OPT
> > >          },
> > >
> > > So we can tune max threads and thread batch size?  I understand max
> > > threads, but what is batch size?  In my testing on 10G NICs with a
> backend
> > > that will service 10G throughput I see about 1.5 GB per minute of SH
> > > throughput.  To Lindsay's other point, will this patch improve SH
> > > throughput?  My systems can write at 1.5 GB / Sec and NICs can to 1.2
> GB /
> > > sec but I only see ~1.5 GB per _minute_ of SH throughput.  If we can
> not
> > > only make SH multi threaded, but improve the performance of a single
> > > thread that would be awesome.  Super bonus points if we can have some
> sort
> > > of tunible that can limit the bandwidth each thread can consume.  It
> would
> > > be great to be able to crank things up when the systems aren't busy and
> > > slow things down when load increases.
> > This patch is not merged because I thought we needed throttling feature
> > to go in before we can merge this for better control of the self-heal
> > speed. We are doing that for 3.8. So expect to see both of these for 3.8.
>
> Great news!  You da man Pranith, next time I am on your side of the world
> beers are on me :)
>
> -b
>
> >
> > Pranith
> > >
> > > -b
> > >
> > >
> > >> --Humble
> > >>
> > >>
> > >> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee
> > >> <atin.mukherjee83 at gmail.com>
> > >> wrote:
> > >>
> > >>> -Atin
> > >>> Sent from one plus one
> > >>> On Oct 13, 2015 3:16 AM, "Ben Turner" <bturner at redhat.com> wrote:
> > >>>> ----- Original Message -----
> > >>>>> From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com>
> > >>>>> To: "gluster-users" <gluster-users at gluster.org>
> > >>>>> Sent: Friday, October 9, 2015 9:18:11 AM
> > >>>>> Subject: [Gluster-users] Speed up heal performance
> > >>>>>
> > >>>>> Is there any way to max out heal performance? My cluster is unused
> > >>> overnight,
> > >>>>> and lightly used at lunchtimes, it would be handy to speed up a
> heal.
> > >>>>>
> > >>>>> The only tuneable I found was cluster.self-heal-window-size, which
> > >>> doesn't
> > >>>>> seem to make much difference.
> > >>>> I don't know of any way to speed this up, maybe someone else could
> chime
> > >>> in here that knows the heal daemon better than me.  Maybe you could
> open
> > >>> an
> > >>> RFE on this?  In my testing I only see 2 files getting healed at a
> time
> > >>> per
> > >>> replica pair.  I would like to see this be multi threaded(if its not
> > >>> already) with the ability to tune it to control resource
> usage(similar to
> > >>> what we did in the rebalance refactoring done recently).  If you let
> me
> > >>> know the BZ # I'll add my data + suggestions, I have been testing
> this
> > >>> pretty extensively in recent weeks and good data + some ideas on how
> to
> > >>> speed things up.
> > >>> Good news is we already have a WIP patch review.glusterd.org/10851
> to
> > >>> introduce multi threaded shd. Credits to Richard/Shreyas from
> facebook
> > >>> for
> > >>> this. IIRC, we also have a BZ for the same but the patch is in rfc
> as of
> > >>> now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can
> correct
> > >>> me otherwise.
> > >>>> -b
> > >>>>
> > >>>>> thanks,
> > >>>>> --
> > >>>>> Lindsay
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> Gluster-users mailing list
> > >>>>> Gluster-users at gluster.org
> > >>>>> http://www.gluster.org/mailman/listinfo/gluster-users
> > >>>> _______________________________________________
> > >>>> Gluster-users mailing list
> > >>>> Gluster-users at gluster.org
> > >>>> http://www.gluster.org/mailman/listinfo/gluster-users
> > >>> _______________________________________________
> > >>> Gluster-users mailing list
> > >>> Gluster-users at gluster.org
> > >>> http://www.gluster.org/mailman/listinfo/gluster-users
> > >>>
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> >
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151014/f50f1263/attachment.html>