[Gluster-users] disperse volume brick counts limits in RHES

[Serkan Çoban]

In our use case every node has 26 bricks. I am using 60 nodes, one 9PB
volume with 16+4 EC configuration, each brick in a sub-volume is on
different host.
We put 15-20k 2GB files every day into 10-15 folders. So it is 1500K
files/folder. Our gluster version is 3.7.11.
Heal speed in this environment is 8-10MB/sec/brick.

I did some tests for parallel self heal feature with version 3.9, two
servers 26 bricks each, 8+2 and 16+4 EC configuration.
This was a small test environment and the results are as I said 8+2 is
2x faster then 16+4 with parallel self heal threads set to 2/4.
In 1-2 months our new servers arriving, I will do detailed tests for
heal performance for 8+2 and 16+4 and inform you the results.


On Fri, May 5, 2017 at 2:54 PM, Pranith Kumar Karampuri
<pkarampu at redhat.com> wrote:
>
>
> On Fri, May 5, 2017 at 5:19 PM, Pranith Kumar Karampuri
> <pkarampu at redhat.com> wrote:
>>
>>
>>
>> On Fri, May 5, 2017 at 2:38 PM, Serkan Çoban <cobanserkan at gmail.com>
>> wrote:
>>>
>>> It is the over all time, 8TB data disk healed 2x faster in 8+2
>>> configuration.
>>
>>
>> Wow, that is counter intuitive for me. I will need to explore about this
>> to find out why that could be. Thanks a lot for this feedback!
>
>
> From memory I remember you said you have a lot of small files hosted on the
> volume, right? It could be because of the bug
> https://review.gluster.org/17151 is fixing. That is the only reason I could
> guess right now. We will try to test this kind of case if you could give us
> a bit more details about average file-size/depth of directories etc to
> simulate similar looking directory structure.
>
>>
>>
>>>
>>>
>>> On Fri, May 5, 2017 at 10:00 AM, Pranith Kumar Karampuri
>>> <pkarampu at redhat.com> wrote:
>>> >
>>> >
>>> > On Fri, May 5, 2017 at 11:42 AM, Serkan Çoban <cobanserkan at gmail.com>
>>> > wrote:
>>> >>
>>> >> Healing gets slower as you increase m in m+n configuration.
>>> >> We are using 16+4 configuration without any problems other then heal
>>> >> speed.
>>> >> I tested heal speed with 8+2 and 16+4 on 3.9.0 and see that heals on
>>> >> 8+2 is faster by 2x.
>>> >
>>> >
>>> > As you increase number of nodes that are participating in an EC set
>>> > number
>>> > of parallel heals increase. Is the heal speed you saw improved per file
>>> > or
>>> > the over all time it took to heal the data?
>>> >
>>> >>
>>> >>
>>> >>
>>> >> On Fri, May 5, 2017 at 9:04 AM, Ashish Pandey <aspandey at redhat.com>
>>> >> wrote:
>>> >> >
>>> >> > 8+2 and 8+3 configurations are not the limitation but just
>>> >> > suggestions.
>>> >> > You can create 16+3 volume without any issue.
>>> >> >
>>> >> > Ashish
>>> >> >
>>> >> > ________________________________
>>> >> > From: "Alastair Neil" <ajneil.tech at gmail.com>
>>> >> > To: "gluster-users" <gluster-users at gluster.org>
>>> >> > Sent: Friday, May 5, 2017 2:23:32 AM
>>> >> > Subject: [Gluster-users] disperse volume brick counts limits in RHES
>>> >> >
>>> >> >
>>> >> > Hi
>>> >> >
>>> >> > we are deploying a large (24node/45brick) cluster and noted that the
>>> >> > RHES
>>> >> > guidelines limit the number of data bricks in a disperse set to 8.
>>> >> > Is
>>> >> > there
>>> >> > any reason for this.  I am aware that you want this to be a power of
>>> >> > 2,
>>> >> > but
>>> >> > as we have a large number of nodes we were planning on going with
>>> >> > 16+3.
>>> >> > Dropping to 8+2 or 8+3 will be a real waste for us.
>>> >> >
>>> >> > Thanks,
>>> >> >
>>> >> >
>>> >> > Alastair
>>> >> >
>>> >> >
>>> >> > _______________________________________________
>>> >> > Gluster-users mailing list
>>> >> > Gluster-users at gluster.org
>>> >> > http://lists.gluster.org/mailman/listinfo/gluster-users
>>> >> >
>>> >> >
>>> >> > _______________________________________________
>>> >> > Gluster-users mailing list
>>> >> > Gluster-users at gluster.org
>>> >> > http://lists.gluster.org/mailman/listinfo/gluster-users
>>> >> _______________________________________________
>>> >> Gluster-users mailing list
>>> >> Gluster-users at gluster.org
>>> >> http://lists.gluster.org/mailman/listinfo/gluster-users
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Pranith
>>
>>
>>
>>
>> --
>> Pranith
>
>
>
>
> --
> Pranith