<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 23, 2018 at 4:16 PM, Hu Bert <span dir="ltr"><<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Well, over the weekend about 200GB were copied, so now there are<br>
~400GB copied to the brick. That's far beyond a speed of 10GB per<br>
hour. If I copied the 1.6 TB directly, that would be done within max 2<br>
days. But with the self heal this will take at least 20 days minimum.<br>
<br>
Why is the performance that bad? No chance of speeding this up?<br></blockquote><div><br></div><div>What kind of data do you have?</div><div>How many directories in the filesystem?</div><div>On average how many files per directory?</div><div>What is the depth of your directory hierarchy on average?</div><div>What is average filesize?</div><div><br></div><div>Based on this data we can see if anything can be improved. Or if there are some</div><div>enhancements that need to be implemented in gluster to address this kind of</div><div>data layout<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
2018-07-20 9:41 GMT+02:00 Hu Bert <<a href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>>:<br>
> hmm... no one any idea?<br>
><br>
> Additional question: the hdd on server gluster12 was changed, so far<br>
> ~220 GB were copied. On the other 2 servers i see a lot of entries in<br>
> glustershd.log, about 312.000 respectively 336.000 entries there<br>
> yesterday, most of them (current log output) looking like this:<br>
><br>
> [2018-07-20 07:30:49.757595] I [MSGID: 108026]<br>
> [afr-self-heal-common.c:1724:<wbr>afr_log_selfheal] 0-shared-replicate-3:<br>
> Completed data selfheal on 0d863a62-0dd8-401c-b699-<wbr>2b642d9fd2b6.<br>
> sources=0 [2] sinks=1<br>
> [2018-07-20 07:30:49.992398] I [MSGID: 108026]<br>
> [afr-self-heal-metadata.c:52:_<wbr>_afr_selfheal_metadata_do]<br>
> 0-shared-replicate-3: performing metadata selfheal on<br>
> 0d863a62-0dd8-401c-b699-<wbr>2b642d9fd2b6<br>
> [2018-07-20 07:30:50.243551] I [MSGID: 108026]<br>
> [afr-self-heal-common.c:1724:<wbr>afr_log_selfheal] 0-shared-replicate-3:<br>
> Completed metadata selfheal on 0d863a62-0dd8-401c-b699-<wbr>2b642d9fd2b6.<br>
> sources=0 [2] sinks=1<br>
><br>
> or like this:<br>
><br>
> [2018-07-20 07:38:41.726943] I [MSGID: 108026]<br>
> [afr-self-heal-metadata.c:52:_<wbr>_afr_selfheal_metadata_do]<br>
> 0-shared-replicate-3: performing metadata selfheal on<br>
> 9276097a-cdac-4d12-9dc6-<wbr>04b1ea4458ba<br>
> [2018-07-20 07:38:41.855737] I [MSGID: 108026]<br>
> [afr-self-heal-common.c:1724:<wbr>afr_log_selfheal] 0-shared-replicate-3:<br>
> Completed metadata selfheal on 9276097a-cdac-4d12-9dc6-<wbr>04b1ea4458ba.<br>
> sources=[0] 2 sinks=1<br>
> [2018-07-20 07:38:44.755800] I [MSGID: 108026]<br>
> [afr-self-heal-entry.c:887:<wbr>afr_selfheal_entry_do]<br>
> 0-shared-replicate-3: performing entry selfheal on<br>
> 9276097a-cdac-4d12-9dc6-<wbr>04b1ea4458ba<br>
><br>
> is this behaviour normal? I'd expect these messages on the server with<br>
> the failed brick, not on the other ones.<br>
><br>
> 2018-07-19 8:31 GMT+02:00 Hu Bert <<a href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>>:<br>
>> Hi there,<br>
>><br>
>> sent this mail yesterday, but somehow it didn't work? Wasn't archived,<br>
>> so please be indulgent it you receive this mail again :-)<br>
>><br>
>> We are currently running a replicate setup and are experiencing a<br>
>> quite poor performance. It got even worse when within a couple of<br>
>> weeks 2 bricks (disks) crashed. Maybe some general information of our<br>
>> setup:<br>
>><br>
>> 3 Dell PowerEdge R530 (Xeon E5-1650 v3 Hexa-Core, 64 GB DDR4, OS on<br>
>> separate disks); each server has 4 10TB disks -> each is a brick;<br>
>> replica 3 setup (see gluster volume status below). Debian stretch,<br>
>> kernel 4.9.0, gluster version 3.12.12. Servers and clients are<br>
>> connected via 10 GBit ethernet.<br>
>><br>
>> About a month ago and 2 days ago a disk died (on different servers);<br>
>> disk were replaced, were brought back into the volume and full self<br>
>> heal started. But the speed for this is quite... disappointing. Each<br>
>> brick has ~1.6TB of data on it (mostly the infamous small files). The<br>
>> full heal i started yesterday copied only ~50GB within 24 hours (48<br>
>> hours: about 100GB) - with<br>
>> this rate it would take weeks until the self heal finishes.<br>
>><br>
>> After the first heal (started on gluster13 about a month ago, took<br>
>> about 3 weeks) finished we had a terrible performance; CPU on one or<br>
>> two of the nodes (gluster11, gluster12) was up to 1200%, consumed by<br>
>> the brick process of the former crashed brick (bricksdd1),<br>
>> interestingly not on the server with the failed this, but on the other<br>
>> 2 ones...<br>
>><br>
>> Well... am i doing something wrong? Some options wrongly configured?<br>
>> Terrible setup? Anyone got an idea? Any additional information needed?<br>
>><br>
>><br>
>> Thx in advance :-)<br>
>><br>
>> gluster volume status<br>
>><br>
>> Volume Name: shared<br>
>> Type: Distributed-Replicate<br>
>> Volume ID: e879d208-1d8c-4089-85f3-<wbr>ef1b3aa45d36<br>
>> Status: Started<br>
>> Snapshot Count: 0<br>
>> Number of Bricks: 4 x 3 = 12<br>
>> Transport-type: tcp<br>
>> Bricks:<br>
>> Brick1: gluster11:/gluster/bricksda1/<wbr>shared<br>
>> Brick2: gluster12:/gluster/bricksda1/<wbr>shared<br>
>> Brick3: gluster13:/gluster/bricksda1/<wbr>shared<br>
>> Brick4: gluster11:/gluster/bricksdb1/<wbr>shared<br>
>> Brick5: gluster12:/gluster/bricksdb1/<wbr>shared<br>
>> Brick6: gluster13:/gluster/bricksdb1/<wbr>shared<br>
>> Brick7: gluster11:/gluster/bricksdc1/<wbr>shared<br>
>> Brick8: gluster12:/gluster/bricksdc1/<wbr>shared<br>
>> Brick9: gluster13:/gluster/bricksdc1/<wbr>shared<br>
>> Brick10: gluster11:/gluster/bricksdd1/<wbr>shared<br>
>> Brick11: gluster12:/gluster/bricksdd1_<wbr>new/shared<br>
>> Brick12: gluster13:/gluster/bricksdd1_<wbr>new/shared<br>
>> Options Reconfigured:<br>
>> cluster.shd-max-threads: 4<br>
>> performance.md-cache-timeout: 60<br>
>> cluster.lookup-optimize: on<br>
>> cluster.readdir-optimize: on<br>
>> performance.cache-refresh-<wbr>timeout: 4<br>
>> performance.parallel-readdir: on<br>
>> server.event-threads: 8<br>
>> client.event-threads: 8<br>
>> performance.cache-max-file-<wbr>size: 128MB<br>
>> performance.write-behind-<wbr>window-size: 16MB<br>
>> performance.io-thread-count: 64<br>
>> cluster.min-free-disk: 1%<br>
>> performance.cache-size: 24GB<br>
>> nfs.disable: on<br>
>> transport.address-family: inet<br>
>> performance.high-prio-threads: 32<br>
>> performance.normal-prio-<wbr>threads: 32<br>
>> performance.low-prio-threads: 32<br>
>> performance.least-prio-<wbr>threads: 8<br>
>> performance.io-cache: on<br>
>> server.allow-insecure: on<br>
>> performance.strict-o-direct: off<br>
>> transport.listen-backlog: 100<br>
>> server.outstanding-rpc-limit: 128<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>