[Gluster-users] Cascading errors and very bad write performance

Fri Aug 7 12:04:09 UTC 2015

Hi Vijay, 

My brick logs issue and big performance problem have begun when I upgraded Gluster into 3.7.3 version; before write throughput was good enough (~500MBs) -but not as good as with GlusterFS 3.5.3 (especially with distributed volumes)- and didn’t notice these problème with brick-logs.

OK… in live:

i just disable to quota for my home volume and now my performance appears to be relatively better (around 300MBs) but i still see the logs (from storage1 and its replicate storage2) growing up with only this kind of lines:
[2015-08-07 11:16:51.746142] E [dict.c:1418:dict_copy_with_ref] (-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60) [0x7f85e9a6a410] -->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88) [0x7f85e9a6a188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4) [0x3e99c20674] ) 0-dict: invalid argument: dict [Argument invalide]

After a few minutes: my write throughput seems to be now correct (~550MBs) but the log are still growing up (to not say exploding). So one part of the problem looks like taking its origin in the quota system management.
… after a few minutes (and still only 1 client connected), now it is the read operation which is very very slow… -I’m gonna become crazy! :/-
# ddt -t 50g /home/
Writing to /home/ddt.11293 ... syncing ... done.
sleeping 10 seconds ... done.
Reading from /home/ddt.11293 ... done.
35840MiB    KiB/s  CPU%
Write      568201     5
Read       567008     4
# ddt -t 50g /home/
Writing to /home/ddt.11397 ... syncing ... done.
sleeping 10 seconds ... done.
Reading from /home/ddt.11397 ... done.
51200MiB    KiB/s  CPU%
Write      573631     5
Read       164716     1

and my log are still exploding…

After having re-enabled the quota on my volume: 
# ddt -t 50g /home/
Writing to /home/ddt.11817 ... syncing ... done.
sleeping 10 seconds ... done.
Reading from /home/ddt.11817 ... done.
51200MiB    KiB/s  CPU%
Write      269608     3
Read       160219     1

Thanks 
Geoffrey 
------------------------------------------------------
Geoffrey Letessier
Responsable informatique & ingénieur système
UPR 9080 - CNRS - Laboratoire de Biochimie Théorique
Institut de Biologie Physico-Chimique
13, rue Pierre et Marie Curie - 75005 Paris
Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr

Le 7 août 2015 à 06:28, Vijaikumar M <vmallika at redhat.com> a écrit :

> Hi Geoffrey,
> 
> Some performance improvements has been done in quota in glusterfs-3.7.3.
> Could you upgrade to glusterfs-3.7.3 and see if this helps
> 
> Thanks,
> Vijay
> 
> 
> On Friday 07 August 2015 05:02 AM, Geoffrey Letessier wrote:
>> Hi,
>> 
>> No idea to help me fix this issue? (big logs, small write performance (/4), etc.)
>> 
>> For comparison, here to volumes: 
>>  - home: distributed on 4 bricks / 2 nodes  (and replicated on 4 other bricks / 2 other nodes):
>> # ddt -t 35g /home
>> Writing to /home/ddt.24172 ... syncing ... done.
>> sleeping 10 seconds ... done.
>> Reading from /home/ddt.24172 ... done.
>> 33792MiB    KiB/s  CPU%
>> Write      103659     1
>> Read       391955     3
>> 
>>  - workdir: distributed on 4 bricks / 2 nodes (one the same RAID volumes and servers than home):
>> # ddt -t 35g /workdir
>> Writing to /workdir/ddt.24717 ... syncing ... done.
>> sleeping 10 seconds ... done.
>> Reading from /workdir/ddt.24717 ... done.
>> 35840MiB    KiB/s  CPU%
>> Write      738314     4
>> Read       536497     4
>> 
>> For information, previously on 3.5.3-2 version, I obtained roughly 1.1GBs for workdir volume and ~550-600MBs for home.
>> 
>> All my tests (CP, RSYNC, etc.) provides me the same result (write throughput between 100MBs and 150MBs)
>> 
>> Thanks.
>> Geoffrey
>> ------------------------------------------------------
>> Geoffrey Letessier
>> Responsable informatique & ingénieur système
>> UPR 9080 - CNRS - Laboratoire de Biochimie Théorique
>> Institut de Biologie Physico-Chimique
>> 13, rue Pierre et Marie Curie - 75005 Paris
>> Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr
>> 
>> Le 5 août 2015 à 10:40, Geoffrey Letessier <geoffrey.letessier at cnrs.fr> a écrit :
>> 
>>> Hello,
>>> 
>>> In addition, knowing I have reactivated the log (brick-log-level = INFO not CRITICAL) only for the file creation duration (i.e. a few minutes), do you have noticed the log sizes and the number of lines inside:
>>> # ls -lh storage*
>>> -rw-------  1 letessier  staff    18M  5 aoû 00:54 storage1__export-brick_home-brick1-data.log
>>> -rw-------  1 letessier  staff   2,1K  5 aoû 00:54 storage1__export-brick_home-brick2-data.log
>>> -rw-------  1 letessier  staff    15M  5 aoû 00:56 storage2__export-brick_home-brick1-data.log
>>> -rw-------  1 letessier  staff   2,1K  5 aoû 00:54 storage2__export-brick_home-brick2-data.log
>>> -rw-------  1 letessier  staff    47M  5 aoû 00:55 storage3__export-brick_home-brick1-data.log
>>> -rw-------  1 letessier  staff   2,1K  5 aoû 00:54 storage3__export-brick_home-brick2-data.log
>>> -rw-------  1 letessier  staff    47M  5 aoû 00:55 storage4__export-brick_home-brick1-data.log
>>> -rw-------  1 letessier  staff   2,1K  5 aoû 00:55 storage4__export-brick_home-brick2-data.log
>>> 
>>> # wc -l storage*
>>>    55381 storage1__export-brick_home-brick1-data.log
>>>       17 storage1__export-brick_home-brick2-data.log
>>>    41636 storage2__export-brick_home-brick1-data.log
>>>       17 storage2__export-brick_home-brick2-data.log
>>>   270360 storage3__export-brick_home-brick1-data.log
>>>       17 storage3__export-brick_home-brick2-data.log
>>>   270358 storage4__export-brick_home-brick1-data.log
>>>       17 storage4__export-brick_home-brick2-data.log
>>>   637803 total
>>> 
>>> If the let brick-log-level to INFO, the brick log files in each server will consume all my /var partition capacity within only a few hours/days…
>>> 
>>> Thanks in advance,
>>> Geoffrey
>>> ------------------------------------------------------
>>> Geoffrey Letessier
>>> Responsable informatique & ingénieur système
>>> UPR 9080 - CNRS - Laboratoire de Biochimie Théorique
>>> Institut de Biologie Physico-Chimique
>>> 13, rue Pierre et Marie Curie - 75005 Paris
>>> Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr
>>> 
>>> Le 5 août 2015 à 01:12, Geoffrey Letessier <geoffrey.letessier at cnrs.fr> a écrit :
>>> 
>>>> Hello,
>>>> 
>>>> Since the problem motioned previously (all errors noticed in brick log files), i notice a very very bad performance: i can note my write performance divided by 4 than previously -knowing it was not so good before.
>>>> Now, a write of a 33GB file, my write throughput is around 150MBs (with Infiniband), before it was around 550-600MBs; and this, both with RDMA and TCP protocol.
>>>> 
>>>> During this test, more than 40 000 error lines (as the following) were added to the brick log files.
>>>> [2015-08-04 22:34:27.337622] E [dict.c:1418:dict_copy_with_ref] (-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60) [0x7f021c6f7410] -->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88) [0x7f021c6f7188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4) [0x7f0229cba674] ) 0-dict: invalid argument: dict [Argument invalide]
>>>> 
>>>> 
>>>> All brick log files are in attachments.
>>>> 
>>>> Thanks in advance for all your help and fix,
>>>> Best,
>>>> Geoffrey
>>>> 
>>>> PS: question: is it possible to easily downgrade GlusterFS to a previous version from 3.7 (for example: v3.5)?
>>>> 
>>>> ------------------------------------------------------
>>>> Geoffrey Letessier
>>>> Responsable informatique & ingénieur système
>>>> UPR 9080 - CNRS - Laboratoire de Biochimie Théorique
>>>> Institut de Biologie Physico-Chimique
>>>> 13, rue Pierre et Marie Curie - 75005 Paris
>>>> Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr
>>>> <bricks-logs.tgz>
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150807/94fff53e/attachment.html>