[Gluster-users] Problems running dbench on 3.3

Tue Feb 19 22:32:10 UTC 2013

They were just self-heal double-checks performed in the lookup path, as we
perform the initial lookup unlocked for performance reasons. Those lines by
themselves do not indicate anything harmful (yet). I admit the log messages
could have been better though!

Avati

On Tue, Feb 19, 2013 at 2:20 PM, Marc Seeger <marc.seeger at acquia.com> wrote:

> I'll give it a try. Isn't it somewhat concerning that this would heal to
> missing gfids and failing self-heals though?
>
> On 19.02.2013, at 21:22, Anand Avati <anand.avati at gmail.com> wrote:
>
> We run dbench in the pre-commit script for every patch. Looks like you are
> running dbench on the same directory from multiple clients. Is that even
> supposed to work? Did you instead lunch to run each dbench within their own
> subdirectory?
>
> Avati
>
> On Tue, Feb 19, 2013 at 12:07 PM, Marc Seeger <marc.seeger at acquia.com>wrote:
>
>> To test gluster's behavior under heavy load, I'm currently doing this on
>> two machines sharing a common /mnt/gfs gluster mount:
>>
>> ssh bal-6.example.com apt-get install dbench && dbench 6 -t 60 -D
>> /mnt/gfs
>> ssh bal-7.example.com apt-get install dbench && dbench 6 -t 60 -D
>> /mnt/gfs
>>
>>
>> One of the processes usually dies pretty quickly like this:
>>
>> [608] open /mnt/gfs/clients/client5/~dmtmp/PWRPNT/PCBENCHM.PPT failed for
>> handle 10003 (No such file or directory)
>> (610) ERROR: handle 10003 was not found,
>> Child failed with status 1
>>
>>
>> And the logs are full of things like this (ignore the initial timestamp,
>> that's from our logging):
>>
>> [2013-02-19 14:38:38.714493] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  data missing-entry gfid self-heal failed on
>> /clients/client5/~dmtmp/PM/MOVED.DOC,
>> [2013-02-19 14:38:38.724494] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client3/~dmtmp,
>> [2013-02-19 14:38:38.734495] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  data missing-entry gfid self-heal failed on
>> /clients/client4/~dmtmp/PM/EVENTS.DOC,
>> [2013-02-19 14:38:38.734495] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  data missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/PM/MOVED.DOC,
>> [2013-02-19 14:38:38.734495] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  data missing-entry gfid self-heal failed on
>> /clients/client1/~dmtmp/PM/MOVED.DOC,
>> [2013-02-19 14:38:38.734495] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  data missing-entry gfid self-heal failed on
>> /clients/client0/~dmtmp/PM/MOVED.DOC,
>> [2013-02-19 14:38:38.734495] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client4/~dmtmp/PM,  [build-2
>> system.rb:340], I,
>> [2013-02-19T14:39:50.189970 #20802]  INFO -- :
>> [2013-02-19 14:38:36.041890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /,
>> [2013-02-19 14:38:36.041890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /,
>> [2013-02-19 14:38:36.041890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /,
>> [2013-02-19 14:38:36.041890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /,
>> [2013-02-19 14:38:36.041890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /,
>> [2013-02-19 14:38:36.051890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients,
>> [2013-02-19 14:38:36.071890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2,
>> [2013-02-19 14:38:36.071890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3,
>> [2013-02-19 14:38:36.071890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client2,
>> [2013-02-19 14:38:36.081890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client3,
>> [2013-02-19 14:38:36.091890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp,
>> [2013-02-19 14:38:36.091890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp,
>> [2013-02-19 14:38:36.101890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client2/~dmtmp,
>> [2013-02-19 14:38:36.101890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client3/~dmtmp,
>> [2013-02-19 14:38:36.111890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/WORD,
>> [2013-02-19 14:38:36.111890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/WORD,
>> [2013-02-19 14:38:36.131890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client2/~dmtmp/WORD,
>> [2013-02-19 14:38:36.141890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client3/~dmtmp/WORD,
>> [2013-02-19 14:38:36.151890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/WORD/CHAP10.DOC,
>> [2013-02-19 14:38:36.151890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/WORD/CHAP10.DOC,
>> [2013-02-19 14:38:36.161890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/WORD/BASEMACH.DOC,
>> [2013-02-19 14:38:36.161890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/WORD/BASEMACH.DOC,
>> [2013-02-19 14:38:36.171890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entr [build-2 system.rb:340], I,
>> [2013-02-19T14:39:50.189970 #20802]  INFO -- : y missing-entry gfid
>> self-heal failed on /clients/client2/~dmtmp/WORD/FACTS.DOC,
>> [2013-02-19 14:38:36.181890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/WORD/FACTS.DOC,
>> [2013-02-19 14:38:36.201890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/EXCEL,
>> [2013-02-19 14:38:36.201890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/EXCEL,
>> [2013-02-19 14:38:36.201890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client2/~dmtmp/EXCEL,
>> [2013-02-19 14:38:36.201890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client3/~dmtmp/EXCEL,
>> [2013-02-19 14:38:36.211890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client0/~dmtmp,
>> [2013-02-19 14:38:36.211890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/EXCEL/PCMAGCD.XLS,
>> [2013-02-19 14:38:36.211890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/EXCEL/PCMAGCD.XLS,
>> [2013-02-19 14:38:36.241890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/EXCEL/SALES.XLS,
>> [2013-02-19 14:38:36.241890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/EXCEL/SALES.XLS,
>> [2013-02-19 14:38:36.271890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/PWRPNT,
>> [2013-02-19 14:38:36.271890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/PWRPNT,
>> [2013-02-19 14:38:36.281890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client2/~dmtmp/PWRPNT,
>> [2013-02-19 14:38:36.281890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  entry self-heal failed on /clients/client3/~dmtmp/PWRPNT,
>> [2013-02-19 14:38:36.291890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/PWRPNT/PCBENCHM.PPT,
>> [2013-02-19 14:38:36.311890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/PWRPNT/PCBENCHM.PPT,
>> [2013-02-19 14:38:36.351890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/PWRPNT/ZD16.BMP,
>> [2013-02-19 14:38:36.351890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client3/~dmtmp/PWRPNT/ZD16.BMP,
>> [2013-02-19 14:38:36.381890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid self-heal failed on
>> /clients/client2/~dmtmp/PWRPNT/PPTOOLS1.PPA,
>> [2013-02-19 14:38:36.391890] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0:
>> background  meta-data data entry missing-entry gfid sel [build-2
>> system.rb:340]
>>
>>
>>
>> Any ideas? Can somebody confirm this happens for them too?
>>
>> The setup is ubuntu lucid machines running 3.3.1 from this PPA:
>> https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130219/0eec10ce/attachment.html>