[Gluster-users] Problems running dbench on 3.3

Marc Seeger marc.seeger at acquia.com
Thu Feb 21 09:23:07 UTC 2013 

Ok, confirmed. Letting dbench work on different subfolders makes it work fine.
In terms of log messages:
I would love to monitor for problems with gluster. Failing self heals in the logs might not really be a problem per-se, any suggestions of what else I could parse the logs for?

Cheers,
Marc

On 19.02.2013, at 23:32, Anand Avati <anand.avati at gmail.com> wrote:

> They were just self-heal double-checks performed in the lookup path, as we perform the initial lookup unlocked for performance reasons. Those lines by themselves do not indicate anything harmful (yet). I admit the log messages could have been better though!
> 
> Avati
> 
> On Tue, Feb 19, 2013 at 2:20 PM, Marc Seeger <marc.seeger at acquia.com> wrote:
> I'll give it a try. Isn't it somewhat concerning that this would heal to missing gfids and failing self-heals though?
> 
> On 19.02.2013, at 21:22, Anand Avati <anand.avati at gmail.com> wrote:
> 
>> We run dbench in the pre-commit script for every patch. Looks like you are running dbench on the same directory from multiple clients. Is that even supposed to work? Did you instead lunch to run each dbench within their own subdirectory?
>> 
>> Avati
>> 
>> On Tue, Feb 19, 2013 at 12:07 PM, Marc Seeger <marc.seeger at acquia.com> wrote:
>> To test gluster's behavior under heavy load, I'm currently doing this on two machines sharing a common /mnt/gfs gluster mount:
>> 
>> ssh bal-6.example.com apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs
>> ssh bal-7.example.com apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs
>> 
>> 
>> One of the processes usually dies pretty quickly like this:
>> 
>> [608] open /mnt/gfs/clients/client5/~dmtmp/PWRPNT/PCBENCHM.PPT failed for handle 10003 (No such file or directory)
>> (610) ERROR: handle 10003 was not found,
>> Child failed with status 1
>> 
>> 
>> And the logs are full of things like this (ignore the initial timestamp, that's from our logging):
>> 
>> [2013-02-19 14:38:38.714493] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client5/~dmtmp/PM/MOVED.DOC, 
>> [2013-02-19 14:38:38.724494] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp, 
>> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client4/~dmtmp/PM/EVENTS.DOC, 
>> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PM/MOVED.DOC, 
>> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client1/~dmtmp/PM/MOVED.DOC, 
>> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client0/~dmtmp/PM/MOVED.DOC, 
>> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client4/~dmtmp/PM,  [build-2 system.rb:340], I,  
>> [2013-02-19T14:39:50.189970 #20802]  INFO -- : 
>> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
>> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
>> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
>> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
>> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
>> [2013-02-19 14:38:36.051890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients, 
>> [2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2, 
>> [2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3, 
>> [2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2, 
>> [2013-02-19 14:38:36.081890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3, 
>> [2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp, 
>> [2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp, 
>> [2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2/~dmtmp, 
>> [2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp, 
>> [2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD, 
>> [2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD, 
>> [2013-02-19 14:38:36.131890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2/~dmtmp/WORD, 
>> [2013-02-19 14:38:36.141890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp/WORD, 
>> [2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/CHAP10.DOC, 
>> [2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/CHAP10.DOC, 
>> [2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/BASEMACH.DOC, 
>> [2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/BASEMACH.DOC, 
>> [2013-02-19 14:38:36.171890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entr [build-2 system.rb:340], I,  
>> [2013-02-19T14:39:50.189970 #20802]  INFO -- : y missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/FACTS.DOC, 
>> [2013-02-19 14:38:36.181890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/FACTS.DOC, 
>> [2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL, 
>> [2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL, 
>> [2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2/~dmtmp/EXCEL, 
>> [2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp/EXCEL, 
>> [2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client0/~dmtmp, 
>> [2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/PCMAGCD.XLS, 
>> [2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/PCMAGCD.XLS, 
>> [2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/SALES.XLS, 
>> [2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/SALES.XLS, 
>> [2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT, 
>> [2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT, 
>> [2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2/~dmtmp/PWRPNT, 
>> [2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp/PWRPNT, 
>> [2013-02-19 14:38:36.291890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PCBENCHM.PPT, 
>> [2013-02-19 14:38:36.311890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/PCBENCHM.PPT, 
>> [2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/ZD16.BMP, 
>> [2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/ZD16.BMP, 
>> [2013-02-19 14:38:36.381890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PPTOOLS1.PPA, 
>> [2013-02-19 14:38:36.391890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid sel [build-2 system.rb:340]
>> 
>> 
>> 
>> Any ideas? Can somebody confirm this happens for them too?
>> 
>> The setup is ubuntu lucid machines running 3.3.1 from this PPA: https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3
>> 
>> 
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130221/cf24a5e9/attachment.html>

More information about the Gluster-users mailing list