[Gluster-users] Problems running dbench on 3.3

Tue Feb 19 22:20:14 UTC 2013

I'll give it a try. Isn't it somewhat concerning that this would heal to missing gfids and failing self-heals though?

On 19.02.2013, at 21:22, Anand Avati <anand.avati at gmail.com> wrote:

> We run dbench in the pre-commit script for every patch. Looks like you are running dbench on the same directory from multiple clients. Is that even supposed to work? Did you instead lunch to run each dbench within their own subdirectory?
> 
> Avati
> 
> On Tue, Feb 19, 2013 at 12:07 PM, Marc Seeger <marc.seeger at acquia.com> wrote:
> To test gluster's behavior under heavy load, I'm currently doing this on two machines sharing a common /mnt/gfs gluster mount:
> 
> ssh bal-6.example.com apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs
> ssh bal-7.example.com apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs
> 
> 
> One of the processes usually dies pretty quickly like this:
> 
> [608] open /mnt/gfs/clients/client5/~dmtmp/PWRPNT/PCBENCHM.PPT failed for handle 10003 (No such file or directory)
> (610) ERROR: handle 10003 was not found,
> Child failed with status 1
> 
> 
> And the logs are full of things like this (ignore the initial timestamp, that's from our logging):
> 
> [2013-02-19 14:38:38.714493] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client5/~dmtmp/PM/MOVED.DOC, 
> [2013-02-19 14:38:38.724494] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp, 
> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client4/~dmtmp/PM/EVENTS.DOC, 
> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PM/MOVED.DOC, 
> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client1/~dmtmp/PM/MOVED.DOC, 
> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  data missing-entry gfid self-heal failed on /clients/client0/~dmtmp/PM/MOVED.DOC, 
> [2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client4/~dmtmp/PM,  [build-2 system.rb:340], I,  
> [2013-02-19T14:39:50.189970 #20802]  INFO -- : 
> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
> [2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /, 
> [2013-02-19 14:38:36.051890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients, 
> [2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2, 
> [2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3, 
> [2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2, 
> [2013-02-19 14:38:36.081890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3, 
> [2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp, 
> [2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp, 
> [2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2/~dmtmp, 
> [2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp, 
> [2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD, 
> [2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD, 
> [2013-02-19 14:38:36.131890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2/~dmtmp/WORD, 
> [2013-02-19 14:38:36.141890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp/WORD, 
> [2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/CHAP10.DOC, 
> [2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/CHAP10.DOC, 
> [2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/BASEMACH.DOC, 
> [2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/BASEMACH.DOC, 
> [2013-02-19 14:38:36.171890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entr [build-2 system.rb:340], I,  
> [2013-02-19T14:39:50.189970 #20802]  INFO -- : y missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/FACTS.DOC, 
> [2013-02-19 14:38:36.181890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/FACTS.DOC, 
> [2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL, 
> [2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL, 
> [2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2/~dmtmp/EXCEL, 
> [2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp/EXCEL, 
> [2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client0/~dmtmp, 
> [2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/PCMAGCD.XLS, 
> [2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/PCMAGCD.XLS, 
> [2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/SALES.XLS, 
> [2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/SALES.XLS, 
> [2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT, 
> [2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT, 
> [2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client2/~dmtmp/PWRPNT, 
> [2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  entry self-heal failed on /clients/client3/~dmtmp/PWRPNT, 
> [2013-02-19 14:38:36.291890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PCBENCHM.PPT, 
> [2013-02-19 14:38:36.311890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/PCBENCHM.PPT, 
> [2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/ZD16.BMP, 
> [2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/ZD16.BMP, 
> [2013-02-19 14:38:36.381890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PPTOOLS1.PPA, 
> [2013-02-19 14:38:36.391890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background  meta-data data entry missing-entry gfid sel [build-2 system.rb:340]
> 
> 
> 
> Any ideas? Can somebody confirm this happens for them too?
> 
> The setup is ubuntu lucid machines running 3.3.1 from this PPA: https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130219/0001a074/attachment.html>