[Gluster-users] Replication delay

Fri Jan 24 10:28:38 UTC 2014

----- Messaggio originale -----
> Da: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> A: "Fabio Rosati" <fabio.rosati at geminformatica.it>
> Cc: "Gluster-users at gluster.org List" <gluster-users at gluster.org>
> Inviato: Venerdì, 24 gennaio 2014 11:02:15
> Oggetto: Re: [Gluster-users] Replication delay
> 
> 
> 
> ----- Original Message -----
> > From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> > To: "Fabio Rosati" <fabio.rosati at geminformatica.it>
> > Cc: "Gluster-users at gluster.org List" <gluster-users at gluster.org>
> > Sent: Friday, January 24, 2014 3:29:19 PM
> > Subject: Re: [Gluster-users] Replication delay
> > 
> > Hi Fabio,
> >      This is a known issue that has been addressed on master. It may be
> >      backported to 3.5. When a file is undergoing changes, it may appear in
> >      'gluster volume heal <volname> info' output even when it doesn't need
> >      any self-heal.
> > 
> > Pranith
> 
> Sorry, I just saw that there is a self-heal happening for 15 minutes when you
> stop the VMs. How are you checking that the self-heal is happening?

When I stop the VM for alfresco.qc2, "heal info" still reports alfresco.qc2 as in need for healing for about 15min.
It seems this is a real out-of-sync situation because if I check the two bricks I get different modification times up until they are healed (no more reported by "heal info"). This is the bricks' status for alfresco.qc2 while the VM is halted:

[root at networker ~]# ll /glustexp/pri1/brick/
totale 27769492
-rw-------. 2 qemu qemu 8212709376 24 gen 11:16 alfresco.qc2
[...]

[root at networker2 ~]# ll /glustexp/pri1/brick/
totale 27769384
-rw-------. 2 qemu qemu 8212709376 24 gen 11:05 alfresco.qc2
[...]

Bricks' status AFTER "heal info" doesn't report alfresco.qc2 anymore:

[root at networker ~]# ll /glustexp/pri1/brick/
totale 27769492
-rw-------. 2 qemu qemu 8212709376 24 gen 11:05 alfresco.qc2

[root at networker2 ~]# ll /glustexp/pri1/brick/
totale 27769384
-rw-------. 2 qemu qemu 8212709376 24 gen 11:05 alfresco.qc2

Thanks for helping!

Fabio

> 
> Pranith
> > 
> > ----- Original Message -----
> > > From: "Fabio Rosati" <fabio.rosati at geminformatica.it>
> > > To: "Gluster-users at gluster.org List" <gluster-users at gluster.org>
> > > Sent: Friday, January 24, 2014 3:17:27 PM
> > > Subject: [Gluster-users] Replication delay
> > > 
> > > Hi All,
> > > 
> > > in a distributed-replicated volume hosting some VMs disk images
> > > (GlusterFS
> > > 3.4.2 on CentOS 6.5, qemu-kvm with glusterfs native support, no fuse
> > > mount),
> > > I always get the same two files that need healing:
> > > 
> > > 
> > > 
> > > [root at networker ~]# gluster volume heal gv_pri info
> > > Gathering Heal info on volume gv_pri has been successful
> > > 
> > > 
> > > 
> > > 
> > > Brick nw1glus.gem.local:/glustexp/pri1/brick
> > > Number of entries: 2
> > > /alfresco.qc2
> > > /remlog.qc2
> > > 
> > > 
> > > 
> > > 
> > > Brick nw2glus.gem.local:/glustexp/pri1/brick
> > > Number of entries: 2
> > > /alfresco.qc2
> > > /remlog.qc2
> > > 
> > > 
> > > 
> > > 
> > > Brick nw3glus.gem.local:/glustexp/pri2/brick
> > > Number of entries: 0
> > > 
> > > 
> > > 
> > > 
> > > Brick nw4glus.gem.local:/glustexp/pri2/brick
> > > Number of entries: 0
> > > 
> > > This is not a split-brain situation (I checked) and If I stop the two VMs
> > > that use these images, I get the two files healed/synced in about 15min.
> > > This is too much time, IMHO.
> > > In this volume there are other VM images with (smaller) disk images
> > > replicated on the same bricks and they get synced "in real-time".
> > > 
> > > These are the volume's details, the host "networker" is nw1glus.gem.local
> > > :
> > > 
> > > 
> > > 
> > > [root at networker ~]# gluster volume info gv_pri
> > > 
> > > Volume Name: gv_pri
> > > Type: Distributed-Replicate
> > > Volume ID: 3d91b91e-4d72-484f-8655-e5ed8d38bb28
> > > Status: Started
> > > Number of Bricks: 2 x 2 = 4
> > > Transport-type: tcp
> > > Bricks:
> > > Brick1: nw1glus.gem.local:/glustexp/pri1/brick
> > > Brick2: nw2glus.gem.local:/glustexp/pri1/brick
> > > Brick3: nw3glus.gem.local:/glustexp/pri2/brick
> > > Brick4: nw4glus.gem.local:/glustexp/pri2/brick
> > > Options Reconfigured:
> > > server.allow-insecure: on
> > > storage.owner-uid: 107
> > > storage.owner-gid: 107
> > > 
> > > [root at networker ~]# gluster volume status gv_pri detail
> > > 
> > > 
> > > Status of volume: gv_pri
> > > ------------------------------------------------------------------------------
> > > Brick : Brick nw1glus.gem.local:/glustexp/pri1/brick
> > > Port : 50178
> > > Online : Y
> > > Pid : 25721
> > > File System : xfs
> > > Device : /dev/mapper/vg_guests-lv_brick1
> > > Mount Options : rw,noatime
> > > Inode Size : 512
> > > Disk Space Free : 168.4GB
> > > Total Disk Space : 194.9GB
> > > Inode Count : 102236160
> > > Free Inodes : 102236130
> > > ------------------------------------------------------------------------------
> > > Brick : Brick nw2glus.gem.local:/glustexp/pri1/brick
> > > Port : 50178
> > > Online : Y
> > > Pid : 27832
> > > File System : xfs
> > > Device : /dev/mapper/vg_guests-lv_brick1
> > > Mount Options : rw,noatime
> > > Inode Size : 512
> > > Disk Space Free : 168.4GB
> > > Total Disk Space : 194.9GB
> > > Inode Count : 102236160
> > > Free Inodes : 102236130
> > > ------------------------------------------------------------------------------
> > > Brick : Brick nw3glus.gem.local:/glustexp/pri2/brick
> > > Port : 50182
> > > Online : Y
> > > Pid : 14571
> > > File System : xfs
> > > Device : /dev/mapper/vg_guests-lv_brick2
> > > Mount Options : rw,noatime
> > > Inode Size : 512
> > > Disk Space Free : 418.3GB
> > > Total Disk Space : 433.8GB
> > > Inode Count : 227540992
> > > Free Inodes : 227540973
> > > ------------------------------------------------------------------------------
> > > Brick : Brick nw4glus.gem.local:/glustexp/pri2/brick
> > > Port : 50181
> > > Online : Y
> > > Pid : 21942
> > > File System : xfs
> > > Device : /dev/mapper/vg_guests-lv_brick2
> > > Mount Options : rw,noatime
> > > Inode Size : 512
> > > Disk Space Free : 418.3GB
> > > Total Disk Space : 433.8GB
> > > Inode Count : 227540992
> > > Free Inodes : 227540973
> > > 
> > > fuse-mount of the gv_pri volume:
> > > 
> > > 
> > > 
> > > [root at networker ~]# ll -h /mnt/gluspri/
> > > totale 37G
> > > -rw-------. 1 qemu qemu 7,7G 24 gen 10:21 alfresco.qc2
> > > -rw-------. 1 qemu qemu 4,2G 24 gen 10:22 check_mk-salmo.qc2
> > > -rw-------. 1 qemu qemu 27M 23 gen 16:42 newnxserver.qc2
> > > -rw-------. 1 qemu qemu 1,1G 23 gen 13:38 newubutest1.qc2
> > > -rw-------. 1 qemu qemu 11G 24 gen 10:17 nxserver.qc2
> > > -rw-------. 1 qemu qemu 8,1G 24 gen 10:17 remlog.qc2
> > > -rw-------. 1 qemu qemu 5,6G 24 gen 10:19 ubutest1.qc2
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Do you think this is the expected behaviour, maybe due to caching? What
> > > if
> > > the most updated node goes down while the VMs are running?
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Thanks a lot,
> > > 
> > > 
> > > 
> > > 
> > > Fabio Rosati
> > > 
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > 
>