[Gluster-users] geo-replication 3.6.7 -

Wed Dec 2 17:14:26 UTC 2015

Hi,

we need some help...we are running a 6 node distributed-replicated 
master-volume which needs to be geo-replicated to another equal 
configured volume.
recently both locations were updated from 3.4.7 to 3.5.6 to 
3.6.7...running on ubuntu 14.04 lts.
since update to 3.5.6 the geo-replication remains in status 'hybrid 
crawl' as shown below.

[ 16:31:46 ] - root at gluster-ger-ber-09  /home/admin $gluster volume 
geo-replication ger-ber-01 gluster-wien-02::aut-wien-01 status detail

MASTER NODE           MASTER VOL    MASTER BRICK 
SLAVE                               STATUS     CHECKPOINT STATUS    
CRAWL STATUS       FILES SYNCD    FILES PENDING    BYTES PENDING    
DELETES PENDING    FILES SKIPPED
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
gluster-ger-ber-09    ger-ber-01    /gluster-export 
gluster-wien-05-int::aut-wien-01    Active N/A                  
Changelog Crawl    214645 4                0                2 11
gluster-ger-ber-12    ger-ber-01    /gluster-export 
gluster-wien-06-int::aut-wien-01    Passive N/A                  
N/A                0 0                0                0 0
gluster-ger-ber-11    ger-ber-01    /gluster-export 
gluster-wien-03-int::aut-wien-01    Active N/A                  Hybrid 
Crawl       2831091 0                0                0 491
gluster-ger-ber-10    ger-ber-01    /gluster-export 
gluster-wien-02-int::aut-wien-01    Passive N/A                  
N/A                0 0                0                0 0
gluster-ger-ber-07    ger-ber-01    /gluster-export 
gluster-wien-07-int::aut-wien-01    Active N/A                  Hybrid 
Crawl       2886662 0                0                0 474
gluster-ger-ber-08    ger-ber-01    /gluster-export 
gluster-wien-04-int::aut-wien-01    Passive N/A                  
N/A                0 0                0                0 0
[ 16:31:51 ] - root at gluster-ger-ber-09  /home/admin $

on a slave we found a lot of repeating messages like :

root at gluster-wien-07  /var/log/glusterfs/geo-replication-slaves $tail -f 
6a071cfa-b150-4f0b-b1ed-96ab5d4bd671\:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.gluster.log
...
[2015-12-01 23:30:19.782137] W 
[client-rpc-fops.c:240:client3_3_mknod_cbk] 0-aut-wien-01-client-2: 
remote operation failed: File exists. Path: 
<gfid:135991e9-5591-4217-b932-04cce065982f>/rfvg2cmFNJ8Xt9HG.png
[2015-12-01 23:30:19.782178] W 
[client-rpc-fops.c:240:client3_3_mknod_cbk] 0-aut-wien-01-client-3: 
remote operation failed: File exists. Path: 
<gfid:135991e9-5591-4217-b932-04cce065982f>/rfvg2cmFNJ8Xt9HG.png
[2015-12-01 23:30:19.782573] W [fuse-bridge.c:1261:fuse_err_cbk] 
0-glusterfs-fuse: 258115: SETXATTR() 
/.gfid/135991e9-5591-4217-b932-04cce065982f => -1 (File exists)
...

all mentioned files have the same structure on the master volume...they 
appear on two sub-volumes.
as you can see below, on one subvolume the mentioned file was created on 
the 8th jun. on that day we had a rebalance on the master-volume.
I'm not really sure if this dht link file should have 0 bytes on this 
subvolume ?
i already read this but i believe our situation is a bit more complicated...
https://www.gluster.org/pipermail/gluster-users/2015-February/020608.html

is it safe to delete those link files and the corresponding gfid on the 
affected bricks ?

best regards
dietmar

tron at dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png master_gfid_file.txt
e95a43c2-4969-4662-8ea9-72e02e82df90 
thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png

tron at dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png 
ls-lisa-gluster.gluster-ger-ber-*
ls-lisa-gluster.gluster-ger-ber-07-int.out:121247406422 612 -rw-rw-rw- 2 
root root 619294 Okt 18  2014 
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-ger-ber-08-int.out:121289252163 612 -rw-rw-rw- 2 
root root 619294 Okt 18  2014 
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-ger-ber-11-int.out:43282956834 4 ---------T 2 
root root 619294 Jun  8 09:52 
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-ger-ber-12-int.out:65271376731 4 ---------T 2 
root root 619294 Jun  8 09:52 
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png

tron at dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png 
ls-lisa-gluster.gluster-wien-0*
ls-lisa-gluster.gluster-wien-04.out:82283414551 612 -rw-rw-rw- 2 root 
root 619294 Oct 18  2014 
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
ls-lisa-gluster.gluster-wien-05.out:82278502871 612 -rw-rw-rw- 2 root 
root 619294 Oct 18  2014 
/gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png
tron at dp-server:~/gluster-9$

-- 
Dietmar Putz
3Q Medien GmbH
Wetzlarer Str. 86
D-14482 Potsdam

Telefax:  +49 (0)331 / 2797 866 - 1
Telefon:  +49 (0)331 / 2792 866 - 8
Mobile:   +49 171 / 90 160 39
Mail:     putz at 3qmedien.net