[Gluster-users] geo-replication not syncing files...

Wed Nov 11 11:00:44 UTC 2015

the port 2503 is listening on all gluster nodes and i can successfully 
login to the participating slaves...

[ 09:55:15 ] - root at gluster-ger-ber-09  ~/.ssh $ssh -p 2503 
gluster-wien-05 hostname
gluster-wien-05
[ 09:55:19 ] - root at gluster-ger-ber-09  ~/.ssh $

the authorized_keys on all slaves contain corresponding lines for the 
tar command and gsyncd for each master node.
the gsyncd.conf on all master nodes are prepared to use port 2503 for 
the ssh_command and the ssh_command_tar.

privilege-separation is used on all nodes...:

UsePrivilegeSeparation yes

i have to examine if it possibly has changed by ubuntu upgrade, openssh 
package was upgraded in 14.04...in 12.04 with 3.4.7 geo-replication 
worked fine on port 2503.
i just changed UsePrivilegeSeparation to 'no' but for the first 10 min 
nothing changed...

because the slave volume already contain about 99% of the master data by 
the former 3.4.7 geo-replication i assumed changelog could be the right 
setting...

does anybody have further hints...?

best regards
dietmar

On 11.11.2015 02:38, Wade Fitzpatrick wrote:
> Your ssh commands connect to port 2503 - is that port listening on the 
> slaves?
> Does it use privilege-separation?
>
> Don't force it to changelog without an initial sync using xsync.
>
> The warning "fuse: xlator does not implement release_cbk" was fixed in 
> 3.6.0alpha1 but looks like it could easily be backported 
> https://github.com/gluster/glusterfs/commit/bca9eab359710eb3b826c6441126e2e56f774df5
>
> On 11/11/2015 3:20 AM, Dietmar Putz wrote:
>> Hi all,
>>
>> i need some help with a geo-replication issue...
>> recently i upgraded two 6-node distributed-replicated gluster from 
>> ubuntu 12.04.5 lts to 14.04.3 lts resp. glusterfs 3.4.7 to 3.5.6
>> since then the geo-replication does not start syncing but remains as 
>> shown in the 'status detail' output below for about 48h.
>>
>> I followed the hints for upgrade with an existing geo-replication :
>> http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5
>>
>> the master_gfid_file.txt was created and applied to the slave volume. 
>> geo-replication was started with 'force' option.
>> in the gluster.log on the slave i can find thousands of lines with 
>> messages like :
>> ".../.gfid/1abb953b-aa9d-4fa3-9a72-415204057572 => -1 (Operation not 
>> permitted)"
>> and no files are synced.
>>
>> I'm not sure whats going on and since there are about 40TByte of data 
>> already replicated by the old 3.4.7 setup I have some fear to try 
>> around...
>> so i got some questions...maybe somebody can give me some hints...
>>
>> 1. as shown in the example below the trusted.gfid of the same file 
>> differs in master and slave volume. as far as i understood the 
>> upgrade-howto after applying the master_gfid_file.txt on the slave 
>> they should be the same on master and slave...is that right ?
>> 2. as shown in the config below the change_detector is 'xsync'. 
>> Somewhere i red that xsync is used for the initial replication and is 
>> changing to 'change_log' later on when the entire sync is done. 
>> should i try to modify the change_detector to 'change_log', does it 
>> make sense...?
>>
>> any other idea which could help me to solve this problem....?
>>
>> best regards
>> dietmar
>>
>>
>>
>>
>> [ 11:10:01 ] - root at gluster-ger-ber-09  ~ $glusterfs --version
>> glusterfs 3.5.6 built on Sep 16 2015 15:27:30
>> ...
>> [ 11:11:37 ] - root at gluster-ger-ber-09  ~ $cat 
>> /var/lib/glusterd/glusterd.info | grep operating-version
>> operating-version=30501
>>
>>
>> [ 10:55:35 ] - root at gluster-ger-ber-09  ~ $gluster volume 
>> geo-replication ger-ber-01 ssh://gluster-wien-02::aut-wien-01 status 
>> detail
>>
>> MASTER NODE           MASTER VOL    MASTER BRICK 
>> SLAVE                                 STATUS         CHECKPOINT 
>> STATUS    CRAWL STATUS    FILES SYNCD    FILES PENDING    BYTES 
>> PENDING    DELETES PENDING    FILES SKIPPED
>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 
>>
>> gluster-ger-ber-09    ger-ber-01    /gluster-export 
>> gluster-wien-05-int::aut-wien-01      Active N/A                  
>> Hybrid Crawl    0 8191 0                0 0
>> gluster-ger-ber-11    ger-ber-01    /gluster-export 
>> ssh://gluster-wien-02::aut-wien-01 Not Started N/A                  
>> N/A             N/A N/A              N/A              N/A N/A
>> gluster-ger-ber-10    ger-ber-01    /gluster-export 
>> ssh://gluster-wien-02::aut-wien-01 Not Started N/A                  
>> N/A             N/A N/A              N/A              N/A N/A
>> gluster-ger-ber-12    ger-ber-01    /gluster-export 
>> ssh://gluster-wien-02::aut-wien-01 Not Started N/A                  
>> N/A             N/A N/A              N/A              N/A N/A
>> gluster-ger-ber-07    ger-ber-01    /gluster-export 
>> ssh://gluster-wien-02::aut-wien-01 Not Started N/A                  
>> N/A             N/A N/A              N/A              N/A N/A
>> gluster-ger-ber-08    ger-ber-01    /gluster-export 
>> gluster-wien-04-int::aut-wien-01      Passive N/A                  
>> N/A             0 0 0                0 0
>> [ 10:55:48 ] - root at gluster-ger-ber-09  ~ $
>>
>>
>> [ 10:56:56 ] - root at gluster-ger-ber-09  ~ $gluster volume 
>> geo-replication ger-ber-01 ssh://gluster-wien-02::aut-wien-01 config
>> special_sync_mode: partial
>> state_socket_unencoded: 
>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.socket
>> gluster_log_file: 
>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.gluster.log
>> ssh_command: ssh -p 2503 -oPasswordAuthentication=no 
>> -oStrictHostKeyChecking=no -i 
>> /var/lib/glusterd/geo-replication/secret.pem
>> ignore_deletes: true
>> change_detector: xsync
>> ssh_command_tar: ssh -p 2503 -oPasswordAuthentication=no 
>> -oStrictHostKeyChecking=no -i 
>> /var/lib/glusterd/geo-replication/tar_ssh.pem
>> working_dir: 
>> /var/run/gluster/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01
>> remote_gsyncd: /nonexistent/gsyncd
>> log_file: 
>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.log
>> socketdir: /var/run
>> state_file: 
>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.status
>> state_detail_file: 
>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01-detail.status
>> session_owner: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
>> gluster_command_dir: /usr/sbin/
>> pid_file: 
>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/ssh%3A%2F%2Froot%4082.199.131.2%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.pid
>> georep_session_working_dir: 
>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-02_aut-wien-01/
>> gluster_params: aux-gfid-mount
>> volume_id: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
>> [ 11:10:01 ] - root at gluster-ger-ber-09  ~ $
>>
>>
>>
>> [ 12:45:34 ] - root at gluster-wien-05 
>> /var/log/glusterfs/geo-replication-slaves $tail -f 
>> 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671\:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.gluster.log
>> [2015-11-10 12:59:16.097932] W [fuse-bridge.c:1942:fuse_create_cbk] 
>> 0-glusterfs-fuse: 54267: /.gfid/1abb953b-aa9d-4fa3-9a72-415204057572 
>> => -1 (Operation not permitted)
>> [2015-11-10 12:59:16.098044] W [defaults.c:1381:default_release] 
>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.6/xlator/mount/fuse.so(+0xfb4d) 
>> [0x7fc9cd104b4d] 
>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.5.6/xlator/mount/fuse.so(free_fuse_state+0x85) 
>> [0x7fc9cd0fab95] 
>> (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(fd_unref+0x10e) 
>> [0x7fc9cf52ec9e]))) 0-fuse: xlator does not implement release_cbk
>> ...
>>
>>
>> grep 1abb953b-aa9d-4fa3-9a72-415204057572 master_gfid_file.txt
>> 1abb953b-aa9d-4fa3-9a72-415204057572 1050/hyve/364/14158.mp4
>>
>> putz at sdn-de-gate-01:~/central$ ./mycommand.sh -H 
>> gluster-ger,gluster-wien -c "getfattr -m . -d -e hex 
>> /gluster-export/1050/hyve/364/14158.mp4"
>> ...
>> master volume :
>> -----------------------------------------------------
>> Host : gluster-ger-ber-09-int
>> # file: gluster-export/1050/hyve/364/14158.mp4
>> trusted.afr.ger-ber-01-client-6=0x000000000000000000000000
>> trusted.afr.ger-ber-01-client-7=0x000000000000000000000000
>> trusted.gfid=0x1abb953baa9d4fa39a72415204057572
>> trusted.glusterfs.6a071cfa-b150-4f0b-b1ed-96ab5d4bd671.xtime=0x54bff5c40008dd7f 
>>
>> -----------------------------------------------------
>> Host : gluster-ger-ber-10-int
>> # file: gluster-export/1050/hyve/364/14158.mp4
>> trusted.afr.ger-ber-01-client-6=0x000000000000000000000000
>> trusted.afr.ger-ber-01-client-7=0x000000000000000000000000
>> trusted.gfid=0x1abb953baa9d4fa39a72415204057572
>> trusted.glusterfs.6a071cfa-b150-4f0b-b1ed-96ab5d4bd671.xtime=0x54bff5c40008dd7f 
>>
>> ...
>> slave volume :
>> Host : gluster-wien-04
>> # file: gluster-export/1050/hyve/364/14158.mp4
>> trusted.afr.aut-wien-01-client-2=0x000000000000000000000000
>> trusted.afr.aut-wien-01-client-3=0x000000000000000000000000
>> trusted.gfid=0x129ba62c3d214b34beb366fb1e2c8e4b
>> trusted.glusterfs.6a071cfa-b150-4f0b-b1ed-96ab5d4bd671.xtime=0x54bff5c40008dd7f 
>>
>> -----------------------------------------------------
>> Host : gluster-wien-05
>> # file: gluster-export/1050/hyve/364/14158.mp4
>> trusted.afr.aut-wien-01-client-2=0x000000000000000000000000
>> trusted.afr.aut-wien-01-client-3=0x000000000000000000000000
>> trusted.gfid=0x129ba62c3d214b34beb366fb1e2c8e4b
>> trusted.glusterfs.6a071cfa-b150-4f0b-b1ed-96ab5d4bd671.xtime=0x54bff5c40008dd7f 
>>
>> -----------------------------------------------------
>> ...
>> putz at sdn-de-gate-01:~/central$
>>
>>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Dietmar Putz
3Q Medien GmbH
Wetzlarer Str. 86
D-14482 Potsdam

Telefax:  +49 (0)331 / 2797 866 - 1
Telefon:  +49 (0)331 / 2792 866 - 8
Mobile:   +49 171 / 90 160 39
Mail:     putz at 3qmedien.net

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151111/e8272d7f/attachment.html>