[Gluster-users] geo-rep: remote operation failed - No such file or directory

[ML mail]

Hello,

I noticed that the geo-replication of a volume has STATUS "Faulty" and while looking in the *.gluster.log file in /var/log/glusterfs/geo-replication-slaves/ on my slave I can see the following relevant problem:

[2016-02-15 10:58:40.402516] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-myvolume-geo-client-0: changing port to 49152 (from 0)
[2016-02-15 10:58:40.403928] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-myvolume-geo-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-02-15 10:58:40.404130] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-myvolume-geo-client-0: Connected to myvolume-geo-client-0, attached to remote volume '/data/myvolume-geo/brick'.
[2016-02-15 10:58:40.404150] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-myvolume-geo-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-02-15 10:58:40.410150] I [fuse-bridge.c:5137:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-02-15 10:58:40.410223] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-myvolume-geo-client-0: Server lk version = 1
[2016-02-15 10:58:40.410370] I [fuse-bridge.c:4030:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.23
[2016-02-15 10:58:45.662416] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 0-myvolume-geo-dht: renaming /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_03_Rosen.JPG-chunking-2242590604-0.FpKL3SIUb9vKHyjd.part (hash=myvolume-geo-client-0/cache=myvolume-geo-client-0) => /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_03_Rosen.JPG-chunking-2242590604-0 (hash=myvolume-geo-client-0/cache=<nul>)
[2016-02-15 10:58:45.665144] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 0-myvolume-geo-dht: renaming /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_03_Rosen.JPG-chunking-2242590604-1.C6l0DEurb2y3Azw4.part (hash=myvolume-geo-client-0/cache=myvolume-geo-client-0) => /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_03_Rosen.JPG-chunking-2242590604-1 (hash=myvolume-geo-client-0/cache=<nul>)
[2016-02-15 10:58:45.749829] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 0-myvolume-geo-dht: renaming /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_02_Pilz.JPG-chunking-628343631-0.ajEnSguUZ7EkzjzT.part (hash=myvolume-geo-client-0/cache=myvolume-geo-client-0) => /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_02_Pilz.JPG-chunking-628343631-0 (hash=myvolume-geo-client-0/cache=<nul>)
[2016-02-15 10:58:45.750225] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-myvolume-geo-client-0: remote operation failed. Path: /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_02_Pilz.JPG-chunking-628343631-0.ajEnSguUZ7EkzjzT.part (9164caeb-740d-4429-a3bd-c85f40c35e11) [No such file or directory]
[2016-02-15 10:58:45.750418] W [fuse-bridge.c:1777:fuse_rename_cbk] 0-glusterfs-fuse: 60: /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_02_Pilz.JPG-chunking-628343631-0.ajEnSguUZ7EkzjzT.part -> /.gfid/94310944-7f8a-421d-a51f-1e23e28da9cc/Bild_02_Pilz.JPG-chunking-628343631-0 => -1 (Device or resource busy)
[2016-02-15 10:58:45.767788] I [fuse-bridge.c:4984:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-bZ9SMt
[2016-02-15 10:58:45.768063] W [glusterfsd.c:1236:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7feb610820a4] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7feb626f45b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) [0x7feb626f4429] ) 0-: received signum (15), shutting down
[2016-02-15 10:58:45.768093] I [fuse-bridge.c:5683:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-bZ9SMt'.
[2016-02-15 10:58:54.871855] I [dict.c:473:dict_get] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_setxattr_cbk+0x26) [0x7f8313dfb166] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/system/posix-acl.so(handling_other_acl_related_xattr+0x20) [0x7f8313dfb060] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93) [0x7f831f3f40c3] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-02-15 10:58:54.871914] I [dict.c:473:dict_get] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_setxattr_cbk+0x26) [0x7f8313dfb166] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.6/xlator/system/posix-acl.so(handling_other_acl_related_xattr+0xb0) [0x7f8313dfb0f0] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0x93) [0x7f831f3f40c3] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]

This error gets repeated forever with always the same files. I tried to stop and restart the geo-rep on the master but still the same problem and geo replication does not proceed. Does anyone have an idea how to fix this?

I am using GlusterFS 3.7.6 on Debian 8 with a two node replicate volume (1 brick per node) and one single off-site slave node for geo-rep.

Regards
ML