[Gluster-users] geo-replication 3.6.7 - no transition from hybrid to changelog crawl
Milind Changire
milindchangire at gmail.com
Thu Jan 21 06:06:50 UTC 2016
Dietmar,
I just learnt that in 3.6.7, the ignore_deletes option gets ignored in the
code and in fact the DELETE operation does get propagated to the slave and
enforced i.e. you should see entries getting deleted on the slave
irrespective of the state of the ignore_deletes option as dumped by the
geo-replication config command.
I too see the "Reserved option" being output at the shell command-line when
attempting to change the ignore_deletes config option. Since the option
gets ignored, changing the option value is not needed.
-----
On a different note, if its not too much to ask, please consider upgrading
to the latest 3.7.x builds. As usual, there's much that's been fixed since
3.6.7 and features that have been added as well. The fixes contain
enforcing the ignore_deletes option and it can actually be toggled and is
effective as well. The fix sets the default value of ignore_deletes to
false which helps propagating the DELETE operations to slave clusters.
--
Milind
On Thu, Jan 21, 2016 at 2:28 AM, Dietmar Putz <putz at 3qmedien.net> wrote:
> Hi Milind,
>
> thanks again for your answer...
> damn...i found an old mail from Venky Shankar and obviously i had a wrong
> view on ignore_deletes....
> but
> [ 20:42:53 ] - root at gluster-ger-ber-07 ~ $gluster volume geo-replication
> ger-ber-01 gluster-wien-07::aut-wien-vol-01 config ignore_deletes false
> Reserved option
> geo-replication command failed
> [ 20:43:06 ] - root at gluster-ger-ber-07 ~ $gluster volume geo-replication
> ger-ber-01 gluster-wien-07::aut-wien-vol-01 config ignore_deletes no
> Reserved option
> geo-replication command failed
> [ 20:43:11 ] - root at gluster-ger-ber-07 ~
> i stopped the geo-replication and tried it again...same result.
> possibly i should start a new replication from scratch and set
> ignore_deletes to false before starting the replication.
>
> meanwhile the reason for the new 'operation not permitted' messages were
> found...
> a directory on the master and a file on the slave in the same
> directory-level with the same name...
>
> best regards
> dietmar
>
>
>
>
>
>
> Am 20.01.2016 um 19:28 schrieb Milind Changire:
>
> Dietmar,
> I just looked at your very first post describing the problem and I found
>
> ignore_deletes: true
>
> in the geo-replication config command output.
>
> If you'd like the slave volume to replicate file deletion as well, then
> the "ignore_deletes" should be set to "false"
> That should help resolve the CREATE + DELETE + CREATE issue.
>
> If this doesn't help, then strace output for gsyncd could be the savior.
>
> -----
> To add further ...
> Lately we've stumbled across another issue with the CREATE + RENAME +
> CREATE sequence on geo-replication restart. Two fixes have been posted and
> are available for review upstream.
>
> geo-rep: avoid creating multiple entries with same gfid --
> <http://review.gluster.org/13186>http://review.gluster.org/13186
> geo-rep: hard-link rename issues on changelog replay --
> <http://review.gluster.org/13189>http://review.gluster.org/13189
>
> I'll post info about the fix propagation plan for the 3.6.x series later.
>
> --
> Milind
>
>
>
> On Wed, Jan 20, 2016 at 11:23 PM, Dietmar Putz <putz at 3qmedien.net> wrote:
>
>> Hi Milind,
>>
>> thank you for your reply...
>> meanwhile i realized that the setfattr command don't work so i decided to
>> delete the affected files and directories but without stopping the
>> geo-replication...i did it before i red you mail.
>> the affected folders are already replicated with the same gfid like on
>> the master...so this is solved for the moment.
>> afterwards i did not receive the 'errcode: 23' messages on the masters
>> and the 'Operation not permitted' messages on the slaves for 2 1/2 days but
>> the geo-replication restarted all the time about every 2 - 4 hours on each
>> active master / slave with below shown "OSError: [Errno 16] Device or
>> resource busy" message on master and slave.
>> anytime the geo-replication restarts i saw following line embedded in the
>> event of restart (as shown far below) :
>>
>> I [dht-layout.c:663:dht_layout_normalize] 0-aut-wien-vol-01-dht: Found
>> anomalies in /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x (gfid
>> = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
>>
>> i have found 24 of these hidden .dst* folders. All of them are stored in
>> the same 2 different subfolders on the master and slave but the shown
>> .dstXXb70G3x is the only one that just exists on the slave volume. I
>> checked that folder and deleted it on the slave since it was empty. i
>> believe that such folders somehow belongs to the geo-replication process
>> but i have no details.
>> however, about one day after deletion this folder was recreated on the
>> slave again but since deletion there are no more 'Found anomalies' messages.
>> currently the geo-replication still restarts frequently with the far
>> below shown message and unfortunately some 'Operation not permitted'
>> messages appears again but for different files than before.
>> I already checked all folders on master/slave for different gfid's but
>> there are no more different gfid's. i guess there is no way around to
>> compare all gfid's of all files on master and slave...since it is a
>> dist-repl. volume there are several million lines.
>> if geo-replication then still not works i will start the suggested strace
>> of the gsyncd. in regard to the strace i have two questions...
>>
>> do i need to start the strace on all active masters / slaves or is it
>> sufficient to trace one active master and the corresponding active slave ?
>> should i try to capture a geo-rep restart by the strace or is it
>> sufficient to let it run one minute randomly ?
>>
>>
>> maybe this is of interest to solve the problem...on the active masters
>> there are lines like :
>> ...
>> [2016-01-19 18:34:58.441606] W [master(/gluster-export):1015:process]
>> _GMaster: incomplete sync, retrying changelogs: XSYNC-CHANGELOG.1453225971
>> [2016-01-19 18:36:27.313515] W [master(/gluster-export):1015:process]
>> _GMaster: incomplete sync, retrying changelogs: XSYNC-CHANGELOG.1453225971
>> [2016-01-19 18:37:56.337660] W [master(/gluster-export):996:process]
>> _GMaster: changelogs XSYNC-CHANGELOG.1453225971 could not be processed -
>> moving on...
>> [2016-01-19 18:37:56.339124] W [master(/gluster-export):1000:process]
>> _GMaster: SKIPPED GFID = 5a47cc07-f32f-4685-ae8e-4969995f3f1c,<huge list
>> with gfid's>
>>
>> they end up in a huge list of comma separated gfid's. is there any hint
>> to get benefit out of this xsync-changelogs, a way to find out what was
>> incomplete ?
>>
>> one more thing that concerns me...i'm trying to understand the
>> distributed geo-replication.
>> the master volume is a living object, accessed by some clients who are
>> upload, delete and recreate files and folders. not very frequent but i
>> observed the mentioned two folders with different gfid's on master / slave
>> and now they are deleted by any client on the master volume. The
>> geo-replication is still in hybrid crawl and afaik can not delete or rename
>> files and folders on the slave volume until changelog mode is reached. When
>> now a client recreates the same folders on the master again they get a new
>> gfid assigned which are different from the still existing gfid's on the
>> slave i believe...so geo-replication should get in conflict again because
>> of existing folders on the slave with the same path but a different gfid
>> than on the master...like for any other files which are deleted and later
>> recreated while geo-rep is in hybrid crawl. is that right ?
>> if so it will be difficult to reach the changelog modus on large gluster
>> volumes because in our case the initial hybrid crawl took some days for
>> about 45 TB...or r/w access needs to be stopped for that time ?
>>
>> thanks in advance and
>> best regards
>> dietmar
>>
>>
>>
>>
>> Am 19.01.2016 um 10:53 schrieb Milind Changire:
>>
>>> Hi Dietmar,
>>> After discussion with Aravinda we realized that unfortunately the
>>> suggestion to:
>>> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR>
>>> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <file-path>
>>>
>>> won't work with 3.6.7, since provision for that workaround was added
>>> after 3.6.7.
>>>
>>> There's an alternative way to achieve the geo-replication:
>>> 1. stop geo-replication
>>> 2. delete files and directories with conflicting gfid on SLAVE
>>> 3. use the "touch" command to touch files and directories with
>>> conflicting gfid
>>> on MASTER
>>> 4. start geo-replication
>>>
>>> This *should* get things correctly replicated to SLAVE.
>>> Geo-replication should start with hybrid-crawl and trigger the
>>> replication to SLAVE.
>>>
>>> If not, then there's more to look at.
>>> You could then send us output of the strace command for the gsyncd
>>> process, while
>>> geo-replication is running:
>>> # strace -ff -p <gsyncd-pid> -o gsyncd-strace
>>>
>>> You could terminate strace after about one minute and send us all the
>>> gsyncd-strace.<pid>
>>> files which will help us debug the issue if its not resolved by the
>>> alternative
>>> mechanism mentioned above.
>>>
>>> Also, crawl status Hybrid Crawl is not an entirely bad thing. It just
>>> could mean
>>> that there's a lot of entries that are being processed. However, if
>>> things don't
>>> return back to the normal state after trying out the alternative
>>> suggestion, we
>>> could take a look at the strace output and get some clues.
>>>
>>> --
>>> Milind
>>>
>>> ----- Original Message -----
>>> From: "Dietmar Putz" <putz at 3qmedien.net>
>>> To: "Aravinda" <avishwan at redhat.com>, gluster-users at gluster.org,
>>> "Milind Changire" <mchangir at redhat.com>
>>> Sent: Thursday, January 14, 2016 11:07:55 PM
>>> Subject: Re: [Gluster-users] geo-replication 3.6.7 - no transition from
>>> hybrid to changelog crawl
>>>
>>> Hello all,
>>>
>>> after some days of inactivity i started another attempt to solve this
>>> geo-replication issue...step by step.
>>> it looks like that some of the directories on the slave volume does not
>>> have the same gfid like the corresponding directory on the master volume.
>>>
>>> for example :
>>>
>>> on a master-node i can see a lot of 'errcode: 23' lines like :
>>> [2016-01-14 09:58:36.96585] W [master(/gluster-export):301:regjob]
>>> _GMaster: Rsync: .gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50 [errcode: 23]
>>>
>>> on the corresponding slave the corresponding message :
>>> [2016-01-14 09:57:06.070452] W [fuse-bridge.c:1967:fuse_create_cbk]
>>> 0-glusterfs-fuse: 1185648: /.gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50
>>> => -1 (Operation not permitted)
>>>
>>> This is the file on the master, the file is still not replicated to the
>>> slave.
>>>
>>> 120533444364 97332 -rw-r--r-- 2 2001 2001 99662854 Jan 8 13:40
>>>
>>> /gluster-export/3912/uploads/BSZ-2015/Z_002895D0-C832-4698-84E6-89F34CDEC2AE_20144555_ST_1.mp4
>>> 120533444364 97332 -rw-r--r-- 2 2001 2001 99662854 Jan 8 13:40
>>> /gluster-export/.glusterfs/a8/d0/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50
>>>
>>> The directory on the slave already contain some files, all of them are
>>> not available on the master anymore, obviously deleted in meantime on
>>> the master by a client.
>>> i have deleted and recreated this file on the master and observed the
>>> logs for recurrence of the newly created gfid of this file...same as
>>> before.
>>>
>>> in http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20703
>>> a user reports a geo-replication problem which is possibly caused by
>>> different gfid's of underlying directories.
>>> and yes, the directory of this file-example above shows that the gfid of
>>> the underlying directory differs from the gfid on the master while the
>>> most other directories have the same gfid.
>>>
>>> master :
>>> ...
>>> # file: gluster-export/3912/uploads/BSP-2012
>>> trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350
>>> --------------
>>> # file: gluster-export/3912/uploads/BSZ-2003
>>> trusted.gfid=0xe80adc088e604234b778997d8e8c2018
>>> --------------
>>> # file: gluster-export/3912/uploads/BSZ-2004
>>> trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced
>>> --------------
>>> # file: gluster-export/3912/uploads/BSZ-2010
>>> trusted.gfid=0x8044e436407d4ed3a67c81df8a7ad47f ###
>>> --------------
>>> # file: gluster-export/3912/uploads/BSZ-2015
>>> trusted.gfid=0x0c30f50480204e02b65d4716a048b029 ###
>>>
>>> slave :
>>> ...
>>> # file: gluster-export/3912/uploads/BSP-2012
>>> trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350
>>> --------------
>>> # file: gluster-export/3912/uploads/BSZ-2003
>>> trusted.gfid=0xe80adc088e604234b778997d8e8c2018
>>> --------------
>>> # file: gluster-export/3912/uploads/BSZ-2004
>>> trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced
>>> --------------
>>> # file: gluster-export/3912/uploads/BSZ-2010
>>> trusted.gfid=0xd83e8fb568c74e33a2091c547512a6ce ###
>>> --------------
>>> # file: gluster-export/3912/uploads/BSZ-2015
>>> trusted.gfid=0xa406e1bec7f3454d8f2ce9c5f9c70eb3 ###
>>>
>>>
>>> now the question...how to fix this..?
>>> in the thread above Aravinda wrote :
>>>
>>> ...
>>> To fix the issue,
>>> -----------------
>>> Find the parent directory of "main.mdb",
>>> Get the GFID of that directory, using getfattr
>>> Check the GFID of the same directory in Slave(To confirm GFIDs are
>>> different)
>>> To fix the issue, Delete that directory in Slave.
>>> Set virtual xattr for that directory and all the files inside that
>>> directory.
>>> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR>
>>> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <file-path>
>>>
>>> Geo-rep will recreate the directory with Proper GFID and starts sync.
>>>
>>> deletion of the affected slave directory might be helpful...
>>> but do i have to execute above shown setfattr commands on the master or
>>> do they just speed up synchronization ?
>>> usually sync should start automatically or could there be a problem
>>> because crawl status is still in 'hybrid crawl'...?
>>>
>>> thanks in advance...
>>> best regards
>>> dietmar
>>>
>>>
>>>
>>>
>>> On 04.01.2016 12:08, Dietmar Putz wrote:
>>>
>>>> Hello Aravinda,
>>>>
>>>> thank you for your reply.
>>>> i just made a 'find /gluster-export -type f -exec ls -lisa {} \; >
>>>> ls-lisa-gluster-export-`hostname`.out' on each brick and checked the
>>>> output for files with less than 2 link counts.
>>>> i found nothing...all files on each brick have exact 2 links.
>>>>
>>>> the entire output for all bricks contain more than 7 million lines
>>>> including .glusterfs but without non relevant directories and files..
>>>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v
>>>> 'indices|landfill|changelogs|health_check' | wc -l
>>>> 7007316
>>>>
>>>> link count is on $4 :
>>>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v
>>>> 'indices|landfill|changelogs|health_check' | awk '{if($4=="2")print}'
>>>> | tail -1
>>>> 62648153697 4 -rw-rw-rw- 2 root root 1713 Jan 4 01:44
>>>> /gluster-export/3500/files/16/01/387233/3500-6dqMmBcVby97PQtR.ism
>>>>
>>>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v
>>>> 'indices|landfill|changelogs|health_check' | awk '{if($4=="1")print}'
>>>> tron at dp-server:~/geo_rep_3$
>>>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v
>>>> 'indices|landfill|changelogs|health_check' | awk '{if($4!="2")print}'
>>>> tron at dp-server:~/geo_rep_3$
>>>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v
>>>> 'indices|landfill|changelogs|health_check' | awk '{print $4}' | sort |
>>>> uniq -c
>>>> 7007316 2
>>>> tron at dp-server:~/geo_rep_3$
>>>>
>>>> If i understood you right this can not be the reason for the problem.
>>>> is there any other hint which i can check on the master or slave to
>>>> analyse the problem....?
>>>>
>>>> Any help would be very appreciated
>>>> best regards
>>>> dietmar
>>>>
>>>>
>>>>
>>>> Am 04.01.2016 um 07:14 schrieb Aravinda:
>>>>
>>>>> Hi,
>>>>>
>>>>> Looks like issue with Geo-rep due to race between Create and Rename.
>>>>> Geo-replication uses gfid-access (Mount Volume with aux-gfid-mount)
>>>>> to create and rename files. If Create and Rename replayed more than
>>>>> once then Geo-rep creates a two files with same GFID(not hardlink).
>>>>> This causes one file without backend GFID link.
>>>>>
>>>>> Milind is working on the patch to disallow the creation of second
>>>>> file with same GFID.
>>>>> @Milind, Please provide more update about your patch.
>>>>>
>>>>> As a workaround, identify all the files in Slave volume which do not
>>>>> have backend links and delete those files(Only in Slaves, keep backup
>>>>> if required)
>>>>>
>>>>> In Brick backend, Crawl and look for files with link count less than
>>>>> 2. (Exclude .glusterfs and .trashcan directory)
>>>>>
>>>>> regards
>>>>> Aravinda
>>>>>
>>>>> On 01/02/2016 09:56 PM, Dietmar Putz wrote:
>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>> one more time i need some help with a geo-replication problem.
>>>>>> recently i started a new geo-replication. the master volume contains
>>>>>> about 45 TB data and the slave volume was new created before
>>>>>> geo-replication setup was done.
>>>>>> master and slave is a 6 node distributed replicated volume running
>>>>>> glusterfs-server 3.6.7-ubuntu1~trusty1.
>>>>>> geo-rep was starting without problems. since few days the slave
>>>>>> volume contains about 200 GB more data than the master volume and i
>>>>>> expected that the crawl status changes from 'hybrid crawl' to
>>>>>> 'changelog crawl' but it remains in 'hybrid crawl'.
>>>>>> the 'status detail' output far below shows more than 10 million
>>>>>> synced files while the entire master volume contains just about 2
>>>>>> million files. some tests show that files are not deleted on the
>>>>>> slave volume.
>>>>>> as far as i know the hybrid crawl has the limitation of not
>>>>>> replicating deletes and renames to the slave thus the geo-rep needs
>>>>>> to achieve the 'changelog crawl' status after initial sync...
>>>>>> usually this should happen more or less automatically, is this right ?
>>>>>>
>>>>>> the geo-rep frequently fails with below shown "OSError: [Errno 16]
>>>>>> Device or resource busy", this error appears about every 3-4 hours
>>>>>> on each active master node.
>>>>>> i guess the frequent appearance of this error prevent geo-rep from
>>>>>> changing to 'changelog crawl', does somebody experienced such
>>>>>> problem, is this the cause of the problem ?
>>>>>>
>>>>>> i found some similar reports on gluster.org for gfs 3.5, 3.6 and 3.7
>>>>>> but none of them point me to a solution...
>>>>>> does anybody know a solution or is there a workaround to achieve the
>>>>>> changelog crawl status...?
>>>>>>
>>>>>> Any help would be very appreciated
>>>>>> best regards
>>>>>> dietmar
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Master gluster-ger-ber-07:
>>>>>> -----------------------------
>>>>>>
>>>>>> [2016-01-02 11:39:48.122546] I [master(/gluster-export):1343:crawl]
>>>>>> _GMaster: processing xsync changelog
>>>>>>
>>>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a
>>>>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724692
>>>>>> [2016-01-02 11:42:55.182342] I [master(/gluster-export):1343:crawl]
>>>>>> _GMaster: processing xsync changelog
>>>>>>
>>>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a
>>>>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724751
>>>>>> [2016-01-02 11:44:11.168962] I [master(/gluster-export):1340:crawl]
>>>>>> _GMaster: finished hybrid crawl syncing, stime: (-1, 0)
>>>>>> [2016-01-02 11:44:11.246845] I
>>>>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master
>>>>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...
>>>>>> [2016-01-02 11:44:11.265209] I
>>>>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 3
>>>>>> seconds
>>>>>> [2016-01-02 11:44:11.896940] I [master(/gluster-export):1192:crawl]
>>>>>> _GMaster: slave's time: (-1, 0)
>>>>>> [2016-01-02 11:44:12.171761] E [repce(/gluster-export):207:__call__]
>>>>>> RepceClient: call 18897:139899553576768:1451735052.09 (entry_ops)
>>>>>> failed on peer with OSError
>>>>>> [2016-01-02 11:44:12.172101] E
>>>>>> [syncdutils(/gluster-export):270:log_raise_exception] <top>: FAIL:
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
>>>>>> line 164, in main
>>>>>> main_i()
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
>>>>>> line 643, in main_i
>>>>>> local.service_loop(*[r for r in [remote] if r])
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
>>>>>> line 1344, in service_loop
>>>>>> g2.crawlwrap()
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
>>>>>> line 539, in crawlwrap
>>>>>> self.crawl(no_stime_update=no_stime_update)
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
>>>>>> line 1204, in crawl
>>>>>> self.process(changes)
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
>>>>>> line 956, in process
>>>>>> self.process_change(change, done, retry)
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
>>>>>> line 920, in process_change
>>>>>> self.slave.server.entry_ops(entries)
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py",
>>>>>> line 226, in __call__
>>>>>> return self.ins(self.meth, *a)
>>>>>> File
>>>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py",
>>>>>> line 208, in __call__
>>>>>> raise res
>>>>>> OSError: [Errno 16] Device or resource busy
>>>>>> [2016-01-02 11:44:12.258982] I
>>>>>> [syncdutils(/gluster-export):214:finalize] <top>: exiting.
>>>>>> [2016-01-02 11:44:12.321808] I [repce(agent):92:service_loop]
>>>>>> RepceServer: terminating on reaching EOF.
>>>>>> [2016-01-02 11:44:12.349766] I [syncdutils(agent):214:finalize]
>>>>>> <top>: exiting.
>>>>>> [2016-01-02 11:44:12.435992] I [monitor(monitor):141:set_state]
>>>>>> Monitor: new state: faulty
>>>>>> [2016-01-02 11:44:23.164284] I [monitor(monitor):215:monitor]
>>>>>> Monitor: ------------------------------------------------------------
>>>>>> [2016-01-02 11:44:23.169981] I [monitor(monitor):216:monitor]
>>>>>> Monitor: starting gsyncd worker
>>>>>> [2016-01-02 11:44:23.216662] I [changelogagent(agent):72:__init__]
>>>>>> ChangelogAgent: Agent listining...
>>>>>> [2016-01-02 11:44:23.239778] I [gsyncd(/gluster-export):633:main_i]
>>>>>> <top>: syncing: gluster://localhost:ger-ber-01 ->
>>>>>> ssh://root@gluster-wien-07-int:gluster://localhost:aut-wien-vol-01
>>>>>> [2016-01-02 11:44:26.358613] I
>>>>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up xsync
>>>>>> change detection mode
>>>>>> [2016-01-02 11:44:26.358983] I
>>>>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as
>>>>>> the sync engine
>>>>>> [2016-01-02 11:44:26.359985] I
>>>>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up
>>>>>> changelog change detection mode
>>>>>> [2016-01-02 11:44:26.360243] I
>>>>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as
>>>>>> the sync engine
>>>>>> [2016-01-02 11:44:26.361159] I
>>>>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up
>>>>>> changeloghistory change detection mode
>>>>>> [2016-01-02 11:44:26.361377] I
>>>>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as
>>>>>> the sync engine
>>>>>> [2016-01-02 11:44:26.402601] I
>>>>>> [master(/gluster-export):1311:register] _GMaster: xsync temp
>>>>>> directory:
>>>>>>
>>>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6e
>>>>>> 41d8d6e56984/xsync
>>>>>> [2016-01-02 11:44:26.402848] I
>>>>>> [resource(/gluster-export):1318:service_loop] GLUSTER: Register
>>>>>> time: 1451735066
>>>>>> [2016-01-02 11:44:27.26012] I
>>>>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master
>>>>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...
>>>>>> [2016-01-02 11:44:27.31605] I
>>>>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 1
>>>>>> seconds
>>>>>> [2016-01-02 11:44:27.66868] I [master(/gluster-export):1226:crawl]
>>>>>> _GMaster: starting history crawl... turns: 1, stime: (-1, 0)
>>>>>> [2016-01-02 11:44:27.67043] I [master(/gluster-export):1229:crawl]
>>>>>> _GMaster: stime not available, abandoning history crawl
>>>>>> [2016-01-02 11:44:27.112426] I
>>>>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master
>>>>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...
>>>>>> [2016-01-02 11:44:27.117506] I
>>>>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 60
>>>>>> seconds
>>>>>> [2016-01-02 11:44:27.140610] I [master(/gluster-export):1333:crawl]
>>>>>> _GMaster: starting hybrid crawl..., stime: (-1, 0)
>>>>>> [2016-01-02 11:45:23.417233] I [monitor(monitor):141:set_state]
>>>>>> Monitor: new state: Stable
>>>>>> [2016-01-02 11:45:48.225915] I [master(/gluster-export):1343:crawl]
>>>>>> _GMaster: processing xsync changelog
>>>>>>
>>>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a
>>>>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735067
>>>>>> [2016-01-02 11:47:08.65231] I [master(/gluster-export):1343:crawl]
>>>>>> _GMaster: processing xsync changelog
>>>>>>
>>>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6
>>>>>> e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735148
>>>>>> ...
>>>>>>
>>>>>>
>>>>>> slave gluster-wien-07 :
>>>>>> ------------------------
>>>>>>
>>>>>> [2016-01-02 11:44:12.007744] W [fuse-bridge.c:1261:fuse_err_cbk]
>>>>>> 0-glusterfs-fuse: 1959820: SETXATTR()
>>>>>> /.gfid/5e436e5b-086b-4720-9e70-0e49c8e09698 => -1 (File exists)
>>>>>> [2016-01-02 11:44:12.010970] W
>>>>>> [client-rpc-fops.c:240:client3_3_mknod_cbk]
>>>>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists.
>>>>>> Path:
>>>>>> <gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm
>>>>>> [2016-01-02 11:44:12.011327] W
>>>>>> [client-rpc-fops.c:240:client3_3_mknod_cbk]
>>>>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists.
>>>>>> Path:
>>>>>> <gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm
>>>>>> [2016-01-02 11:44:12.012054] W [fuse-bridge.c:1261:fuse_err_cbk]
>>>>>> 0-glusterfs-fuse: 1959822: SETXATTR()
>>>>>> /.gfid/666bceac-7c14-4efd-81fe-8185458fcf1f => -1 (File exists)
>>>>>> [2016-01-02 11:44:12.024743] W
>>>>>> [client-rpc-fops.c:240:client3_3_mknod_cbk]
>>>>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists.
>>>>>> Path: <gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png
>>>>>> [2016-01-02 11:44:12.024970] W
>>>>>> [client-rpc-fops.c:240:client3_3_mknod_cbk]
>>>>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists.
>>>>>> Path: <gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png
>>>>>> [2016-01-02 11:44:12.025601] W [fuse-bridge.c:1261:fuse_err_cbk]
>>>>>> 0-glusterfs-fuse: 1959823: SETXATTR()
>>>>>> /.gfid/5bfd6f99-07e8-4b2f-844b-aa0b6535c055 => -1 (File exists)
>>>>>> [2016-01-02 11:44:12.100688] I
>>>>>> [dht-selfheal.c:1065:dht_selfheal_layout_new_directory]
>>>>>> 0-aut-wien-vol-01-dht: chunk size = 0xffffffff / 57217563 = 0x4b
>>>>>> [2016-01-02 11:44:12.100765] I
>>>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
>>>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to
>>>>>> aut-wien-vol-01-replicate-0
>>>>>> [2016-01-02 11:44:12.100785] I
>>>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
>>>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to
>>>>>> aut-wien-vol-01-replicate-1
>>>>>> [2016-01-02 11:44:12.100800] I
>>>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
>>>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to
>>>>>> aut-wien-vol-01-replicate-2
>>>>>> [2016-01-02 11:44:12.100839] I [MSGID: 109036]
>>>>>> [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]
>>>>>> 0-aut-wien-vol-01-dht: Setting layout of
>>>>>> <gfid:d4815ee4-3348-4105-9136-d0219d956ed8>/.dstXXX0HUpRD with
>>>>>> [Subvol_name: aut-wien-vol-01-re
>>>>>> plicate-0, Err: -1 , Start: 0 , Stop: 1430439074 ], [Subvol_name:
>>>>>> aut-wien-vol-01-replicate-1, Err: -1 , Start: 1430439075 , Stop:
>>>>>> 2860878149 ], [Subvol_name: aut-wien-vol-01-replicate-2, Err: -1 ,
>>>>>> Start: 2860878150 , Stop: 4294967295 ],
>>>>>> [2016-01-02 11:44:12.114192] W
>>>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
>>>>>> 0-aut-wien-vol-01-client-2: remote operation failed: File exists.
>>>>>> Path: <gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x
>>>>>> [2016-01-02 11:44:12.114275] W
>>>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
>>>>>> 0-aut-wien-vol-01-client-3: remote operation failed: File exists.
>>>>>> Path: <gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x
>>>>>> [2016-01-02 11:44:12.114879] W [fuse-bridge.c:1261:fuse_err_cbk]
>>>>>> 0-glusterfs-fuse: 1959831: SETXATTR()
>>>>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2 => -1 (File exists)
>>>>>> [2016-01-02 11:44:12.118473] I
>>>>>> [dht-layout.c:663:dht_layout_normalize] 0-aut-wien-vol-01-dht: Found
>>>>>> anomalies in
>>>>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x (gfid =
>>>>>> 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
>>>>>> [2016-01-02 11:44:12.118537] I
>>>>>> [dht-selfheal.c:1065:dht_selfheal_layout_new_directory]
>>>>>> 0-aut-wien-vol-01-dht: chunk size = 0xffffffff / 57217563 = 0x4b
>>>>>> [2016-01-02 11:44:12.118562] I
>>>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
>>>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to
>>>>>> aut-wien-vol-01-replicate-2
>>>>>> [2016-01-02 11:44:12.118579] I
>>>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
>>>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to
>>>>>> aut-wien-vol-01-replicate-0
>>>>>> [2016-01-02 11:44:12.118613] I
>>>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
>>>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to
>>>>>> aut-wien-vol-01-replicate-1
>>>>>> [2016-01-02 11:44:12.120352] I [MSGID: 109036]
>>>>>> [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]
>>>>>> 0-aut-wien-vol-01-dht: Setting layout of
>>>>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x with
>>>>>> [Subvol_name: aut-wien-vol-01-rep
>>>>>> licate-0, Err: -1 , Start: 1430439075 , Stop: 2860878149 ],
>>>>>> [Subvol_name: aut-wien-vol-01-replicate-1, Err: -1 , Start:
>>>>>> 2860878150 , Stop: 4294967295 ], [Subvol_name:
>>>>>> aut-wien-vol-01-replicate-2, Err: -1 , Start: 0 , Stop: 1430439074 ],
>>>>>> [2016-01-02 11:44:12.630949] I [fuse-bridge.c:4927:fuse_thread_proc]
>>>>>> 0-fuse: unmounting /tmp/gsyncd-aux-mount-tOUOsz
>>>>>> [2016-01-02 11:44:12.633952] W [glusterfsd.c:1211:cleanup_and_exit]
>>>>>> (--> 0-: received signum (15), shutting down
>>>>>> [2016-01-02 11:44:12.633964] I [fuse-bridge.c:5607:fini] 0-fuse:
>>>>>> Unmounting '/tmp/gsyncd-aux-mount-tOUOsz'.
>>>>>> [2016-01-02 11:44:23.946702] I [MSGID: 100030]
>>>>>> [glusterfsd.c:2035:main] 0-/usr/sbin/glusterfs: Started running
>>>>>> /usr/sbin/glusterfs version 3.6.7 (args: /usr/sbin/glusterfs
>>>>>> --aux-gfid-mount --log-file=/var/log/glusterfs/geo-replication-slav
>>>>>>
>>>>>> es/6a071cfa-b150-4f0b-b1ed-96ab5d4bd671:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log
>>>>>> --volfile-server=localhost --volfile-id=aut-wien-vol-01
>>>>>> --client-pid=-1 /tmp/gsyncd-aux-mount-otU3wS)
>>>>>> [2016-01-02 11:44:24.042128] I [dht-shared.c:337:dht_init_regex]
>>>>>> 0-aut-wien-vol-01-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>>>>>> [2016-01-02 11:44:24.046315] I [client.c:2268:notify]
>>>>>> 0-aut-wien-vol-01-client-0: parent translators are ready, attempting
>>>>>> connect on transport
>>>>>> [2016-01-02 11:44:24.046532] I [client.c:2268:notify]
>>>>>> 0-aut-wien-vol-01-client-1: parent translators are ready, attempting
>>>>>> connect on transport
>>>>>> [2016-01-02 11:44:24.046664] I [client.c:2268:notify]
>>>>>> 0-aut-wien-vol-01-client-2: parent translators are ready, attempting
>>>>>> connect on transport
>>>>>> [2016-01-02 11:44:24.046806] I [client.c:2268:notify]
>>>>>> 0-aut-wien-vol-01-client-3: parent translators are ready, attempting
>>>>>> connect on transport
>>>>>> [2016-01-02 11:44:24.046940] I [client.c:2268:notify]
>>>>>> 0-aut-wien-vol-01-client-4: parent translators are ready, attempting
>>>>>> connect on transport
>>>>>> [2016-01-02 11:44:24.047070] I [client.c:2268:notify]
>>>>>> 0-aut-wien-vol-01-client-5: parent translators are ready, attempting
>>>>>> connect on transport
>>>>>> Final graph:
>>>>>>
>>>>>> +------------------------------------------------------------------------------+
>>>>>>
>>>>>> 1: volume aut-wien-vol-01-client-0
>>>>>> 2: type protocol/client
>>>>>> 3: option ping-timeout 10
>>>>>> 4: option remote-host gluster-wien-02-int
>>>>>> 5: option remote-subvolume /gluster-export
>>>>>> 6: option transport-type socket
>>>>>> 7: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
>>>>>> 8: option password 8777e154-476c-449a-89b2-3199872e4a1f
>>>>>> 9: option send-gids true
>>>>>> 10: end-volume
>>>>>> 11:
>>>>>> 12: volume aut-wien-vol-01-client-1
>>>>>> 13: type protocol/client
>>>>>> 14: option ping-timeout 10
>>>>>> 15: option remote-host gluster-wien-03-int
>>>>>> 16: option remote-subvolume /gluster-export
>>>>>> 17: option transport-type socket
>>>>>> 18: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
>>>>>> 19: option password 8777e154-476c-449a-89b2-3199872e4a1f
>>>>>> 20: option send-gids true
>>>>>> 21: end-volume
>>>>>> 22:
>>>>>> 23: volume aut-wien-vol-01-replicate-0
>>>>>> 24: type cluster/replicate
>>>>>> 25: subvolumes aut-wien-vol-01-client-0 aut-wien-vol-01-client-1
>>>>>> 26: end-volume
>>>>>> 27:
>>>>>> 28: volume aut-wien-vol-01-client-2
>>>>>> 29: type protocol/client
>>>>>> 30: option ping-timeout 10
>>>>>> 31: option remote-host gluster-wien-04-int
>>>>>> 32: option remote-subvolume /gluster-export
>>>>>> 33: option transport-type socket
>>>>>> 34: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
>>>>>> 35: option password 8777e154-476c-449a-89b2-3199872e4a1f
>>>>>> 36: option send-gids true
>>>>>> 37: end-volume
>>>>>> 38:
>>>>>> 39: volume aut-wien-vol-01-client-3
>>>>>> 40: type protocol/client
>>>>>> 41: option ping-timeout 10
>>>>>> 42: option remote-host gluster-wien-05-int
>>>>>> 43: option remote-subvolume /gluster-export
>>>>>> 44: option transport-type socket
>>>>>> 45: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
>>>>>> 46: option password 8777e154-476c-449a-89b2-3199872e4a1f
>>>>>> 47: option send-gids true
>>>>>> 48: end-volume
>>>>>> 49:
>>>>>> 50: volume aut-wien-vol-01-replicate-1
>>>>>> 51: type cluster/replicate
>>>>>> 52: subvolumes aut-wien-vol-01-client-2 aut-wien-vol-01-client-3
>>>>>> 53: end-volume
>>>>>> 54:
>>>>>> 55: volume aut-wien-vol-01-client-4
>>>>>> 56: type protocol/client
>>>>>> 57: option ping-timeout 10
>>>>>> 58: option remote-host gluster-wien-06-int
>>>>>> 59: option remote-subvolume /gluster-export
>>>>>> 60: option transport-type socket
>>>>>> 61: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
>>>>>> 62: option password 8777e154-476c-449a-89b2-3199872e4a1f
>>>>>> 63: option send-gids true
>>>>>> 64: end-volume
>>>>>> 65:
>>>>>> 66: volume aut-wien-vol-01-client-5
>>>>>> 67: type protocol/client
>>>>>> 68: option ping-timeout 10
>>>>>> 69: option remote-host gluster-wien-07-int
>>>>>> 70: option remote-subvolume /gluster-export
>>>>>> 71: option transport-type socket
>>>>>> 72: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
>>>>>> 73: option password 8777e154-476c-449a-89b2-3199872e4a1f
>>>>>> 74: option send-gids true
>>>>>> 75: end-volume
>>>>>> 76:
>>>>>> 77: volume aut-wien-vol-01-replicate-2
>>>>>> 78: type cluster/replicate
>>>>>> 79: subvolumes aut-wien-vol-01-client-4 aut-wien-vol-01-client-5
>>>>>> 80: end-volume
>>>>>> 81:
>>>>>> 82: volume aut-wien-vol-01-dht
>>>>>> 83: type cluster/distribute
>>>>>> 84: subvolumes aut-wien-vol-01-replicate-0
>>>>>> aut-wien-vol-01-replicate-1 aut-wien-vol-01-replicate-2
>>>>>> 85: end-volume
>>>>>> 86:
>>>>>> 87: volume aut-wien-vol-01-write-behind
>>>>>> 88: type performance/write-behind
>>>>>> 89: subvolumes aut-wien-vol-01-dht
>>>>>> 90: end-volume
>>>>>> 91:
>>>>>> 92: volume aut-wien-vol-01-read-ahead
>>>>>> 93: type performance/read-ahead
>>>>>> 94: subvolumes aut-wien-vol-01-write-behind
>>>>>> 95: end-volume
>>>>>> 96:
>>>>>> 97: volume aut-wien-vol-01-io-cache
>>>>>> 98: type performance/io-cache
>>>>>> 99: option min-file-size 0
>>>>>> 100: option cache-timeout 2
>>>>>> 101: option cache-size 1024MB
>>>>>> 102: subvolumes aut-wien-vol-01-read-ahead
>>>>>> 103: end-volume
>>>>>> 104:
>>>>>> 105: volume aut-wien-vol-01-quick-read
>>>>>> 106: type performance/quick-read
>>>>>> 107: option cache-size 1024MB
>>>>>> 108: subvolumes aut-wien-vol-01-io-cache
>>>>>> 109: end-volume
>>>>>> 110:
>>>>>> 111: volume aut-wien-vol-01-open-behind
>>>>>> 112: type performance/open-behind
>>>>>> 113: subvolumes aut-wien-vol-01-quick-read
>>>>>> 114: end-volume
>>>>>> 115:
>>>>>> 116: volume aut-wien-vol-01-md-cache
>>>>>> 117: type performance/md-cache
>>>>>> 118: subvolumes aut-wien-vol-01-open-behind
>>>>>> 119: end-volume
>>>>>> 120:
>>>>>> 121: volume aut-wien-vol-01
>>>>>> 122: type debug/io-stats
>>>>>> 123: option latency-measurement off
>>>>>> 124: option count-fop-hits off
>>>>>> 125: subvolumes aut-wien-vol-01-md-cache
>>>>>> 126: end-volume
>>>>>> 127:
>>>>>> 128: volume gfid-access-autoload
>>>>>> 129: type features/gfid-access
>>>>>> 130: subvolumes aut-wien-vol-01
>>>>>> 131: end-volume
>>>>>> 132:
>>>>>> 133: volume meta-autoload
>>>>>> 134: type meta
>>>>>> 135: subvolumes gfid-access-autoload
>>>>>> 136: end-volume
>>>>>> 137:
>>>>>>
>>>>>> +------------------------------------------------------------------------------+
>>>>>>
>>>>>> [2016-01-02 11:44:24.047642] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
>>>>>> 0-aut-wien-vol-01-client-5: changing port to 49153 (from 0)
>>>>>> [2016-01-02 11:44:24.047927] I
>>>>>> [client-handshake.c:1413:select_server_supported_programs]
>>>>>> 0-aut-wien-vol-01-client-5: Using Program GlusterFS 3.3, Num
>>>>>> (1298437), Version (330)
>>>>>> [2016-01-02 11:44:24.048044] I
>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-5: Connected to aut-wien-vol-01-client-5,
>>>>>> attached to remote volume '/gluster-export'.
>>>>>> [2016-01-02 11:44:24.048050] I
>>>>>> [client-handshake.c:1210:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-5: Server and Client lk-version numbers are
>>>>>> not same, reopening the fds
>>>>>> [2016-01-02 11:44:24.048088] I [MSGID: 108005]
>>>>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-2:
>>>>>> Subvolume 'aut-wien-vol-01-client-5' came back up; going online.
>>>>>> [2016-01-02 11:44:24.048114] I
>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>> 0-aut-wien-vol-01-client-5: Server lk version = 1
>>>>>> [2016-01-02 11:44:24.048124] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
>>>>>> 0-aut-wien-vol-01-client-0: changing port to 49153 (from 0)
>>>>>> [2016-01-02 11:44:24.048132] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
>>>>>> 0-aut-wien-vol-01-client-1: changing port to 49153 (from 0)
>>>>>> [2016-01-02 11:44:24.048138] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
>>>>>> 0-aut-wien-vol-01-client-2: changing port to 49153 (from 0)
>>>>>> [2016-01-02 11:44:24.048146] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
>>>>>> 0-aut-wien-vol-01-client-3: changing port to 49153 (from 0)
>>>>>> [2016-01-02 11:44:24.048153] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
>>>>>> 0-aut-wien-vol-01-client-4: changing port to 49153 (from 0)
>>>>>> [2016-01-02 11:44:24.049070] I
>>>>>> [client-handshake.c:1413:select_server_supported_programs]
>>>>>> 0-aut-wien-vol-01-client-0: Using Program GlusterFS 3.3, Num
>>>>>> (1298437), Version (330)
>>>>>> [2016-01-02 11:44:24.049094] I
>>>>>> [client-handshake.c:1413:select_server_supported_programs]
>>>>>> 0-aut-wien-vol-01-client-3: Using Program GlusterFS 3.3, Num
>>>>>> (1298437), Version (330)
>>>>>> [2016-01-02 11:44:24.049113] I
>>>>>> [client-handshake.c:1413:select_server_supported_programs]
>>>>>> 0-aut-wien-vol-01-client-2: Using Program GlusterFS 3.3, Num
>>>>>> (1298437), Version (330)
>>>>>> [2016-01-02 11:44:24.049131] I
>>>>>> [client-handshake.c:1413:select_server_supported_programs]
>>>>>> 0-aut-wien-vol-01-client-1: Using Program GlusterFS 3.3, Num
>>>>>> (1298437), Version (330)
>>>>>> [2016-01-02 11:44:24.049224] I
>>>>>> [client-handshake.c:1413:select_server_supported_programs]
>>>>>> 0-aut-wien-vol-01-client-4: Using Program GlusterFS 3.3, Num
>>>>>> (1298437), Version (330)
>>>>>> [2016-01-02 11:44:24.049307] I
>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-0: Connected to aut-wien-vol-01-client-0,
>>>>>> attached to remote volume '/gluster-export'.
>>>>>> [2016-01-02 11:44:24.049312] I
>>>>>> [client-handshake.c:1210:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-0: Server and Client lk-version numbers are
>>>>>> not same, reopening the fds
>>>>>> [2016-01-02 11:44:24.049324] I [MSGID: 108005]
>>>>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-0:
>>>>>> Subvolume 'aut-wien-vol-01-client-0' came back up; going online.
>>>>>> [2016-01-02 11:44:24.049384] I
>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-3: Connected to aut-wien-vol-01-client-3,
>>>>>> attached to remote volume '/gluster-export'.
>>>>>> [2016-01-02 11:44:24.049389] I
>>>>>> [client-handshake.c:1210:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-3: Server and Client lk-version numbers are
>>>>>> not same, reopening the fds
>>>>>> [2016-01-02 11:44:24.049400] I [MSGID: 108005]
>>>>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-1:
>>>>>> Subvolume 'aut-wien-vol-01-client-3' came back up; going online.
>>>>>> [2016-01-02 11:44:24.049418] I
>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-2: Connected to aut-wien-vol-01-client-2,
>>>>>> attached to remote volume '/gluster-export'.
>>>>>> [2016-01-02 11:44:24.049422] I
>>>>>> [client-handshake.c:1210:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-2: Server and Client lk-version numbers are
>>>>>> not same, reopening the fds
>>>>>> [2016-01-02 11:44:24.049460] I
>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-1: Connected to aut-wien-vol-01-client-1,
>>>>>> attached to remote volume '/gluster-export'.
>>>>>> [2016-01-02 11:44:24.049465] I
>>>>>> [client-handshake.c:1210:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-1: Server and Client lk-version numbers are
>>>>>> not same, reopening the fds
>>>>>> [2016-01-02 11:44:24.049493] I
>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>> 0-aut-wien-vol-01-client-0: Server lk version = 1
>>>>>> [2016-01-02 11:44:24.049567] I
>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>> 0-aut-wien-vol-01-client-3: Server lk version = 1
>>>>>> [2016-01-02 11:44:24.049632] I
>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-4: Connected to aut-wien-vol-01-client-4,
>>>>>> attached to remote volume '/gluster-export'.
>>>>>> [2016-01-02 11:44:24.049638] I
>>>>>> [client-handshake.c:1210:client_setvolume_cbk]
>>>>>> 0-aut-wien-vol-01-client-4: Server and Client lk-version numbers are
>>>>>> not same, reopening the fds
>>>>>> [2016-01-02 11:44:24.052103] I [fuse-bridge.c:5086:fuse_graph_setup]
>>>>>> 0-fuse: switched to graph 0
>>>>>> [2016-01-02 11:44:24.052150] I
>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>> 0-aut-wien-vol-01-client-2: Server lk version = 1
>>>>>> [2016-01-02 11:44:24.052163] I
>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>> 0-aut-wien-vol-01-client-4: Server lk version = 1
>>>>>> [2016-01-02 11:44:24.052192] I
>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>> 0-aut-wien-vol-01-client-1: Server lk version = 1
>>>>>> [2016-01-02 11:44:24.052204] I [fuse-bridge.c:4015:fuse_init]
>>>>>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22
>>>>>> kernel 7.20
>>>>>> [2016-01-02 11:44:24.053991] I
>>>>>> [afr-common.c:1491:afr_local_discovery_cbk]
>>>>>> 0-aut-wien-vol-01-replicate-2: selecting local read_child
>>>>>> aut-wien-vol-01-client-5
>>>>>> [2016-01-02 11:45:48.613563] W
>>>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
>>>>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists.
>>>>>> Path: /keys
>>>>>> [2016-01-02 11:45:48.614131] W
>>>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
>>>>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists.
>>>>>> Path: /keys
>>>>>> [2016-01-02 11:45:48.614436] W [fuse-bridge.c:1261:fuse_err_cbk]
>>>>>> 0-glusterfs-fuse: 12: SETXATTR()
>>>>>> /.gfid/00000000-0000-0000-0000-000000000001 => -1 (File exists)
>>>>>> ...
>>>>>>
>>>>>>
>>>>>> [ 13:41:40 ] - root at gluster-ger-ber-07
>>>>>> /var/log/glusterfs/geo-replication/ger-ber-01 $gluster volume
>>>>>> geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 status
>>>>>> detail
>>>>>>
>>>>>> MASTER NODE MASTER VOL MASTER BRICK
>>>>>> SLAVE STATUS CHECKPOINT
>>>>>> STATUS CRAWL STATUS FILES SYNCD FILES PENDING BYTES
>>>>>> PENDING DELETES PENDING FILES SKIPPED
>>>>>>
>>>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>>
>>>>>> gluster-ger-ber-07 ger-ber-01 /gluster-export
>>>>>> gluster-wien-07-int::aut-wien-vol-01 Active N/A
>>>>>> Hybrid Crawl 10743644 8192 0 0 0
>>>>>> gluster-ger-ber-11 ger-ber-01 /gluster-export
>>>>>> gluster-wien-03-int::aut-wien-vol-01 Active N/A
>>>>>> Hybrid Crawl 16037091 8192 0 0 0
>>>>>> gluster-ger-ber-10 ger-ber-01 /gluster-export
>>>>>> gluster-wien-02-int::aut-wien-vol-01 Passive N/A
>>>>>> N/A 0 0 0 0 0
>>>>>> gluster-ger-ber-12 ger-ber-01 /gluster-export
>>>>>> gluster-wien-06-int::aut-wien-vol-01 Passive N/A
>>>>>> N/A 0 0 0 0 0
>>>>>> gluster-ger-ber-09 ger-ber-01 /gluster-export
>>>>>> gluster-wien-05-int::aut-wien-vol-01 Active N/A
>>>>>> Hybrid Crawl 16180514 8192 0 0 0
>>>>>> gluster-ger-ber-08 ger-ber-01 /gluster-export
>>>>>> gluster-wien-04-int::aut-wien-vol-01 Passive N/A
>>>>>> N/A 0 0 0 0 0
>>>>>>
>>>>>>
>>>>>> [ 13:41:55 ] - root at gluster-ger-ber-07
>>>>>> /var/log/glusterfs/geo-replication/ger-ber-01 $gluster volume
>>>>>> geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 config
>>>>>> special_sync_mode: partial
>>>>>> state_socket_unencoded:
>>>>>>
>>>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.socket
>>>>>> gluster_log_file:
>>>>>>
>>>>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log
>>>>>> ssh_command: ssh -p 2503 -oPasswordAuthentication=no
>>>>>> -oStrictHostKeyChecking=no -i
>>>>>> /var/lib/glusterd/geo-replication/secret.pem
>>>>>> ignore_deletes: true
>>>>>> change_detector: changelog
>>>>>> ssh_command_tar: ssh -p 2503 -oPasswordAuthentication=no
>>>>>> -oStrictHostKeyChecking=no -i
>>>>>> /var/lib/glusterd/geo-replication/tar_ssh.pem
>>>>>> state_file:
>>>>>>
>>>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.status
>>>>>> remote_gsyncd: /nonexistent/gsyncd
>>>>>> log_file:
>>>>>>
>>>>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.log
>>>>>> changelog_log_file:
>>>>>>
>>>>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-changes.log
>>>>>> socketdir: /var/run
>>>>>> working_dir:
>>>>>>
>>>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01
>>>>>> state_detail_file:
>>>>>>
>>>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-detail.status
>>>>>> session_owner: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
>>>>>> gluster_command_dir: /usr/sbin/
>>>>>> pid_file:
>>>>>>
>>>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.pid
>>>>>> georep_session_working_dir:
>>>>>>
>>>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/
>>>>>> gluster_params: aux-gfid-mount
>>>>>> volume_id: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
>>>>>> [ 13:42:11 ] - root at gluster-ger-ber-07
>>>>>> /var/log/glusterfs/geo-replication/ger-ber-01 $
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160121/36c61dd7/attachment.html>
More information about the Gluster-users
mailing list