[Gluster-users] geo-replication 3.6.7 - no transition from hybrid to changelog crawl
Dietmar Putz
putz at 3qmedien.net
Wed Jan 20 20:58:24 UTC 2016
Hi Milind,
thanks again for your answer...
damn...i found an old mail from Venky Shankar and obviously i had a
wrong view on ignore_deletes....
but
[ 20:42:53 ] - root at gluster-ger-ber-07 ~ $gluster volume
geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 config
ignore_deletes false
Reserved option
geo-replication command failed
[ 20:43:06 ] - root at gluster-ger-ber-07 ~ $gluster volume
geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 config
ignore_deletes no
Reserved option
geo-replication command failed
[ 20:43:11 ] - root at gluster-ger-ber-07 ~
i stopped the geo-replication and tried it again...same result.
possibly i should start a new replication from scratch and set
ignore_deletes to false before starting the replication.
meanwhile the reason for the new 'operation not permitted' messages were
found...
a directory on the master and a file on the slave in the same
directory-level with the same name...
best regards
dietmar
Am 20.01.2016 um 19:28 schrieb Milind Changire:
> Dietmar,
> I just looked at your very first post describing the problem and I found
>
> ignore_deletes: true
>
> in the geo-replication config command output.
>
> If you'd like the slave volume to replicate file deletion as well,
> then the "ignore_deletes" should be set to "false"
> That should help resolve the CREATE + DELETE + CREATE issue.
>
> If this doesn't help, then strace output for gsyncd could be the savior.
>
> -----
> To add further ...
> Lately we've stumbled across another issue with the CREATE + RENAME +
> CREATE sequence on geo-replication restart. Two fixes have been posted
> and are available for review upstream.
>
> geo-rep: avoid creating multiple entries with same gfid --
> http://review.gluster.org/13186
> geo-rep: hard-link rename issues on changelog replay --
> http://review.gluster.org/13189
>
> I'll post info about the fix propagation plan for the 3.6.x series later.
>
> --
> Milind
>
>
>
> On Wed, Jan 20, 2016 at 11:23 PM, Dietmar Putz <putz at 3qmedien.net
> <mailto:putz at 3qmedien.net>> wrote:
>
> Hi Milind,
>
> thank you for your reply...
> meanwhile i realized that the setfattr command don't work so i
> decided to delete the affected files and directories but without
> stopping the geo-replication...i did it before i red you mail.
> the affected folders are already replicated with the same gfid
> like on the master...so this is solved for the moment.
> afterwards i did not receive the 'errcode: 23' messages on the
> masters and the 'Operation not permitted' messages on the slaves
> for 2 1/2 days but the geo-replication restarted all the time
> about every 2 - 4 hours on each active master / slave with below
> shown "OSError: [Errno 16] Device or resource busy" message on
> master and slave.
> anytime the geo-replication restarts i saw following line embedded
> in the event of restart (as shown far below) :
>
> I [dht-layout.c:663:dht_layout_normalize] 0-aut-wien-vol-01-dht:
> Found anomalies in
> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x (gfid =
> 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
>
> i have found 24 of these hidden .dst* folders. All of them are
> stored in the same 2 different subfolders on the master and slave
> but the shown .dstXXb70G3x is the only one that just exists on the
> slave volume. I checked that folder and deleted it on the slave
> since it was empty. i believe that such folders somehow belongs to
> the geo-replication process but i have no details.
> however, about one day after deletion this folder was recreated on
> the slave again but since deletion there are no more 'Found
> anomalies' messages.
> currently the geo-replication still restarts frequently with the
> far below shown message and unfortunately some 'Operation not
> permitted' messages appears again but for different files than before.
> I already checked all folders on master/slave for different gfid's
> but there are no more different gfid's. i guess there is no way
> around to compare all gfid's of all files on master and
> slave...since it is a dist-repl. volume there are several million
> lines.
> if geo-replication then still not works i will start the suggested
> strace of the gsyncd. in regard to the strace i have two questions...
>
> do i need to start the strace on all active masters / slaves or is
> it sufficient to trace one active master and the corresponding
> active slave ?
> should i try to capture a geo-rep restart by the strace or is it
> sufficient to let it run one minute randomly ?
>
>
> maybe this is of interest to solve the problem...on the active
> masters there are lines like :
> ...
> [2016-01-19 18:34:58.441606] W
> [master(/gluster-export):1015:process] _GMaster: incomplete sync,
> retrying changelogs: XSYNC-CHANGELOG.1453225971
> [2016-01-19 18:36:27.313515] W
> [master(/gluster-export):1015:process] _GMaster: incomplete sync,
> retrying changelogs: XSYNC-CHANGELOG.1453225971
> [2016-01-19 18:37:56.337660] W
> [master(/gluster-export):996:process] _GMaster: changelogs
> XSYNC-CHANGELOG.1453225971 could not be processed - moving on...
> [2016-01-19 18:37:56.339124] W
> [master(/gluster-export):1000:process] _GMaster: SKIPPED GFID =
> 5a47cc07-f32f-4685-ae8e-4969995f3f1c,<huge list with gfid's>
>
> they end up in a huge list of comma separated gfid's. is there any
> hint to get benefit out of this xsync-changelogs, a way to find
> out what was incomplete ?
>
> one more thing that concerns me...i'm trying to understand the
> distributed geo-replication.
> the master volume is a living object, accessed by some clients who
> are upload, delete and recreate files and folders. not very
> frequent but i observed the mentioned two folders with different
> gfid's on master / slave and now they are deleted by any client on
> the master volume. The geo-replication is still in hybrid crawl
> and afaik can not delete or rename files and folders on the slave
> volume until changelog mode is reached. When now a client
> recreates the same folders on the master again they get a new gfid
> assigned which are different from the still existing gfid's on the
> slave i believe...so geo-replication should get in conflict again
> because of existing folders on the slave with the same path but a
> different gfid than on the master...like for any other files which
> are deleted and later recreated while geo-rep is in hybrid crawl.
> is that right ?
> if so it will be difficult to reach the changelog modus on large
> gluster volumes because in our case the initial hybrid crawl took
> some days for about 45 TB...or r/w access needs to be stopped for
> that time ?
>
> thanks in advance and
> best regards
> dietmar
>
>
>
>
> Am 19.01.2016 um 10:53 schrieb Milind Changire:
>
> Hi Dietmar,
> After discussion with Aravinda we realized that unfortunately
> the suggestion to:
> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR>
> setfattr -n glusterfs.geo-rep.trigger-sync -v "1"
> <file-path>
>
> won't work with 3.6.7, since provision for that workaround was
> added after 3.6.7.
>
> There's an alternative way to achieve the geo-replication:
> 1. stop geo-replication
> 2. delete files and directories with conflicting gfid on SLAVE
> 3. use the "touch" command to touch files and directories with
> conflicting gfid
> on MASTER
> 4. start geo-replication
>
> This *should* get things correctly replicated to SLAVE.
> Geo-replication should start with hybrid-crawl and trigger the
> replication to SLAVE.
>
> If not, then there's more to look at.
> You could then send us output of the strace command for the
> gsyncd process, while
> geo-replication is running:
> # strace -ff -p <gsyncd-pid> -o gsyncd-strace
>
> You could terminate strace after about one minute and send us
> all the gsyncd-strace.<pid>
> files which will help us debug the issue if its not resolved
> by the alternative
> mechanism mentioned above.
>
> Also, crawl status Hybrid Crawl is not an entirely bad thing.
> It just could mean
> that there's a lot of entries that are being processed.
> However, if things don't
> return back to the normal state after trying out the
> alternative suggestion, we
> could take a look at the strace output and get some clues.
>
> --
> Milind
>
> ----- Original Message -----
> From: "Dietmar Putz" <putz at 3qmedien.net
> <mailto:putz at 3qmedien.net>>
> To: "Aravinda" <avishwan at redhat.com
> <mailto:avishwan at redhat.com>>, gluster-users at gluster.org
> <mailto:gluster-users at gluster.org>, "Milind Changire"
> <mchangir at redhat.com <mailto:mchangir at redhat.com>>
> Sent: Thursday, January 14, 2016 11:07:55 PM
> Subject: Re: [Gluster-users] geo-replication 3.6.7 - no
> transition from hybrid to changelog crawl
>
> Hello all,
>
> after some days of inactivity i started another attempt to
> solve this
> geo-replication issue...step by step.
> it looks like that some of the directories on the slave volume
> does not
> have the same gfid like the corresponding directory on the
> master volume.
>
> for example :
>
> on a master-node i can see a lot of 'errcode: 23' lines like :
> [2016-01-14 09:58:36.96585] W [master(/gluster-export):301:regjob]
> _GMaster: Rsync: .gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50
> [errcode: 23]
>
> on the corresponding slave the corresponding message :
> [2016-01-14 09:57:06.070452] W
> [fuse-bridge.c:1967:fuse_create_cbk]
> 0-glusterfs-fuse: 1185648:
> /.gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50
> => -1 (Operation not permitted)
>
> This is the file on the master, the file is still not
> replicated to the
> slave.
>
> 120533444364 97332 -rw-r--r-- 2 2001 2001 99662854 Jan 8 13:40
> /gluster-export/3912/uploads/BSZ-2015/Z_002895D0-C832-4698-84E6-89F34CDEC2AE_20144555_ST_1.mp4
> 120533444364 97332 -rw-r--r-- 2 2001 2001 99662854 Jan 8 13:40
> /gluster-export/.glusterfs/a8/d0/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50
>
> The directory on the slave already contain some files, all of
> them are
> not available on the master anymore, obviously deleted in
> meantime on
> the master by a client.
> i have deleted and recreated this file on the master and
> observed the
> logs for recurrence of the newly created gfid of this
> file...same as before.
>
> in
> http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20703
> a user reports a geo-replication problem which is possibly
> caused by
> different gfid's of underlying directories.
> and yes, the directory of this file-example above shows that
> the gfid of
> the underlying directory differs from the gfid on the master
> while the
> most other directories have the same gfid.
>
> master :
> ...
> # file: gluster-export/3912/uploads/BSP-2012
> trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350
> --------------
> # file: gluster-export/3912/uploads/BSZ-2003
> trusted.gfid=0xe80adc088e604234b778997d8e8c2018
> --------------
> # file: gluster-export/3912/uploads/BSZ-2004
> trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced
> --------------
> # file: gluster-export/3912/uploads/BSZ-2010
> trusted.gfid=0x8044e436407d4ed3a67c81df8a7ad47f ###
> --------------
> # file: gluster-export/3912/uploads/BSZ-2015
> trusted.gfid=0x0c30f50480204e02b65d4716a048b029 ###
>
> slave :
> ...
> # file: gluster-export/3912/uploads/BSP-2012
> trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350
> --------------
> # file: gluster-export/3912/uploads/BSZ-2003
> trusted.gfid=0xe80adc088e604234b778997d8e8c2018
> --------------
> # file: gluster-export/3912/uploads/BSZ-2004
> trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced
> --------------
> # file: gluster-export/3912/uploads/BSZ-2010
> trusted.gfid=0xd83e8fb568c74e33a2091c547512a6ce ###
> --------------
> # file: gluster-export/3912/uploads/BSZ-2015
> trusted.gfid=0xa406e1bec7f3454d8f2ce9c5f9c70eb3 ###
>
>
> now the question...how to fix this..?
> in the thread above Aravinda wrote :
>
> ...
> To fix the issue,
> -----------------
> Find the parent directory of "main.mdb",
> Get the GFID of that directory, using getfattr
> Check the GFID of the same directory in Slave(To confirm GFIDs
> are different)
> To fix the issue, Delete that directory in Slave.
> Set virtual xattr for that directory and all the files inside
> that directory.
> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR>
> setfattr -n glusterfs.geo-rep.trigger-sync -v "1"
> <file-path>
>
> Geo-rep will recreate the directory with Proper GFID and
> starts sync.
>
> deletion of the affected slave directory might be helpful...
> but do i have to execute above shown setfattr commands on the
> master or
> do they just speed up synchronization ?
> usually sync should start automatically or could there be a
> problem
> because crawl status is still in 'hybrid crawl'...?
>
> thanks in advance...
> best regards
> dietmar
>
>
>
>
> On 04.01.2016 12:08, Dietmar Putz wrote:
>
> Hello Aravinda,
>
> thank you for your reply.
> i just made a 'find /gluster-export -type f -exec ls -lisa
> {} \; >
> ls-lisa-gluster-export-`hostname`.out' on each brick and
> checked the
> output for files with less than 2 link counts.
> i found nothing...all files on each brick have exact 2 links.
>
> the entire output for all bricks contain more than 7
> million lines
> including .glusterfs but without non relevant directories
> and files..
> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* |
> egrep -v
> 'indices|landfill|changelogs|health_check' | wc -l
> 7007316
>
> link count is on $4 :
> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* |
> egrep -v
> 'indices|landfill|changelogs|health_check' | awk
> '{if($4=="2")print}'
> | tail -1
> 62648153697 4 -rw-rw-rw- 2 root root 1713 Jan 4 01:44
> /gluster-export/3500/files/16/01/387233/3500-6dqMmBcVby97PQtR.ism
>
> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* |
> egrep -v
> 'indices|landfill|changelogs|health_check' | awk
> '{if($4=="1")print}'
> tron at dp-server:~/geo_rep_3$
> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* |
> egrep -v
> 'indices|landfill|changelogs|health_check' | awk
> '{if($4!="2")print}'
> tron at dp-server:~/geo_rep_3$
> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* |
> egrep -v
> 'indices|landfill|changelogs|health_check' | awk '{print
> $4}' | sort |
> uniq -c
> 7007316 2
> tron at dp-server:~/geo_rep_3$
>
> If i understood you right this can not be the reason for
> the problem.
> is there any other hint which i can check on the master or
> slave to
> analyse the problem....?
>
> Any help would be very appreciated
> best regards
> dietmar
>
>
>
> Am 04.01.2016 um 07:14 schrieb Aravinda:
>
> Hi,
>
> Looks like issue with Geo-rep due to race between
> Create and Rename.
> Geo-replication uses gfid-access (Mount Volume with
> aux-gfid-mount)
> to create and rename files. If Create and Rename
> replayed more than
> once then Geo-rep creates a two files with same
> GFID(not hardlink).
> This causes one file without backend GFID link.
>
> Milind is working on the patch to disallow the
> creation of second
> file with same GFID.
> @Milind, Please provide more update about your patch.
>
> As a workaround, identify all the files in Slave
> volume which do not
> have backend links and delete those files(Only in
> Slaves, keep backup
> if required)
>
> In Brick backend, Crawl and look for files with link
> count less than
> 2. (Exclude .glusterfs and .trashcan directory)
>
> regards
> Aravinda
>
> On 01/02/2016 09:56 PM, Dietmar Putz wrote:
>
> Hello all,
>
> one more time i need some help with a
> geo-replication problem.
> recently i started a new geo-replication. the
> master volume contains
> about 45 TB data and the slave volume was new
> created before
> geo-replication setup was done.
> master and slave is a 6 node distributed
> replicated volume running
> glusterfs-server 3.6.7-ubuntu1~trusty1.
> geo-rep was starting without problems. since few
> days the slave
> volume contains about 200 GB more data than the
> master volume and i
> expected that the crawl status changes from
> 'hybrid crawl' to
> 'changelog crawl' but it remains in 'hybrid crawl'.
> the 'status detail' output far below shows more
> than 10 million
> synced files while the entire master volume
> contains just about 2
> million files. some tests show that files are not
> deleted on the
> slave volume.
> as far as i know the hybrid crawl has the
> limitation of not
> replicating deletes and renames to the slave thus
> the geo-rep needs
> to achieve the 'changelog crawl' status after
> initial sync...
> usually this should happen more or less
> automatically, is this right ?
>
> the geo-rep frequently fails with below shown
> "OSError: [Errno 16]
> Device or resource busy", this error appears about
> every 3-4 hours
> on each active master node.
> i guess the frequent appearance of this error
> prevent geo-rep from
> changing to 'changelog crawl', does somebody
> experienced such
> problem, is this the cause of the problem ?
>
> i found some similar reports on gluster.org
> <http://gluster.org> for gfs 3.5, 3.6 and 3.7
> but none of them point me to a solution...
> does anybody know a solution or is there a
> workaround to achieve the
> changelog crawl status...?
>
> Any help would be very appreciated
> best regards
> dietmar
>
>
>
>
> Master gluster-ger-ber-07:
> -----------------------------
>
> [2016-01-02 11:39:48.122546] I
> [master(/gluster-export):1343:crawl]
> _GMaster: processing xsync changelog
> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a
> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724692
> [2016-01-02 11:42:55.182342] I
> [master(/gluster-export):1343:crawl]
> _GMaster: processing xsync changelog
> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a
> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724751
> [2016-01-02 11:44:11.168962] I
> [master(/gluster-export):1340:crawl]
> _GMaster: finished hybrid crawl syncing, stime:
> (-1, 0)
> [2016-01-02 11:44:11.246845] I
> [master(/gluster-export):490:crawlwrap] _GMaster:
> primary master
> with volume id
> 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...
> [2016-01-02 11:44:11.265209] I
> [master(/gluster-export):501:crawlwrap] _GMaster:
> crawl interval: 3
> seconds
> [2016-01-02 11:44:11.896940] I
> [master(/gluster-export):1192:crawl]
> _GMaster: slave's time: (-1, 0)
> [2016-01-02 11:44:12.171761] E
> [repce(/gluster-export):207:__call__]
> RepceClient: call
> 18897:139899553576768:1451735052.09 (entry_ops)
> failed on peer with OSError
> [2016-01-02 11:44:12.172101] E
> [syncdutils(/gluster-export):270:log_raise_exception]
> <top>: FAIL:
> Traceback (most recent call last):
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
> line 164, in main
> main_i()
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
> line 643, in main_i
> local.service_loop(*[r for r in [remote] if r])
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
> line 1344, in service_loop
> g2.crawlwrap()
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
> line 539, in crawlwrap
> self.crawl(no_stime_update=no_stime_update)
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
> line 1204, in crawl
> self.process(changes)
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
> line 956, in process
> self.process_change(change, done, retry)
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
> line 920, in process_change
> self.slave.server.entry_ops(entries)
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py",
> line 226, in __call__
> return self.ins(self.meth, *a)
> File
> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py",
> line 208, in __call__
> raise res
> OSError: [Errno 16] Device or resource busy
> [2016-01-02 11:44:12.258982] I
> [syncdutils(/gluster-export):214:finalize] <top>:
> exiting.
> [2016-01-02 11:44:12.321808] I
> [repce(agent):92:service_loop]
> RepceServer: terminating on reaching EOF.
> [2016-01-02 11:44:12.349766] I
> [syncdutils(agent):214:finalize]
> <top>: exiting.
> [2016-01-02 11:44:12.435992] I
> [monitor(monitor):141:set_state]
> Monitor: new state: faulty
> [2016-01-02 11:44:23.164284] I
> [monitor(monitor):215:monitor]
> Monitor:
> ------------------------------------------------------------
> [2016-01-02 11:44:23.169981] I
> [monitor(monitor):216:monitor]
> Monitor: starting gsyncd worker
> [2016-01-02 11:44:23.216662] I
> [changelogagent(agent):72:__init__]
> ChangelogAgent: Agent listining...
> [2016-01-02 11:44:23.239778] I
> [gsyncd(/gluster-export):633:main_i]
> <top>: syncing: gluster://localhost:ger-ber-01 ->
> ssh://root@gluster-wien-07-int:gluster://localhost:aut-wien-vol-01
> [2016-01-02 11:44:26.358613] I
> [master(/gluster-export):75:gmaster_builder]
> <top>: setting up xsync
> change detection mode
> [2016-01-02 11:44:26.358983] I
> [master(/gluster-export):413:__init__] _GMaster:
> using 'rsync' as
> the sync engine
> [2016-01-02 11:44:26.359985] I
> [master(/gluster-export):75:gmaster_builder]
> <top>: setting up
> changelog change detection mode
> [2016-01-02 11:44:26.360243] I
> [master(/gluster-export):413:__init__] _GMaster:
> using 'rsync' as
> the sync engine
> [2016-01-02 11:44:26.361159] I
> [master(/gluster-export):75:gmaster_builder]
> <top>: setting up
> changeloghistory change detection mode
> [2016-01-02 11:44:26.361377] I
> [master(/gluster-export):413:__init__] _GMaster:
> using 'rsync' as
> the sync engine
> [2016-01-02 11:44:26.402601] I
> [master(/gluster-export):1311:register] _GMaster:
> xsync temp
> directory:
> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6e
> 41d8d6e56984/xsync
> [2016-01-02 11:44:26.402848] I
> [resource(/gluster-export):1318:service_loop]
> GLUSTER: Register
> time: 1451735066
> [2016-01-02 11:44:27.26012] I
> [master(/gluster-export):490:crawlwrap] _GMaster:
> primary master
> with volume id
> 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...
> [2016-01-02 11:44:27.31605] I
> [master(/gluster-export):501:crawlwrap] _GMaster:
> crawl interval: 1
> seconds
> [2016-01-02 11:44:27.66868] I
> [master(/gluster-export):1226:crawl]
> _GMaster: starting history crawl... turns: 1,
> stime: (-1, 0)
> [2016-01-02 11:44:27.67043] I
> [master(/gluster-export):1229:crawl]
> _GMaster: stime not available, abandoning history
> crawl
> [2016-01-02 11:44:27.112426] I
> [master(/gluster-export):490:crawlwrap] _GMaster:
> primary master
> with volume id
> 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ...
> [2016-01-02 11:44:27.117506] I
> [master(/gluster-export):501:crawlwrap] _GMaster:
> crawl interval: 60
> seconds
> [2016-01-02 11:44:27.140610] I
> [master(/gluster-export):1333:crawl]
> _GMaster: starting hybrid crawl..., stime: (-1, 0)
> [2016-01-02 11:45:23.417233] I
> [monitor(monitor):141:set_state]
> Monitor: new state: Stable
> [2016-01-02 11:45:48.225915] I
> [master(/gluster-export):1343:crawl]
> _GMaster: processing xsync changelog
> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a
> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735067
> [2016-01-02 11:47:08.65231] I
> [master(/gluster-export):1343:crawl]
> _GMaster: processing xsync changelog
> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6
> e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735148
> ...
>
>
> slave gluster-wien-07 :
> ------------------------
>
> [2016-01-02 11:44:12.007744] W
> [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 1959820: SETXATTR()
> /.gfid/5e436e5b-086b-4720-9e70-0e49c8e09698 => -1
> (File exists)
> [2016-01-02 11:44:12.010970] W
> [client-rpc-fops.c:240:client3_3_mknod_cbk]
> 0-aut-wien-vol-01-client-5: remote operation
> failed: File exists.
> Path:
> <gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm
> [2016-01-02 11:44:12.011327] W
> [client-rpc-fops.c:240:client3_3_mknod_cbk]
> 0-aut-wien-vol-01-client-4: remote operation
> failed: File exists.
> Path:
> <gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm
> [2016-01-02 11:44:12.012054] W
> [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 1959822: SETXATTR()
> /.gfid/666bceac-7c14-4efd-81fe-8185458fcf1f => -1
> (File exists)
> [2016-01-02 11:44:12.024743] W
> [client-rpc-fops.c:240:client3_3_mknod_cbk]
> 0-aut-wien-vol-01-client-5: remote operation
> failed: File exists.
> Path:
> <gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png
> [2016-01-02 11:44:12.024970] W
> [client-rpc-fops.c:240:client3_3_mknod_cbk]
> 0-aut-wien-vol-01-client-4: remote operation
> failed: File exists.
> Path:
> <gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png
> [2016-01-02 11:44:12.025601] W
> [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 1959823: SETXATTR()
> /.gfid/5bfd6f99-07e8-4b2f-844b-aa0b6535c055 => -1
> (File exists)
> [2016-01-02 11:44:12.100688] I
> [dht-selfheal.c:1065:dht_selfheal_layout_new_directory]
> 0-aut-wien-vol-01-dht: chunk size = 0xffffffff /
> 57217563 = 0x4b
> [2016-01-02 11:44:12.100765] I
> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
> 0-aut-wien-vol-01-dht: assigning range size
> 0x5542c4a3 to
> aut-wien-vol-01-replicate-0
> [2016-01-02 11:44:12.100785] I
> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
> 0-aut-wien-vol-01-dht: assigning range size
> 0x5542c4a3 to
> aut-wien-vol-01-replicate-1
> [2016-01-02 11:44:12.100800] I
> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
> 0-aut-wien-vol-01-dht: assigning range size
> 0x5542c4a3 to
> aut-wien-vol-01-replicate-2
> [2016-01-02 11:44:12.100839] I [MSGID: 109036]
> [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]
> 0-aut-wien-vol-01-dht: Setting layout of
> <gfid:d4815ee4-3348-4105-9136-d0219d956ed8>/.dstXXX0HUpRD
> with
> [Subvol_name: aut-wien-vol-01-re
> plicate-0, Err: -1 , Start: 0 , Stop: 1430439074
> ], [Subvol_name:
> aut-wien-vol-01-replicate-1, Err: -1 , Start:
> 1430439075 , Stop:
> 2860878149 ], [Subvol_name:
> aut-wien-vol-01-replicate-2, Err: -1 ,
> Start: 2860878150 , Stop: 4294967295 ],
> [2016-01-02 11:44:12.114192] W
> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
> 0-aut-wien-vol-01-client-2: remote operation
> failed: File exists.
> Path:
> <gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x
> [2016-01-02 11:44:12.114275] W
> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
> 0-aut-wien-vol-01-client-3: remote operation
> failed: File exists.
> Path:
> <gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x
> [2016-01-02 11:44:12.114879] W
> [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 1959831: SETXATTR()
> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2 => -1
> (File exists)
> [2016-01-02 11:44:12.118473] I
> [dht-layout.c:663:dht_layout_normalize]
> 0-aut-wien-vol-01-dht: Found
> anomalies in
> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x
> (gfid =
> 00000000-0000-0000-0000-000000000000). Holes=1
> overlaps=0
> [2016-01-02 11:44:12.118537] I
> [dht-selfheal.c:1065:dht_selfheal_layout_new_directory]
> 0-aut-wien-vol-01-dht: chunk size = 0xffffffff /
> 57217563 = 0x4b
> [2016-01-02 11:44:12.118562] I
> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
> 0-aut-wien-vol-01-dht: assigning range size
> 0x5542c4a3 to
> aut-wien-vol-01-replicate-2
> [2016-01-02 11:44:12.118579] I
> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
> 0-aut-wien-vol-01-dht: assigning range size
> 0x5542c4a3 to
> aut-wien-vol-01-replicate-0
> [2016-01-02 11:44:12.118613] I
> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
> 0-aut-wien-vol-01-dht: assigning range size
> 0x5542c4a3 to
> aut-wien-vol-01-replicate-1
> [2016-01-02 11:44:12.120352] I [MSGID: 109036]
> [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]
> 0-aut-wien-vol-01-dht: Setting layout of
> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x
> with
> [Subvol_name: aut-wien-vol-01-rep
> licate-0, Err: -1 , Start: 1430439075 , Stop:
> 2860878149 ],
> [Subvol_name: aut-wien-vol-01-replicate-1, Err: -1
> , Start:
> 2860878150 , Stop: 4294967295 ], [Subvol_name:
> aut-wien-vol-01-replicate-2, Err: -1 , Start: 0 ,
> Stop: 1430439074 ],
> [2016-01-02 11:44:12.630949] I
> [fuse-bridge.c:4927:fuse_thread_proc]
> 0-fuse: unmounting /tmp/gsyncd-aux-mount-tOUOsz
> [2016-01-02 11:44:12.633952] W
> [glusterfsd.c:1211:cleanup_and_exit]
> (--> 0-: received signum (15), shutting down
> [2016-01-02 11:44:12.633964] I
> [fuse-bridge.c:5607:fini] 0-fuse:
> Unmounting '/tmp/gsyncd-aux-mount-tOUOsz'.
> [2016-01-02 11:44:23.946702] I [MSGID: 100030]
> [glusterfsd.c:2035:main] 0-/usr/sbin/glusterfs:
> Started running
> /usr/sbin/glusterfs version 3.6.7 (args:
> /usr/sbin/glusterfs
> --aux-gfid-mount
> --log-file=/var/log/glusterfs/geo-replication-slav
> es/6a071cfa-b150-4f0b-b1ed-96ab5d4bd671:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log
> --volfile-server=localhost
> --volfile-id=aut-wien-vol-01
> --client-pid=-1 /tmp/gsyncd-aux-mount-otU3wS)
> [2016-01-02 11:44:24.042128] I
> [dht-shared.c:337:dht_init_regex]
> 0-aut-wien-vol-01-dht: using regex
> rsync-hash-regex = ^\.(.+)\.[^.]+$
> [2016-01-02 11:44:24.046315] I [client.c:2268:notify]
> 0-aut-wien-vol-01-client-0: parent translators are
> ready, attempting
> connect on transport
> [2016-01-02 11:44:24.046532] I [client.c:2268:notify]
> 0-aut-wien-vol-01-client-1: parent translators are
> ready, attempting
> connect on transport
> [2016-01-02 11:44:24.046664] I [client.c:2268:notify]
> 0-aut-wien-vol-01-client-2: parent translators are
> ready, attempting
> connect on transport
> [2016-01-02 11:44:24.046806] I [client.c:2268:notify]
> 0-aut-wien-vol-01-client-3: parent translators are
> ready, attempting
> connect on transport
> [2016-01-02 11:44:24.046940] I [client.c:2268:notify]
> 0-aut-wien-vol-01-client-4: parent translators are
> ready, attempting
> connect on transport
> [2016-01-02 11:44:24.047070] I [client.c:2268:notify]
> 0-aut-wien-vol-01-client-5: parent translators are
> ready, attempting
> connect on transport
> Final graph:
> +------------------------------------------------------------------------------+
>
> 1: volume aut-wien-vol-01-client-0
> 2: type protocol/client
> 3: option ping-timeout 10
> 4: option remote-host gluster-wien-02-int
> 5: option remote-subvolume /gluster-export
> 6: option transport-type socket
> 7: option username
> 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
> 8: option password
> 8777e154-476c-449a-89b2-3199872e4a1f
> 9: option send-gids true
> 10: end-volume
> 11:
> 12: volume aut-wien-vol-01-client-1
> 13: type protocol/client
> 14: option ping-timeout 10
> 15: option remote-host gluster-wien-03-int
> 16: option remote-subvolume /gluster-export
> 17: option transport-type socket
> 18: option username
> 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
> 19: option password
> 8777e154-476c-449a-89b2-3199872e4a1f
> 20: option send-gids true
> 21: end-volume
> 22:
> 23: volume aut-wien-vol-01-replicate-0
> 24: type cluster/replicate
> 25: subvolumes aut-wien-vol-01-client-0
> aut-wien-vol-01-client-1
> 26: end-volume
> 27:
> 28: volume aut-wien-vol-01-client-2
> 29: type protocol/client
> 30: option ping-timeout 10
> 31: option remote-host gluster-wien-04-int
> 32: option remote-subvolume /gluster-export
> 33: option transport-type socket
> 34: option username
> 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
> 35: option password
> 8777e154-476c-449a-89b2-3199872e4a1f
> 36: option send-gids true
> 37: end-volume
> 38:
> 39: volume aut-wien-vol-01-client-3
> 40: type protocol/client
> 41: option ping-timeout 10
> 42: option remote-host gluster-wien-05-int
> 43: option remote-subvolume /gluster-export
> 44: option transport-type socket
> 45: option username
> 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
> 46: option password
> 8777e154-476c-449a-89b2-3199872e4a1f
> 47: option send-gids true
> 48: end-volume
> 49:
> 50: volume aut-wien-vol-01-replicate-1
> 51: type cluster/replicate
> 52: subvolumes aut-wien-vol-01-client-2
> aut-wien-vol-01-client-3
> 53: end-volume
> 54:
> 55: volume aut-wien-vol-01-client-4
> 56: type protocol/client
> 57: option ping-timeout 10
> 58: option remote-host gluster-wien-06-int
> 59: option remote-subvolume /gluster-export
> 60: option transport-type socket
> 61: option username
> 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
> 62: option password
> 8777e154-476c-449a-89b2-3199872e4a1f
> 63: option send-gids true
> 64: end-volume
> 65:
> 66: volume aut-wien-vol-01-client-5
> 67: type protocol/client
> 68: option ping-timeout 10
> 69: option remote-host gluster-wien-07-int
> 70: option remote-subvolume /gluster-export
> 71: option transport-type socket
> 72: option username
> 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929
> 73: option password
> 8777e154-476c-449a-89b2-3199872e4a1f
> 74: option send-gids true
> 75: end-volume
> 76:
> 77: volume aut-wien-vol-01-replicate-2
> 78: type cluster/replicate
> 79: subvolumes aut-wien-vol-01-client-4
> aut-wien-vol-01-client-5
> 80: end-volume
> 81:
> 82: volume aut-wien-vol-01-dht
> 83: type cluster/distribute
> 84: subvolumes aut-wien-vol-01-replicate-0
> aut-wien-vol-01-replicate-1
> aut-wien-vol-01-replicate-2
> 85: end-volume
> 86:
> 87: volume aut-wien-vol-01-write-behind
> 88: type performance/write-behind
> 89: subvolumes aut-wien-vol-01-dht
> 90: end-volume
> 91:
> 92: volume aut-wien-vol-01-read-ahead
> 93: type performance/read-ahead
> 94: subvolumes aut-wien-vol-01-write-behind
> 95: end-volume
> 96:
> 97: volume aut-wien-vol-01-io-cache
> 98: type performance/io-cache
> 99: option min-file-size 0
> 100: option cache-timeout 2
> 101: option cache-size 1024MB
> 102: subvolumes aut-wien-vol-01-read-ahead
> 103: end-volume
> 104:
> 105: volume aut-wien-vol-01-quick-read
> 106: type performance/quick-read
> 107: option cache-size 1024MB
> 108: subvolumes aut-wien-vol-01-io-cache
> 109: end-volume
> 110:
> 111: volume aut-wien-vol-01-open-behind
> 112: type performance/open-behind
> 113: subvolumes aut-wien-vol-01-quick-read
> 114: end-volume
> 115:
> 116: volume aut-wien-vol-01-md-cache
> 117: type performance/md-cache
> 118: subvolumes aut-wien-vol-01-open-behind
> 119: end-volume
> 120:
> 121: volume aut-wien-vol-01
> 122: type debug/io-stats
> 123: option latency-measurement off
> 124: option count-fop-hits off
> 125: subvolumes aut-wien-vol-01-md-cache
> 126: end-volume
> 127:
> 128: volume gfid-access-autoload
> 129: type features/gfid-access
> 130: subvolumes aut-wien-vol-01
> 131: end-volume
> 132:
> 133: volume meta-autoload
> 134: type meta
> 135: subvolumes gfid-access-autoload
> 136: end-volume
> 137:
> +------------------------------------------------------------------------------+
>
> [2016-01-02 11:44:24.047642] I
> [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-aut-wien-vol-01-client-5: changing port to 49153
> (from 0)
> [2016-01-02 11:44:24.047927] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-aut-wien-vol-01-client-5: Using Program
> GlusterFS 3.3, Num
> (1298437), Version (330)
> [2016-01-02 11:44:24.048044] I
> [client-handshake.c:1200:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-5: Connected to
> aut-wien-vol-01-client-5,
> attached to remote volume '/gluster-export'.
> [2016-01-02 11:44:24.048050] I
> [client-handshake.c:1210:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-5: Server and Client
> lk-version numbers are
> not same, reopening the fds
> [2016-01-02 11:44:24.048088] I [MSGID: 108005]
> [afr-common.c:3684:afr_notify]
> 0-aut-wien-vol-01-replicate-2:
> Subvolume 'aut-wien-vol-01-client-5' came back up;
> going online.
> [2016-01-02 11:44:24.048114] I
> [client-handshake.c:188:client_set_lk_version_cbk]
> 0-aut-wien-vol-01-client-5: Server lk version = 1
> [2016-01-02 11:44:24.048124] I
> [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-aut-wien-vol-01-client-0: changing port to 49153
> (from 0)
> [2016-01-02 11:44:24.048132] I
> [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-aut-wien-vol-01-client-1: changing port to 49153
> (from 0)
> [2016-01-02 11:44:24.048138] I
> [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-aut-wien-vol-01-client-2: changing port to 49153
> (from 0)
> [2016-01-02 11:44:24.048146] I
> [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-aut-wien-vol-01-client-3: changing port to 49153
> (from 0)
> [2016-01-02 11:44:24.048153] I
> [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-aut-wien-vol-01-client-4: changing port to 49153
> (from 0)
> [2016-01-02 11:44:24.049070] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-aut-wien-vol-01-client-0: Using Program
> GlusterFS 3.3, Num
> (1298437), Version (330)
> [2016-01-02 11:44:24.049094] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-aut-wien-vol-01-client-3: Using Program
> GlusterFS 3.3, Num
> (1298437), Version (330)
> [2016-01-02 11:44:24.049113] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-aut-wien-vol-01-client-2: Using Program
> GlusterFS 3.3, Num
> (1298437), Version (330)
> [2016-01-02 11:44:24.049131] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-aut-wien-vol-01-client-1: Using Program
> GlusterFS 3.3, Num
> (1298437), Version (330)
> [2016-01-02 11:44:24.049224] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-aut-wien-vol-01-client-4: Using Program
> GlusterFS 3.3, Num
> (1298437), Version (330)
> [2016-01-02 11:44:24.049307] I
> [client-handshake.c:1200:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-0: Connected to
> aut-wien-vol-01-client-0,
> attached to remote volume '/gluster-export'.
> [2016-01-02 11:44:24.049312] I
> [client-handshake.c:1210:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-0: Server and Client
> lk-version numbers are
> not same, reopening the fds
> [2016-01-02 11:44:24.049324] I [MSGID: 108005]
> [afr-common.c:3684:afr_notify]
> 0-aut-wien-vol-01-replicate-0:
> Subvolume 'aut-wien-vol-01-client-0' came back up;
> going online.
> [2016-01-02 11:44:24.049384] I
> [client-handshake.c:1200:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-3: Connected to
> aut-wien-vol-01-client-3,
> attached to remote volume '/gluster-export'.
> [2016-01-02 11:44:24.049389] I
> [client-handshake.c:1210:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-3: Server and Client
> lk-version numbers are
> not same, reopening the fds
> [2016-01-02 11:44:24.049400] I [MSGID: 108005]
> [afr-common.c:3684:afr_notify]
> 0-aut-wien-vol-01-replicate-1:
> Subvolume 'aut-wien-vol-01-client-3' came back up;
> going online.
> [2016-01-02 11:44:24.049418] I
> [client-handshake.c:1200:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-2: Connected to
> aut-wien-vol-01-client-2,
> attached to remote volume '/gluster-export'.
> [2016-01-02 11:44:24.049422] I
> [client-handshake.c:1210:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-2: Server and Client
> lk-version numbers are
> not same, reopening the fds
> [2016-01-02 11:44:24.049460] I
> [client-handshake.c:1200:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-1: Connected to
> aut-wien-vol-01-client-1,
> attached to remote volume '/gluster-export'.
> [2016-01-02 11:44:24.049465] I
> [client-handshake.c:1210:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-1: Server and Client
> lk-version numbers are
> not same, reopening the fds
> [2016-01-02 11:44:24.049493] I
> [client-handshake.c:188:client_set_lk_version_cbk]
> 0-aut-wien-vol-01-client-0: Server lk version = 1
> [2016-01-02 11:44:24.049567] I
> [client-handshake.c:188:client_set_lk_version_cbk]
> 0-aut-wien-vol-01-client-3: Server lk version = 1
> [2016-01-02 11:44:24.049632] I
> [client-handshake.c:1200:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-4: Connected to
> aut-wien-vol-01-client-4,
> attached to remote volume '/gluster-export'.
> [2016-01-02 11:44:24.049638] I
> [client-handshake.c:1210:client_setvolume_cbk]
> 0-aut-wien-vol-01-client-4: Server and Client
> lk-version numbers are
> not same, reopening the fds
> [2016-01-02 11:44:24.052103] I
> [fuse-bridge.c:5086:fuse_graph_setup]
> 0-fuse: switched to graph 0
> [2016-01-02 11:44:24.052150] I
> [client-handshake.c:188:client_set_lk_version_cbk]
> 0-aut-wien-vol-01-client-2: Server lk version = 1
> [2016-01-02 11:44:24.052163] I
> [client-handshake.c:188:client_set_lk_version_cbk]
> 0-aut-wien-vol-01-client-4: Server lk version = 1
> [2016-01-02 11:44:24.052192] I
> [client-handshake.c:188:client_set_lk_version_cbk]
> 0-aut-wien-vol-01-client-1: Server lk version = 1
> [2016-01-02 11:44:24.052204] I
> [fuse-bridge.c:4015:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol
> versions: glusterfs 7.22
> kernel 7.20
> [2016-01-02 11:44:24.053991] I
> [afr-common.c:1491:afr_local_discovery_cbk]
> 0-aut-wien-vol-01-replicate-2: selecting local
> read_child
> aut-wien-vol-01-client-5
> [2016-01-02 11:45:48.613563] W
> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
> 0-aut-wien-vol-01-client-5: remote operation
> failed: File exists.
> Path: /keys
> [2016-01-02 11:45:48.614131] W
> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
> 0-aut-wien-vol-01-client-4: remote operation
> failed: File exists.
> Path: /keys
> [2016-01-02 11:45:48.614436] W
> [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 12: SETXATTR()
> /.gfid/00000000-0000-0000-0000-000000000001 => -1
> (File exists)
> ...
>
>
> [ 13:41:40 ] - root at gluster-ger-ber-07
> /var/log/glusterfs/geo-replication/ger-ber-01
> $gluster volume
> geo-replication ger-ber-01
> gluster-wien-07::aut-wien-vol-01 status
> detail
>
> MASTER NODE MASTER VOL MASTER BRICK
> SLAVE STATUS
> CHECKPOINT
> STATUS CRAWL STATUS FILES SYNCD FILES
> PENDING BYTES
> PENDING DELETES PENDING FILES SKIPPED
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> gluster-ger-ber-07 ger-ber-01 /gluster-export
> gluster-wien-07-int::aut-wien-vol-01 Active N/A
> Hybrid Crawl 10743644 8192 0 0 0
> gluster-ger-ber-11 ger-ber-01 /gluster-export
> gluster-wien-03-int::aut-wien-vol-01 Active N/A
> Hybrid Crawl 16037091 8192 0 0 0
> gluster-ger-ber-10 ger-ber-01 /gluster-export
> gluster-wien-02-int::aut-wien-vol-01 Passive N/A
> N/A 0 0 0 0 0
> gluster-ger-ber-12 ger-ber-01 /gluster-export
> gluster-wien-06-int::aut-wien-vol-01 Passive N/A
> N/A 0 0 0 0 0
> gluster-ger-ber-09 ger-ber-01 /gluster-export
> gluster-wien-05-int::aut-wien-vol-01 Active N/A
> Hybrid Crawl 16180514 8192 0 0 0
> gluster-ger-ber-08 ger-ber-01 /gluster-export
> gluster-wien-04-int::aut-wien-vol-01 Passive N/A
> N/A 0 0 0 0 0
>
>
> [ 13:41:55 ] - root at gluster-ger-ber-07
> /var/log/glusterfs/geo-replication/ger-ber-01
> $gluster volume
> geo-replication ger-ber-01
> gluster-wien-07::aut-wien-vol-01 config
> special_sync_mode: partial
> state_socket_unencoded:
> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.socket
> gluster_log_file:
> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log
> ssh_command: ssh -p 2503 -oPasswordAuthentication=no
> -oStrictHostKeyChecking=no -i
> /var/lib/glusterd/geo-replication/secret.pem
> ignore_deletes: true
> change_detector: changelog
> ssh_command_tar: ssh -p 2503
> -oPasswordAuthentication=no
> -oStrictHostKeyChecking=no -i
> /var/lib/glusterd/geo-replication/tar_ssh.pem
> state_file:
> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.status
> remote_gsyncd: /nonexistent/gsyncd
> log_file:
> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.log
> changelog_log_file:
> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-changes.log
> socketdir: /var/run
> working_dir:
> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01
> state_detail_file:
> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-detail.status
> session_owner: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
> gluster_command_dir: /usr/sbin/
> pid_file:
> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.pid
> georep_session_working_dir:
> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/
> gluster_params: aux-gfid-mount
> volume_id: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
> [ 13:42:11 ] - root at gluster-ger-ber-07
> /var/log/glusterfs/geo-replication/ger-ber-01 $
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> <mailto:Gluster-users at gluster.org>
> http://www.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160120/d39db7c6/attachment.html>
More information about the Gluster-users
mailing list