[Gluster-users] seeding my georeplication

Tue Jan 2 11:20:48 UTC 2018

Additionally, setting the xattr via the command line does not seem to help

getfattr -n glusterfs.gfid.string ./workspace.json
# file: workspace.json
glusterfs.gfid.string="5d624925-be90-422d-b5a2-49a90e5fb735"

sudo setfattr -n glusterfs.gfid.heal -v
"8e122fcb-7572-403a-9737-d4f05ec2cb4e\0workspace.json\0" ./

getfattr -n glusterfs.gfid.string ./workspace.json
# file: workspace.json
glusterfs.gfid.string="5d624925-be90-422d-b5a2-49a90e5fb735"

Paste bin of mount log:

https://pastebin.com/z6pg9dhf

On 21 December 2017 at 18:05, Stephen Remde <stephen.remde at gaist.co.uk>
wrote:

> Thanks for your response (6 months ago!) but I have only just got around
> to following up on this.
>
> Unfortunately, I had already copied and shipped the data to the second
> datacenter before copying the GFIDs so I already stumbled before the first
> hurdle!
>
> I have been using the scripts in the extras/geo-rep provided for an
> earlier version upgrade. With a bit of tinkering, these have given me a
> file contianing the gfid/file pairs needed to sync to the slave before
> enabling georeplication.
>
> Unfortunately, the gsync-sync-gfid program isn't working. On all the files
> it reports that they failed and I see the following in the fuse log:
>
> [2017-12-21 16:36:37.171846] D [MSGID: 0] [dht-common.c:997:dht_revalidate_cbk] 0-video-backup-dht: revalidate lookup of /path returned with op_ret 0 [Invalid argument]
> [2017-12-21 16:36:37.172352] D [fuse-helpers.c:650:fuse_ignore_xattr_set] 0-glusterfs-fuse: allowing setxattr: key [glusterfs.gfid.heal],  client pid [0]
> [2017-12-21 16:36:37.172457] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
> The message "D [MSGID: 0] [dht-common.c:997:dht_revalidate_cbk] 0-video-backup-dht: revalidate lookup of /path returned with op_ret 0 [Invalid argument]" repeated 8 times between [2017-12-21 16:36:37.171846] and [2017-12-21 16:36:37.172225]
> [2017-12-21 16:36:37.172457] D [MSGID: 0] [dht-common.c:2692:dht_lookup] 0-video-backup-dht: Calling fresh lookup for /path/file.txt on video-backup-client-4
> [2017-12-21 16:36:37.173166] D [MSGID: 0] [client-rpc-fops.c:2910:client3_3_lookup_cbk] 0-video-backup-client-4: gfid changed for /path/file.txt
> [2017-12-21 16:36:37.173212] D [MSGID: 0] [client-rpc-fops.c:2941:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-client-4 returned -1 error: Stale file handle [Stale file handle]
> [2017-12-21 16:36:37.173233] D [MSGID: 0] [dht-common.c:2279:dht_lookup_cbk] 0-video-backup-dht: fresh_lookup returned for /path/file.txt with op_ret -1 [Stale file handle]
> [2017-12-21 16:36:37.173250] D [MSGID: 0] [dht-common.c:2359:dht_lookup_cbk] 0-video-backup-dht: Lookup of /path/file.txt for subvolume video-backup-client-4 failed [Stale file handle]
> [2017-12-21 16:36:37.173267] D [MSGID: 0] [dht-common.c:2422:dht_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-dht returned -1 error: Stale file handle [Stale file handle]
> [2017-12-21 16:36:37.173285] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-io-cache returned -1 error: Stale file handle [Stale file handle]
> [2017-12-21 16:36:37.173303] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-quick-read returned -1 error: Stale file handle [Stale file handle]
> [2017-12-21 16:36:37.173320] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-md-cache returned -1 error: Stale file handle [Stale file handle]
> [2017-12-21 16:36:37.173336] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup returned -1 error: Stale file handle [Stale file handle]
> [2017-12-21 16:36:37.173374] D [MSGID: 0] [gfid-access.c:390:ga_heal_cbk] 0-stack-trace: stack-address: 0x7fe39ebd7498, gfid-access-autoload returned -1 error: Stale file handle [Stale file handle]
> [2017-12-21 16:36:37.173405] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 57862: SETXATTR() /path => -1 (Stale file handle)
>
> I notice in slave-upgrade.sh, the .glusterfs contents on each brick are deleted, and the volume restarted before gsync-sync-gfid is run.
>
> I have a good working backup at the moment and deleting the .glusterfs folder worries me. Is this the solution, or is something else wrong?
>
>
>
>
>
>
>
>
> On 27 June 2017 at 06:31, Aravinda <avishwan at redhat.com> wrote:
>
>> Answers inline,
>>
>> @Kotresh, please add if I missed anything.
>>
>> regards
>> Aravinda VKhttp://aravindavk.in
>>
>> On 06/23/2017 06:29 PM, Stephen Remde wrote:
>>
>> I have a ~600tb distributed gluster volume that I want to start using geo
>> replication on.
>>
>> The current volume is on 6 100tb bricks on 2 servers
>>
>> My plan is:
>>
>> 1) copy each of the bricks to a new arrays on the servers locally
>>
>> Before start copying,
>> - Enable changelog in Master volume, using `gluster volume set <volname>
>> changelog.changelog on`
>> - Record the Timestamp. This is required to set the marking once Geo-rep
>> session is created.
>> - Gluster Geo-replication replicates the data with same GFID in target,
>> so if we are copying data manually we should ensure that GFIDs remains same
>> and properly linked as hardlinks in the .glusterfs directory.
>>
>> 2) move the new arrays to the new servers
>> 3) create the volume on the new servers using the arrays
>> 4) fix the layout on the new volume
>>
>>
>> 5) start georeplication (which should be relatively small as most of the
>> data should be already there?)
>>
>> After creating the Geo-replication session it will try to start from the
>> beginning, to avoid that set the marking in each bricks before starting
>> Geo-replication saying till this time data is synced to Slave. Once
>> started, it will start from this timestamp.(I can provide script for the
>> same)
>>
>>
>> Is this likely to succeed?
>>
>> Any advice welcomed.
>>
>> --
>>
>> Dr Stephen Remde
>> Director, Innovation and Research
>>
>>
>> T: 01535 280066
>> M: 07764 740920
>> E: stephen.remde at gaist.co.uk
>> W: www.gaist.co.uk
>>
>>
>> _______________________________________________
>> Gluster-users mailing listGluster-users at gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>
>
> --
>
> Dr Stephen Remde
> Director, Innovation and Research
>
>
> T: 01535 280066
> M: 07764 740920
> E: stephen.remde at gaist.co.uk
> W: www.gaist.co.uk
>

-- 

Dr Stephen Remde
Director, Innovation and Research

T: 01535 280066
M: 07764 740920
E: stephen.remde at gaist.co.uk
W: www.gaist.co.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180102/fda9ed64/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1734 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180102/fda9ed64/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1734 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180102/fda9ed64/attachment-0001.jpg>