<div dir="ltr">Thanks for your response (6 months ago!) but I have only just got around to following up on this.<div><br></div><div>Unfortunately, I had already copied and shipped the data to the second datacenter before copying the GFIDs so I already stumbled before the first hurdle!</div><div><br></div><div>I have been using the scripts in the extras/geo-rep provided for an earlier version upgrade. With a bit of tinkering, these have given me a file contianing the gfid/file pairs needed to sync to the slave before enabling georeplication.</div><div><br></div><div>Unfortunately, the gsync-sync-gfid program isn't working. On all the files it reports that they failed and I see the following in the fuse log:</div><div><pre id="gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)">[2017-12-21 16:36:37.171846] D [MSGID: 0] [dht-common.c:997:dht_revalidate_cbk] 0-video-backup-dht: revalidate lookup of /path returned with op_ret 0 [Invalid argument]
[2017-12-21 16:36:37.172352] D [fuse-helpers.c:650:fuse_ignore_xattr_set] 0-glusterfs-fuse: allowing setxattr: key [glusterfs.gfid.heal], client pid [0]
[2017-12-21 16:36:37.172457] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [dht-common.c:997:dht_revalidate_cbk] 0-video-backup-dht: revalidate lookup of /path returned with op_ret 0 [Invalid argument]" repeated 8 times between [2017-12-21 16:36:37.171846] and [2017-12-21 16:36:37.172225]
[2017-12-21 16:36:37.172457] D [MSGID: 0] [dht-common.c:2692:dht_lookup] 0-video-backup-dht: Calling fresh lookup for /path/file.txt on video-backup-client-4
[2017-12-21 16:36:37.173166] D [MSGID: 0] [client-rpc-fops.c:2910:client3_3_lookup_cbk] 0-video-backup-client-4: gfid changed for /path/file.txt
[2017-12-21 16:36:37.173212] D [MSGID: 0] [client-rpc-fops.c:2941:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-client-4 returned -1 error: Stale file handle [Stale file handle]
[2017-12-21 16:36:37.173233] D [MSGID: 0] [dht-common.c:2279:dht_lookup_cbk] 0-video-backup-dht: fresh_lookup returned for /path/file.txt with op_ret -1 [Stale file handle]
[2017-12-21 16:36:37.173250] D [MSGID: 0] [dht-common.c:2359:dht_lookup_cbk] 0-video-backup-dht: Lookup of /path/file.txt for subvolume video-backup-client-4 failed [Stale file handle]
[2017-12-21 16:36:37.173267] D [MSGID: 0] [dht-common.c:2422:dht_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-dht returned -1 error: Stale file handle [Stale file handle]
[2017-12-21 16:36:37.173285] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-io-cache returned -1 error: Stale file handle [Stale file handle]
[2017-12-21 16:36:37.173303] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-quick-read returned -1 error: Stale file handle [Stale file handle]
[2017-12-21 16:36:37.173320] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-md-cache returned -1 error: Stale file handle [Stale file handle]
[2017-12-21 16:36:37.173336] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup returned -1 error: Stale file handle [Stale file handle]
[2017-12-21 16:36:37.173374] D [MSGID: 0] [gfid-access.c:390:ga_heal_cbk] 0-stack-trace: stack-address: 0x7fe39ebd7498, gfid-access-autoload returned -1 error: Stale file handle [Stale file handle]
[2017-12-21 16:36:37.173405] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 57862: SETXATTR() /path => -1 (Stale file handle)</pre><pre id="gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)"><font face="arial, helvetica, sans-serif">I notice in slave-upgrade.sh, the .glusterfs contents on each brick are deleted, and the volume restarted before </font>gsync-sync-gfid <font face="arial, helvetica, sans-serif">is run.</font></pre><pre id="gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)"><font face="arial, helvetica, sans-serif">I have a good working backup at the moment and deleting the .glusterfs folder worries me. Is this the solution, or is something else wrong?</font></pre><pre id="gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)"><br></pre><pre id="gmail-hterm:copy-to-clipboard-source" style="color:rgb(0,0,0)"><br></pre></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 27 June 2017 at 06:31, Aravinda <span dir="ltr"><<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Answers inline, <br>
<br>
@Kotresh, please add if I missed anything.<br>
<pre class="m_-3569571506311279768moz-signature" cols="72">regards
Aravinda VK
<a class="m_-3569571506311279768moz-txt-link-freetext" href="http://aravindavk.in" target="_blank">http://aravindavk.in</a>
</pre><span class="">
<div class="m_-3569571506311279768moz-cite-prefix">On 06/23/2017 06:29 PM, Stephen Remde
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>I have a ~600tb distributed gluster volume that I want to
start using geo replication on.</div>
<div><br>
</div>
<div>The current volume is on 6 100tb bricks on 2 servers </div>
<div><br>
</div>
<div>My plan is:</div>
<div><br>
</div>
<div>1) copy each of the bricks to a new arrays on the servers
locally</div>
</div>
</blockquote></span>
Before start copying, <br>
- Enable changelog in Master volume, using `gluster volume set
<volname> changelog.changelog on`<br>
- Record the Timestamp. This is required to set the marking once
Geo-rep session is created.<br>
- Gluster Geo-replication replicates the data with same GFID in
target, so if we are copying data manually we should ensure that
GFIDs remains same and properly linked as hardlinks in the
.glusterfs directory.
<span class=""><blockquote type="cite">
<div dir="ltr">
<div>2) move the new arrays to the new servers</div>
<div>3) create the volume on the new servers using the arrays </div>
<div>4) fix the layout on the new volume</div>
</div>
</blockquote>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>5) start georeplication (which should be relatively small
as most of the data should be already there?)</div>
</div>
</blockquote></span>
After creating the Geo-replication session it will try to start from
the beginning, to avoid that set the marking in each bricks before
starting Geo-replication saying till this time data is synced to
Slave. Once started, it will start from this timestamp.(I can
provide script for the same)
<blockquote type="cite"><span class="">
<div dir="ltr">
<div><br>
</div>
<div>Is this likely to succeed?</div>
<div><br>
</div>
Any advice welcomed.<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_-3569571506311279768gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<p><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:rgb(64,64,64)">Dr
Stephen Remde<br>
Director, Innovation and Research</span></p>
<p><img src="cid:part1.112DFA37.F4904B03@redhat.com" style="font-family:Arial,sans-serif;font-size:12pt" height="51" border="0" width="94"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:rgb(64,64,64)"><br>
</span></p>
<p><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(64,64,64)">T:
<a href="tel:01535%20280066" value="+441535280066" target="_blank">01535 280066</a><br>
</span><span style="color:rgb(64,64,64);font-family:Arial,sans-serif;font-size:10pt">M:
<a href="tel:07764%20740920" value="+447764740920" target="_blank">07764 740920</a><br>
</span><span style="color:rgb(64,64,64);font-family:Arial,sans-serif;font-size:10pt">E: </span><a href="mailto:stephen.remde@gaist.co.uk" style="font-family:Arial,sans-serif;font-size:10pt;color:rgb(17,85,204)" target="_blank">stephen.remde@gaist.co.uk</a><u><br>
</u><span style="color:rgb(64,64,64);font-family:Arial,sans-serif;font-size:10pt">W: </span><a href="http://www.gaist.co.uk/" style="font-family:Arial,sans-serif;font-size:10pt;color:rgb(17,85,204)" target="_blank">www.gaist.co.uk</a></p>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_-3569571506311279768mimeAttachmentHeader"></fieldset>
<br>
</span><pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_-3569571506311279768moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_-3569571506311279768moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><p><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:rgb(64,64,64)">Dr Stephen Remde<br>Director, Innovation and Research</span></p><p><img border="0" width="94" height="51" src="cid:image001.jpg@01CFC3AC.A7A9AD60" style="font-family:Arial,sans-serif;font-size:12pt"><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:rgb(64,64,64)"><br></span></p><p><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(64,64,64)">T: 01535 280066<br></span><span style="color:rgb(64,64,64);font-family:Arial,sans-serif;font-size:10pt">M: 07764 740920<br></span><span style="color:rgb(64,64,64);font-family:Arial,sans-serif;font-size:10pt">E: </span><a href="mailto:stephen.remde@gaist.co.uk" style="font-family:Arial,sans-serif;font-size:10pt;color:rgb(17,85,204)" target="_blank">stephen.remde@gaist.co.uk</a><u><br></u><span style="color:rgb(64,64,64);font-family:Arial,sans-serif;font-size:10pt">W: </span><a href="http://www.gaist.co.uk/" style="font-family:Arial,sans-serif;font-size:10pt;color:rgb(17,85,204)" target="_blank">www.gaist.co.uk</a></p></div></div></div></div>
</div>