<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 30, 2018 at 3:51 PM, Krishna Verma <span dir="ltr"><<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div class="m_98662732782548348WordSection1">
<p class="MsoNormal">Hi Kotresh,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Yes, this include the time take to write 1GB file to master. geo-rep was not stopped while the data was copying to master.
</p></div></div></blockquote><div><br></div><div>This way, you can't really measure how much time geo-rep took.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="blue" vlink="purple" lang="EN-US"><div class="m_98662732782548348WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">But now I am trouble, My putty session was timed out while copying data to master and geo replication was active. After I restart putty session My Master data is not syncing with slave. Its Last_synced time is 1hrs behind the current time.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I restart the geo rep and also delete and again create the session but its “LAST_SYNCED” time is same.
</p></div></div></blockquote><div><br></div><div>Unless, geo-rep is Faulty, it would be processing/syncing. You should check logs for any errors.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="blue" vlink="purple" lang="EN-US"><div class="m_98662732782548348WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Please help in this.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">…. It's better if gluster volume has more distribute count like 3*3 or 4*3 :-
<span style="color:red">Are you refereeing to create a distributed volume with 3 master node and 3 slave node?</span></p></div></div></blockquote><div><br></div><div>Yes, that's correct. Please do the test with this. I recommend you to run the actual workload for which you are planning to use gluster instead of copying 1GB file and testing.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="blue" vlink="purple" lang="EN-US"><div class="m_98662732782548348WordSection1"><p class="MsoNormal">
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">/krishna<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span class=""><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
</span><b>Sent:</b> Thursday, August 30, 2018 3:20 PM</p><div><div class="h5"><br>
<b>To:</b> Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>><br>
<b>Cc:</b> Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>>; Gluster Users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:red">EXTERNAL MAIL</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Aug 30, 2018 at 1:52 PM, Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Hi Kotresh,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">After fix the library link on node "<span class="m_98662732782548348m-8085366493599475463gmail-im">noi-poc-gluster
</span>", the status of one mater node is “Active” and another is “Passive”. Can I setup both the master as “Active” ?<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Nope, since it's replica, it's redundant to sync same files from two nodes. Both replicas can't be Active.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Also, when I copy a 1GB size of file from gluster client to master gluster volume which is replicated with the slave volume, it tooks 35 minutes and 49 seconds. Is there any way
to reduce its time taken to rsync data. <u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">How did you measure this time? Does this include the time take for you to write 1GB file to master?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">There are two aspects to consider while measuring this.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">1. Time to write 1GB to master<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2. Time for geo-rep to transfer 1GB to slave. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In your case, since the setup is 1*2 and only one geo-rep worker is Active, Step2 above equals to time for step1 + network transfer time.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">You can measure time in two scenarios<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1. If geo-rep is started while the data is still being written to master. It's one way.
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2. Or stop geo-rep until the 1GB file is written to master and then start geo-rep to get actual geo-rep time.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">To improve replicating speed,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1. You can play around with rsync options depending on the kind of I/O<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> and configure the same for geo-rep as it also uses rsync internally.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2. It's better if gluster volume has more distribute count like 3*3 or 4*3<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> It will help in two ways.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> 1. The files gets distributed on master to multiple bricks<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> 2. So above will help geo-rep as files on multiple bricks are synced in parallel (multiple Actives)<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">NOTE: Gluster master server and one client is in Noida, India Location.
<u></u><u></u></p>
<p class="MsoNormal"> Gluster Slave server and one client is in USA.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Our approach is to transfer data from Noida gluster client will reach to the USA gluster client in a minimum time. Please suggest the best approach to achieve it.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img /glusterfs/ ; date<u></u><u></u></p>
<p class="MsoNormal">Thu Aug 30 12:26:26 IST 2018<u></u><u></u></p>
<p class="MsoNormal">sending incremental file list<u></u><u></u></p>
<p class="MsoNormal">gentoo_root.img<u></u><u></u></p>
<p class="MsoNormal"> 1.07G 100% 490.70kB/s 0:35:36 (xfr#1, to-chk=0/1)<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Is this I/O time to write to master volume? <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">sent 1.07G bytes received 35 bytes 499.65K bytes/sec<u></u><u></u></p>
<p class="MsoNormal">total size is 1.07G speedup is 1.00<u></u><u></u></p>
<p class="MsoNormal">Thu Aug 30 13:02:15 IST 2018<u></u><u></u></p>
<p class="MsoNormal">[root@noi-dcops ~]#<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida gluster]# gluster volume geo-replication status<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE <wbr> SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED<u></u><u></u></p>
<p class="MsoNormal">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>---------------------<u></u><u></u></p>
<p class="MsoNormal">gluster-poc-noida glusterep /data/gluster/gv0 root ssh://gluster-poc-sj::<wbr>glusterep gluster-poc-sj Active Changelog Crawl 2018-08-30 13:42:18<u></u><u></u></p>
<p class="MsoNormal">noi-poc-gluster glusterep /data/gluster/gv0 root ssh://gluster-poc-sj::<wbr>glusterep gluster-poc-sj Passive N/A N/A<u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida gluster]#<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks in advance for your all time support.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">/Krishna<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Thursday, August 30, 2018 10:51 AM<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>To:</b> Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>><br>
<b>Cc:</b> Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>>; Gluster Users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:red">EXTERNAL MAIL</span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Did you fix the library link on node "<span class="m_98662732782548348m-8085366493599475463gmail-im">noi-poc-gluster
</span>" as well?<u></u><u></u></p>
</div>
<p class="MsoNormal">If not please fix it. Please share the geo-rep log this node if it's<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">as different issue.<u></u><u></u></p>
</div>
<p class="MsoNormal">-Kotresh HR<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Thu, Aug 30, 2018 at 12:17 AM, Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">Hi Kotresh,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thank you so much for you input. Geo-replication is now showing “Active” atleast for 1 master node. But its still at faulty state for the 2<sup>nd</sup> master server.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Below is the detail.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida glusterfs]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep status<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED<u></u><u></u></p>
<p class="MsoNormal">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------<u></u><u></u></p>
<p class="MsoNormal">gluster-poc-noida glusterep /data/gluster/gv0 root gluster-poc-sj::glusterep gluster-poc-sj Active Changelog Crawl 2018-08-29 23:56:06<u></u><u></u></p>
<p class="MsoNormal">noi-poc-gluster glusterep /data/gluster/gv0 root gluster-poc-sj::glusterep N/A Faulty N/A N/A
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida glusterfs]# gluster volume status<u></u><u></u></p>
<p class="MsoNormal">Status of volume: glusterep<u></u><u></u></p>
<p class="MsoNormal">Gluster process TCP Port RDMA Port Online Pid<u></u><u></u></p>
<p class="MsoNormal">------------------------------<wbr>------------------------------<wbr>------------------<u></u><u></u></p>
<p class="MsoNormal">Brick gluster-poc-noida:/data/<wbr>gluster/gv0 49152 0 Y 22463<u></u><u></u></p>
<p class="MsoNormal">Brick noi-poc-gluster:/data/gluster/<wbr>gv0 49152 0 Y 19471<u></u><u></u></p>
<p class="MsoNormal">Self-heal Daemon on localhost N/A N/A Y 32087<u></u><u></u></p>
<p class="MsoNormal">Self-heal Daemon on noi-poc-gluster N/A N/A Y 6272<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Task Status of Volume glusterep<u></u><u></u></p>
<p class="MsoNormal">------------------------------<wbr>------------------------------<wbr>------------------<u></u><u></u></p>
<p class="MsoNormal">There are no active volume tasks<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida glusterfs]# gluster volume info<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Volume Name: glusterep<u></u><u></u></p>
<p class="MsoNormal">Type: Replicate<u></u><u></u></p>
<p class="MsoNormal">Volume ID: 4a71bc94-14ce-4b2c-abc4-<wbr>e6a9a9765161<u></u><u></u></p>
<p class="MsoNormal">Status: Started<u></u><u></u></p>
<p class="MsoNormal">Snapshot Count: 0<u></u><u></u></p>
<p class="MsoNormal">Number of Bricks: 1 x 2 = 2<u></u><u></u></p>
<p class="MsoNormal">Transport-type: tcp<u></u><u></u></p>
<p class="MsoNormal">Bricks:<u></u><u></u></p>
<p class="MsoNormal">Brick1: gluster-poc-noida:/data/<wbr>gluster/gv0<u></u><u></u></p>
<p class="MsoNormal">Brick2: noi-poc-gluster:/data/gluster/<wbr>gv0<u></u><u></u></p>
<p class="MsoNormal">Options Reconfigured:<u></u><u></u></p>
<p class="MsoNormal">transport.address-family: inet<u></u><u></u></p>
<p class="MsoNormal">nfs.disable: on<u></u><u></u></p>
<p class="MsoNormal">performance.client-io-threads: off<u></u><u></u></p>
<p class="MsoNormal">geo-replication.indexing: on<u></u><u></u></p>
<p class="MsoNormal">geo-replication.ignore-pid-<wbr>check: on<u></u><u></u></p>
<p class="MsoNormal">changelog.changelog: on<u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida glusterfs]#<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Could you please help me in that also please?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">It would be really a great help from your side.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">/Krishna<u></u><u></u></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Wednesday, August 29, 2018 10:47 AM<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>To:</b> Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>><br>
<b>Cc:</b> Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>>; Gluster Users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:red">EXTERNAL MAIL</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Answer inline<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Tue, Aug 28, 2018 at 4:28 PM, Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">Hi Kotresh,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I created the links before. Below is the detail.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]# ls -l /usr/lib64 | grep libgfch<u></u><u></u></p>
<p class="MsoNormal">lrwxrwxrwx 1 root root 30 Aug 28 14:59 libgfchangelog.so -> /usr/lib64/libgfchangelog.so.1<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The link created is pointing to wrong library. Please fix this<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">#cd /usr/lib64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">#rm libgfchangelog.so<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">#ln -s "libgfchangelog.so.0.0.1" libgfchangelog.so<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">lrwxrwxrwx 1 root root 23 Aug 23 23:35 libgfchangelog.so.0 -> libgfchangelog.so.0.0.1<u></u><u></u></p>
<p class="MsoNormal">-rwxr-xr-x 1 root root 63384 Jul 24 19:11 libgfchangelog.so.0.0.1<u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]# locate libgfchangelog.so<u></u><u></u></p>
<p class="MsoNormal">/usr/lib64/libgfchangelog.so.0<u></u><u></u></p>
<p class="MsoNormal">/usr/lib64/libgfchangelog.so.<wbr>0.0.1<u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]#<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Is it looks good what we exactly need or di I need to create any more link or How to get “libgfchangelog.so” file if missing.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">/Krishna<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Tuesday, August 28, 2018 4:22 PM<br>
<b>To:</b> Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>><br>
<b>Cc:</b> Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>>; Gluster Users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>Subject:</b> Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:red">EXTERNAL MAIL</span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Krishna,<u></u><u></u></p>
</div>
<p class="MsoNormal">As per the output shared, I don't see the file "libgfchangelog.so" which is what is required.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I only see "libgfchangelog.so.0". Please confirm "libgfchangelog.so" is present in "/usr/lib64/".<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If not create a symlink similar to "libgfchangelog.so.0"<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">It should be something like below.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">#ls -l /usr/lib64 | grep libgfch<br>
-rwxr-xr-x. 1 root root 1078 Aug 28 05:56 <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__libgfchangelog.la&d=DwMFaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=77GIqpHy9HY8RQd6lKzSJ-Z1PCuIhZJ3I3IvIuDX-xo&s=kIFnrBaSFV_DdqZezd6PXcDnD8Iy_gVN69ETZYtykEE&e=" target="_blank">
libgfchangelog.la</a><br>
lrwxrwxrwx. 1 root root 23 Aug 28 05:56 libgfchangelog.so -> libgfchangelog.so.0.0.1<br>
lrwxrwxrwx. 1 root root 23 Aug 28 05:56 libgfchangelog.so.0 -> libgfchangelog.so.0.0.1<br>
-rwxr-xr-x. 1 root root 336888 Aug 28 05:56 libgfchangelog.so.0.0.1<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Tue, Aug 28, 2018 at 4:04 PM, Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">Hi Kotresh,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks for the response, I did that also but nothing changed.
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]# ldconfig /usr/lib64<u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]# ldconfig -p | grep libgfchangelog<u></u><u></u></p>
<p class="MsoNormal"> libgfchangelog.so.0 (libc6,x86-64) => /usr/lib64/libgfchangelog.so.0<u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]#<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep stop<u></u><u></u></p>
<p class="MsoNormal">Stopping geo-replication session between glusterep & gluster-poc-sj::glusterep has been successful<u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep start<u></u><u></u></p>
<p class="MsoNormal">Starting geo-replication session between glusterep & gluster-poc-sj::glusterep has been successful<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep status<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED<u></u><u></u></p>
<p class="MsoNormal">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-----------------------------<u></u><u></u></p>
<p class="MsoNormal">gluster-poc-noida glusterep /data/gluster/gv0 root gluster-poc-sj::glusterep N/A Faulty N/A N/A<u></u><u></u></p>
<p class="MsoNormal">noi-poc-gluster glusterep /data/gluster/gv0 root gluster-poc-sj::<wbr>glusterep N/A Faulty N/A N/A<u></u><u></u></p>
<p class="MsoNormal">[root@gluster-poc-noida ~]#<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">/Krishna<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> Tuesday, August 28, 2018 4:00 PM<br>
<b>To:</b> Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>><br>
<b>Cc:</b> Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>>; Gluster Users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>Subject:</b> Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:red">EXTERNAL MAIL</span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Krishna,<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Since your libraries are in /usr/lib64, you should be doing<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">#ldconfig /usr/lib64<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Confirm that below command lists the library<u></u><u></u></p>
</div>
<p class="MsoNormal">#ldconfig -p | grep libgfchangelog<u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Tue, Aug 28, 2018 at 3:52 PM, Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">can you do ldconfig /usr/local/lib and share the output of ldconfig -p<br>
/usr/local/lib | grep libgf<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Aug 28, 2018 at 3:45 PM Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>> wrote:<br>
><br>
> Hi Sunny,<br>
><br>
> I did the mentioned changes given in patch and restart the session for geo-replication. But again same errors in the logs.<br>
><br>
> I have attaching the config files and logs here.<br>
><br>
><br>
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep stop<br>
> Stopping geo-replication session between glusterep & gluster-poc-sj::glusterep has been successful<br>
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep delete<br>
> Deleting geo-replication session between glusterep & gluster-poc-sj::glusterep has been successful<br>
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep create push-pem force<br>
> Creating geo-replication session between glusterep & gluster-poc-sj::glusterep has been successful<br>
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep start<br>
> geo-replication start failed for glusterep gluster-poc-sj::glusterep<br>
> geo-replication command failed<br>
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep start<br>
> geo-replication start failed for glusterep gluster-poc-sj::glusterep<br>
> geo-replication command failed<br>
> [root@gluster-poc-noida ~]# vim /usr/libexec/glusterfs/python/<wbr>syncdaemon/repce.py<br>
> [root@gluster-poc-noida ~]# systemctl restart glusterd<br>
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep start<br>
> Starting geo-replication session between glusterep & gluster-poc-sj::glusterep has been successful<br>
> [root@gluster-poc-noida ~]# gluster volume geo-replication glusterep gluster-poc-sj::glusterep status<br>
><br>
> MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED<br>
> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-----------------------------<br>
> gluster-poc-noida glusterep /data/gluster/gv0 root gluster-poc-sj::glusterep N/A Faulty N/A N/A<br>
> noi-poc-gluster glusterep /data/gluster/gv0 root gluster-poc-sj::glusterep N/A Faulty N/A N/A<br>
> [root@gluster-poc-noida ~]#<br>
><br>
><br>
> /Krishna.<br>
><br>
> -----Original Message-----<br>
> From: Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>><br>
> Sent: Tuesday, August 28, 2018 3:17 PM<br>
> To: Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>><br>
> Cc: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
> Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work<br>
><br>
> EXTERNAL MAIL<br>
><br>
><br>
> With same log message ?<br>
><br>
> Can you please verify that<br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__review.gluster.org_-23_c_glusterfs_-2B_20207_&d=DwIBaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=F0ExtFUfa_YCktOGvy82x3IAxvi2GrbPR72jZ8beuYk&s=fGtkmezHJj5YoLN3dUeVUCcYFnREHyOSk36mRjbTTEQ&e=" target="_blank">
https://urldefense.proofpoint.<wbr>com/v2/url?u=https-3A__review.<wbr>gluster.org_-23_c_glusterfs_-<wbr>2B_20207_&d=DwIBaQ&c=<wbr>aUq983L2pue2FqKFoP6PGHMJQyoJ7k<wbr>l3s3GZ-_haXqY&r=<wbr>0E5nRoxLsT2ZXgCpJM_<wbr>6ZItAWQ2jH8rVLG6tiXhoLFE&m=<wbr>F0ExtFUfa_<wbr>YCktOGvy82x3IAxvi2GrbPR72jZ8be<wbr>uYk&s=<wbr>fGtkmezHJj5YoLN3dUeVUCcYFnREHy<wbr>OSk36mRjbTTEQ&e=</a>
patch is present if not can you please apply that.<br>
> and try with symlinking ln -s /usr/lib64/libgfchangelog.so.0 /usr/lib64/libgfchangelog.so.<br>
><br>
> Please share the log also.<br>
><br>
> Regards,<br>
> Sunny<br>
> On Tue, Aug 28, 2018 at 3:02 PM Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>> wrote:<br>
> ><br>
> > Hi Sunny,<br>
> ><br>
> > Thanks for your response, I tried both, but still I am getting the same error.<br>
> ><br>
> ><br>
> > [root@noi-poc-gluster ~]# ldconfig /usr/lib [root@noi-poc-gluster ~]#<br>
> ><br>
> > [root@noi-poc-gluster ~]# ln -s /usr/lib64/libgfchangelog.so.1<br>
> > /usr/lib64/libgfchangelog.so [root@noi-poc-gluster ~]# ls -l<br>
> > /usr/lib64/libgfchangelog.so lrwxrwxrwx. 1 root root 30 Aug 28 14:59<br>
> > /usr/lib64/libgfchangelog.so -> /usr/lib64/libgfchangelog.so.1<br>
> ><br>
> > /Krishna<br>
> ><br>
> > -----Original Message-----<br>
> > From: Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>><br>
> > Sent: Tuesday, August 28, 2018 2:55 PM<br>
> > To: Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>><br>
> > Cc: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
> > Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not<br>
> > work<br>
> ><br>
> > EXTERNAL MAIL<br>
> ><br>
> ><br>
> > Hi Krish,<br>
> ><br>
> > You can run -<br>
> > #ldconfig /usr/lib<br>
> ><br>
> > If that still does not solves your problem you can do manual symlink<br>
> > like - ln -s /usr/lib64/libgfchangelog.so.1<br>
> > /usr/lib64/libgfchangelog.so<br>
> ><br>
> > Thanks,<br>
> > Sunny Kumar<br>
> > On Tue, Aug 28, 2018 at 1:47 PM Krishna Verma <<a href="mailto:kverma@cadence.com" target="_blank">kverma@cadence.com</a>> wrote:<br>
> > ><br>
> > > Hi<br>
> > ><br>
> > ><br>
> > ><br>
> > > I am getting below error in gsyncd.log<br>
> > ><br>
> > ><br>
> > ><br>
> > > OSError: libgfchangelog.so: cannot open shared object file: No such<br>
> > > file or directory<br>
> > ><br>
> > > [2018-08-28 07:19:41.446785] E [repce(worker /data/gluster/gv0):197:__call_<wbr>_] RepceClient: call failed call=26469:139794524604224:<wbr>1535440781.44 method=init error=OSError<br>
> > ><br>
> > > [2018-08-28 07:19:41.447041] E [syncdutils(worker /data/gluster/gv0):330:log_<wbr>raise_exception] <top>: FAIL:<br>
> > ><br>
> > > Traceback (most recent call last):<br>
> > ><br>
> > > File "/usr/libexec/glusterfs/<wbr>python/syncdaemon/gsyncd.py", line<br>
> > > 311, in main<br>
> > ><br>
> > > func(args)<br>
> > ><br>
> > > File "/usr/libexec/glusterfs/<wbr>python/syncdaemon/subcmds.py", line<br>
> > > 72, in subcmd_worker<br>
> > ><br>
> > > local.service_loop(remote)<br>
> > ><br>
> > > File "/usr/libexec/glusterfs/<wbr>python/syncdaemon/resource.py"<wbr>, line<br>
> > > 1236, in service_loop<br>
> > ><br>
> > > changelog_agent.init()<br>
> > ><br>
> > > File "/usr/libexec/glusterfs/<wbr>python/syncdaemon/repce.py", line<br>
> > > 216, in __call__<br>
> > ><br>
> > > return self.ins(self.meth, *a)<br>
> > ><br>
> > > File "/usr/libexec/glusterfs/<wbr>python/syncdaemon/repce.py", line<br>
> > > 198, in __call__<br>
> > ><br>
> > > raise res<br>
> > ><br>
> > > OSError: libgfchangelog.so: cannot open shared object file: No such<br>
> > > file or directory<br>
> > ><br>
> > > [2018-08-28 07:19:41.457555] I [repce(agent /data/gluster/gv0):80:service_<wbr>loop] RepceServer: terminating on reaching EOF.<br>
> > ><br>
> > > [2018-08-28 07:19:42.440184] I [monitor(monitor):272:monitor] Monitor:<br>
> > > worker died in startup phase brick=/data/gluster/gv0<br>
> > ><br>
> > ><br>
> > ><br>
> > > Below is my file location:<br>
> > ><br>
> > ><br>
> > ><br>
> > > /usr/lib64/libgfchangelog.so.0<br>
> > ><br>
> > > /usr/lib64/libgfchangelog.so.<wbr>0.0.1<br>
> > ><br>
> > ><br>
> > ><br>
> > > What I can do to fix it ?<br>
> > ><br>
> > ><br>
> > ><br>
> > > /Krish<br>
> > ><br>
> > > ______________________________<wbr>_________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > > <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.o" target="_blank">
https://urldefense.proofpoint.<wbr>com/v2/url?u=https-3A__lists.<wbr>gluster.o</a><br>
> > > rg<br>
> > > _mailman_listinfo_gluster-<wbr>2Dusers&d=DwIBaQ&c=<wbr>aUq983L2pue2FqKFoP6PGHM<br>
> > > JQ<br>
> > > yoJ7kl3s3GZ-_haXqY&r=<wbr>0E5nRoxLsT2ZXgCpJM_<wbr>6ZItAWQ2jH8rVLG6tiXhoLFE&m=_<br>
> > > u6<br>
> > > vGRjlVsype7Z8hXDgCONilqVe4sIWk<wbr>XNqqz2n3IQ&s=<wbr>i0EUwtUHurhJHyw9UPpepCdLB<br>
> > > 70<br>
> > > 1mkxoNZWYvU7XXug&e=<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.org_mailman_listinfo_gluster-2Dusers&d=DwMFaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=BuUVeXUwqxdQDoP-IWDI_JPFCQGrRFdzV9g0enLn8kM&s=X0eVYgZ1emUQnEwRTg0AGbAWyoIwIyrE_gzmuolgiPE&e=" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Kotresh H R<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div></div>