[Gluster-users] Geo Replication stops replicating

Thu Jun 6 12:52:47 UTC 2019

You should not have used this one:
>
> gluster-mountbroker remove --volume code-misc --user sas

-- This one is to remove volume/user from mount broker.

Please try setting up mount broker once again.

-Sunny

On Thu, Jun 6, 2019 at 5:28 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>
> Hi Sunny
> Please find the logs attached
>>
>> The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file" repeated 13 times between [2019-06-06 11:51:43.986788] and [2019-06-06 11:52:32.764546]
>>
>> The message "W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]" repeated 13 times between [2019-06-06 11:51:43.986798] and [2019-06-06 11:52:32.764548]
>>
>> The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-06-06 11:53:07.064332] and [2019-06-06 11:53:07.303978]
>>
>> [2019-06-06 11:55:35.624320] I [MSGID: 106495] [glusterd-handler.c:3137:__glusterd_handle_getwd] 0-glusterd: Received getwd req
>>
>> [2019-06-06 11:55:35.884345] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: quotad already stopped
>>
>> [2019-06-06 11:55:35.884373] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: quotad service is stopped
>>
>> [2019-06-06 11:55:35.884459] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
>>
>> [2019-06-06 11:55:35.884473] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: bitd service is stopped
>>
>> [2019-06-06 11:55:35.884554] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: scrub already stopped
>>
>> [2019-06-06 11:55:35.884567] I [MSGID: 106568] [glusterd-svc-mgmt.c:253:glusterd_svc_stop] 0-management: scrub service is stopped
>>
>> [2019-06-06 11:55:35.893823] I [run.c:242:runner_log] (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a) [0x7f7380d60e1a] -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5) [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f738cbc5df5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=code-misc -o features.read-only=on --gd-workdir=/var/lib/glusterd
>>
>> [2019-06-06 11:55:35.900465] I [run.c:242:runner_log] (-->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe8e1a) [0x7f7380d60e1a] -->/usr/lib64/glusterfs/6.2/xlator/mgmt/glusterd.so(+0xe88e5) [0x7f7380d608e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f738cbc5df5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=code-misc -o features.read-only=on --gd-workdir=/var/lib/glusterd
>>
>> [2019-06-06 11:55:43.485284] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
>>
>> The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-06-06 11:55:43.485284] and [2019-06-06 11:55:43.512321]
>>
>> [2019-06-06 11:55:44.055419] I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req
>>
>> [2019-06-06 11:55:44.055473] E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:44.055483] W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]
>>
>> [2019-06-06 11:55:44.056695] I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req
>>
>> [2019-06-06 11:55:44.056725] E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:44.056734] W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]
>>
>> [2019-06-06 11:55:44.057522] I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req
>>
>> [2019-06-06 11:55:44.057552] E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:44.057562] W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]
>>
>> [2019-06-06 11:55:54.655681] I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req
>>
>> [2019-06-06 11:55:54.655741] E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file
>>
>> [2019-06-06 11:55:54.655752] W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]
>
>
> On Thu, Jun 6, 2019 at 5:09 PM Sunny Kumar <sunkumar at redhat.com> wrote:
>>
>> Whats current trackback please share.
>>
>> -Sunny
>>
>>
>> On Thu, Jun 6, 2019 at 4:53 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >
>> > Hi Sunny
>> > I have changed the file in /usr/libexec/glusterfs/peer_mountbroker.py as mentioned in the patch.
>> > Now the "gluster-mountbroker status" command is working fine. But the geo-replication seems to be in the faulty state still.
>> >
>> >
>> > Thankyou
>> > Deepak
>> >
>> > On Thu, Jun 6, 2019 at 4:10 PM Sunny Kumar <sunkumar at redhat.com> wrote:
>> >>
>> >> Above error can be tracked here:
>> >>
>> >> https://bugzilla.redhat.com/show_bug.cgi?id=1709248
>> >>
>> >> and patch link:
>> >> https://review.gluster.org/#/c/glusterfs/+/22716/
>> >>
>> >> You can apply patch and test it however its waiting on regression to
>> >> pass and merge.
>> >>
>> >> -Sunny
>> >>
>> >>
>> >> On Thu, Jun 6, 2019 at 4:00 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >
>> >> > Hi
>> >> > I have followed the following steps to create the geo-replication but the status seems to be in a faulty state.
>> >> >
>> >> > Steps :
>> >> >
>> >> > Installed cluster version 5.6 in totally six nodes.
>> >> >>
>> >> >> glusterfs 5.6
>> >> >>
>> >> >> Repository revision: git://git.gluster.org/glusterfs.git
>> >> >>
>> >> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
>> >> >>
>> >> >> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> >> >>
>> >> >> It is licensed to you under your choice of the GNU Lesser
>> >> >>
>> >> >> General Public License, version 3 or any later version (LGPLv3
>> >> >>
>> >> >> or later), or the GNU General Public License, version 2 (GPLv2),
>> >> >>
>> >> >> in all cases as published by the Free Software Foundation
>> >> >
>> >> >
>> >> > peer_probed the first three nodes and second three nodes.
>> >> >
>> >> >
>> >> >
>> >> > Added new volume in both the clusters
>> >> >
>> >> >
>> >> >
>> >> > execute gluster-mountbroker commands and restarted glusterd.
>> >> >>
>> >> >> gluster-mountbroker setup /var/mountbroker-root sas
>> >> >>
>> >> >> gluster-mountbroker remove --volume code-misc --user sas
>> >> >
>> >> >
>> >> > configured a passwordless sssh from master to slave
>> >> >>
>> >> >> ssh-keygen; ssh-copy-id sas at 192.168.185.107
>> >> >
>> >> > created a common pem pub file
>> >> >>
>> >> >> gluster system:: execute gsec_create
>> >> >
>> >> > created geo-replication session.
>> >> >>
>> >> >> gluster volume geo-replication code-misc sas at 192.168.185.107::code-misc create push-pem
>> >> >
>> >> >  executed the following command in slave
>> >> >>
>> >> >> /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh sas code-misc code-misc
>> >> >
>> >> > started the gluster geo-replication.
>> >> >>
>> >> >> gluster volume geo-replication code-misc sas at 192.168.185.107::code-misc start
>> >> >
>> >> >
>> >> > Now the geo-replication works fine.
>> >> > Tested with 2000 files All seems to sync finely.
>> >> >
>> >> > Now I updated all the node to version 6.2 by using rpms which were built by the source code in a docker container in my personal machine.
>> >> >
>> >> >
>> >> >> gluster --version
>> >> >>
>> >> >> glusterfs 6.2
>> >> >>
>> >> >> Repository revision: git://git.gluster.org/glusterfs.git
>> >> >>
>> >> >> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
>> >> >>
>> >> >> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> >> >>
>> >> >> It is licensed to you under your choice of the GNU Lesser
>> >> >>
>> >> >> General Public License, version 3 or any later version (LGPLv3
>> >> >>
>> >> >> or later), or the GNU General Public License, version 2 (GPLv2),
>> >> >>
>> >> >> in all cases as published by the Free Software Foundation.
>> >> >
>> >> >
>> >> > I have stopped the glusterd daemons in all the node along with the volume and geo-replication.
>> >> > Now I started the daemons, volume and geo-replication session the status seems to be faulty.
>> >> > Also noted that the result of "gluster-mountbroker status" command always end in python exception like this
>> >> >>
>> >> >> Traceback (most recent call last):
>> >> >>
>> >> >>   File "/usr/sbin/gluster-mountbroker", line 396, in <module>
>> >> >>
>> >> >>     runcli()
>> >> >>
>> >> >>   File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 225, in runcli
>> >> >>
>> >> >>     cls.run(args)
>> >> >>
>> >> >>   File "/usr/sbin/gluster-mountbroker", line 275, in run
>> >> >>
>> >> >>     out = execute_in_peers("node-status")
>> >> >>
>> >> >>   File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 127, in execute_in_peers
>> >> >>
>> >> >>     raise GlusterCmdException((rc, out, err, " ".join(cmd)))
>> >> >>
>> >> >> gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Unable to end. Error : Success\n', 'gluster system:: execute mountbroker.py node-status')
>> >> >
>> >> >
>> >> > Is it I or everyone gets an error for gluster-mountbroker command for gluster version greater than 6.0?. Please help.
>> >> >
>> >> > Thank you
>> >> > Deepak
>> >> >
>> >> >
>> >> > On Thu, Jun 6, 2019 at 10:35 AM Sunny Kumar <sunkumar at redhat.com> wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> Updated link for documentation :
>> >> >>
>> >> >> --  https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
>> >> >>
>> >> >> You can use this tool as well:
>> >> >> http://aravindavk.in/blog/gluster-georep-tools/
>> >> >>
>> >> >> -Sunny
>> >> >>
>> >> >> On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar
>> >> >> <khiremat at redhat.com> wrote:
>> >> >> >
>> >> >> > Hi,
>> >> >> >
>> >> >> > I think the steps to setup non-root geo-rep is not followed properly. The following entry is missing in glusterd vol file which is required.
>> >> >> >
>> >> >> > The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file" repeated 33 times between [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]
>> >> >> >
>> >> >> > Could you please the steps from below?
>> >> >> >
>> >> >> > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave
>> >> >> >
>> >> >> > And let us know if you still face the issue.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>
>> >> >> >> Hi Kotresh, Sunny
>> >> >> >> I Have mailed the logs I found in one of the slave machines. Is there anything to do with permission? Please help.
>> >> >> >>
>> >> >> >> On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>
>> >> >> >>> Hi Kotresh, Sunny
>> >> >> >>> Found this log in the slave machine.
>> >> >> >>>>
>> >> >> >>>> [2019-06-05 08:49:10.632583] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
>> >> >> >>>>
>> >> >> >>>> The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-06-05 08:49:10.632583] and [2019-06-05 08:49:10.670863]
>> >> >> >>>>
>> >> >> >>>> The message "I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req" repeated 34 times between [2019-06-05 08:48:41.005398] and [2019-06-05 08:50:37.254063]
>> >> >> >>>>
>> >> >> >>>> The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file" repeated 34 times between [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079]
>> >> >> >>>>
>> >> >> >>>> The message "W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]" repeated 34 times between [2019-06-05 08:48:41.005444] and [2019-06-05 08:50:37.254080]
>> >> >> >>>>
>> >> >> >>>> [2019-06-05 08:50:46.361347] I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req
>> >> >> >>>>
>> >> >> >>>> [2019-06-05 08:50:46.361384] E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file
>> >> >> >>>>
>> >> >> >>>> [2019-06-05 08:50:46.361419] W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]
>> >> >> >>>>
>> >> >> >>>> The message "I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req" repeated 33 times between [2019-06-05 08:50:46.361347] and [2019-06-05 08:52:34.019741]
>> >> >> >>>>
>> >> >> >>>> The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file" repeated 33 times between [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757]
>> >> >> >>>>
>> >> >> >>>> The message "W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]" repeated 33 times between [2019-06-05 08:50:46.361419] and [2019-06-05 08:52:34.019758]
>> >> >> >>>>
>> >> >> >>>> [2019-06-05 08:52:44.426839] I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req
>> >> >> >>>>
>> >> >> >>>> [2019-06-05 08:52:44.426886] E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file
>> >> >> >>>>
>> >> >> >>>> [2019-06-05 08:52:44.426896] W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>
>> >> >> >>>> Thankyou Kotresh
>> >> >> >>>>
>> >> >> >>>> On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar <khiremat at redhat.com> wrote:
>> >> >> >>>>>
>> >> >> >>>>> Ccing Sunny, who was investing similar issue.
>> >> >> >>>>>
>> >> >> >>>>> On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>
>> >> >> >>>>>> Have already added the path in bashrc . Still in faulty state
>> >> >> >>>>>>
>> >> >> >>>>>> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar <khiremat at redhat.com> wrote:
>> >> >> >>>>>>>
>> >> >> >>>>>>> could you please try adding /usr/sbin to $PATH for user 'sas'? If it's bash, add 'export PATH=/usr/sbin:$PATH' in
>> >> >> >>>>>>> /home/sas/.bashrc
>> >> >> >>>>>>>
>> >> >> >>>>>>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> Hi Kortesh
>> >> >> >>>>>>>> Please find the logs of the above error
>> >> >> >>>>>>>> Master log snippet
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> [2019-06-04 11:52:09.254731] I [resource(worker /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing SSH connection between master and slave...
>> >> >> >>>>>>>>>  [2019-06-04 11:52:09.308923] D [repce(worker /home/sas/gluster/data/code-misc):196:push] RepceClient: call 89724:139652759443264:1559649129.31 __repce_version__() ...
>> >> >> >>>>>>>>>  [2019-06-04 11:52:09.602792] E [syncdutils(worker /home/sas/gluster/data/code-misc):311:log_raise_exception] <top>: connection to peer is broken
>> >> >> >>>>>>>>>  [2019-06-04 11:52:09.603312] E [syncdutils(worker /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned error   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/ glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock sas at 192.168.185.107 /usr/libexec/glusterfs/gsyncd slave code-misc sas@   192.168.185.107::code-misc --master-node 192.168.185.106 --master-node-id 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick /home/sas/gluster/data/code-misc --local-node 192.168.185.122 --local-node-   id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120 --slave-log-level DEBUG --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin   error=1
>> >> >> >>>>>>>>>  [2019-06-04 11:52:09.614996] I [repce(agent /home/sas/gluster/data/code-misc):97:service_loop] RepceServer: terminating on reaching EOF.
>> >> >> >>>>>>>>>  [2019-06-04 11:52:09.615545] D [monitor(monitor):271:monitor] Monitor: worker(/home/sas/gluster/data/code-misc) connected
>> >> >> >>>>>>>>>  [2019-06-04 11:52:09.616528] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-misc
>> >> >> >>>>>>>>>  [2019-06-04 11:52:09.619391] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty
>> >> >> >>>>>>>>
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> Slave log snippet
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> [2019-06-04 11:50:09.782668] E [syncdutils(slave 192.168.185.106/home/sas/gluster/data/code-misc):809:logerr] Popen: /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
>> >> >> >>>>>>>>> [2019-06-04 11:50:11.188167] W [gsyncd(slave 192.168.185.125/home/sas/gluster/data/code-misc):305:main] <top>: Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf
>> >> >> >>>>>>>>> [2019-06-04 11:50:11.201070] I [resource(slave 192.168.185.125/home/sas/gluster/data/code-misc):1098:connect] GLUSTER: Mounting gluster volume locally...
>> >> >> >>>>>>>>> [2019-06-04 11:50:11.271231] E [resource(slave 192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter] MountbrokerMounter: glusterd answered mnt=
>> >> >> >>>>>>>>> [2019-06-04 11:50:11.271998] E [syncdutils(slave 192.168.185.125/home/sas/gluster/data/code-misc):805:errlog] Popen: command returned error cmd=/usr/sbin/gluster --remote-host=localhost system:: mount sas user-map-root=sas aux-gfid-mount acl log-level=INFO log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log volfile-server=localhost volfile-id=code-misc client-pid=-1 error=1
>> >> >> >>>>>>>>> [2019-06-04 11:50:11.272113] E [syncdutils(slave 192.168.185.125/home/sas/gluster/data/code-misc):809:logerr] Popen: /usr/sbin/gluster> 2 : failed with this errno (No such file or directory)
>> >> >> >>>>>>>>
>> >> >> >>>>>>>>
>> >> >> >>>>>>>> On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> Hi
>> >> >> >>>>>>>>> As discussed I have upgraded gluster from 4.1 to 6.2 version. But the Geo replication failed to start.
>> >> >> >>>>>>>>> Stays in faulty state
>> >> >> >>>>>>>>>
>> >> >> >>>>>>>>> On Fri, May 31, 2019, 5:32 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> Checked the data. It remains in 2708. No progress.
>> >> >> >>>>>>>>>>
>> >> >> >>>>>>>>>> On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <khiremat at redhat.com> wrote:
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>> That means it could be working and the defunct process might be some old zombie one. Could you check, that data progress ?
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>> On Fri, May 31, 2019 at 4:29 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> Hi
>> >> >> >>>>>>>>>>>> When i change the rsync option the rsync process doesnt seem to start . Only a defunt process is listed in ps aux. Only when i set rsync option to " " and restart all the process the rsync process is listed in ps aux.
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>
>> >> >> >>>>>>>>>>>> On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <khiremat at redhat.com> wrote:
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>> Yes, rsync config option should have fixed this issue.
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>> Could you share the output of the following?
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>> 1. gluster volume geo-replication <MASTERVOL> <SLAVEHOST>::<SLAVEVOL> config rsync-options
>> >> >> >>>>>>>>>>>>> 2. ps -ef | grep rsync
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>> On Fri, May 31, 2019 at 4:11 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>> Done.
>> >> >> >>>>>>>>>>>>>> We got the following result .
>> >> >> >>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>> 1559298781.338234 write(2, "rsync: link_stat \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\" failed: No such file or directory (2)", 128
>> >> >> >>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>> seems like a file is missing ?
>> >> >> >>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <khiremat at redhat.com> wrote:
>> >> >> >>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>> Hi,
>> >> >> >>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>> Could you take the strace with with more string size? The argument strings are truncated.
>> >> >> >>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>> strace -s 500 -ttt -T -p <rsync pid>
>> >> >> >>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:17 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>> Hi Kotresh
>> >> >> >>>>>>>>>>>>>>>> The above-mentioned work around did not work properly.
>> >> >> >>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>> On Fri, May 31, 2019 at 3:16 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>> Hi Kotresh
>> >> >> >>>>>>>>>>>>>>>>> We have tried the above-mentioned rsync option and we are planning to have the version upgrade to 6.0.
>> >> >> >>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>> On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <khiremat at redhat.com> wrote:
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>> Hi,
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>> This looks like the hang because stderr buffer filled up with errors messages and no one reading it.
>> >> >> >>>>>>>>>>>>>>>>>> I think this issue is fixed in latest releases. As a workaround, you can do following and check if it works.
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>> Prerequisite:
>> >> >> >>>>>>>>>>>>>>>>>>  rsync version should be > 3.1.0
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>> Workaround:
>> >> >> >>>>>>>>>>>>>>>>>> gluster volume geo-replication <MASTERVOL> <SLAVEHOST>::<SLAVEVOL> config rsync-options "--ignore-missing-args"
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>> Thanks,
>> >> >> >>>>>>>>>>>>>>>>>> Kotresh HR
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>> On Thu, May 30, 2019 at 5:39 PM deepu srinivasan <sdeepugd at gmail.com> wrote:
>> >> >> >>>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>> Hi
>> >> >> >>>>>>>>>>>>>>>>>>> We were evaluating Gluster geo Replication between two DCs one is in US west and one is in US east. We took multiple trials for different file size.
>> >> >> >>>>>>>>>>>>>>>>>>> The Geo Replication tends to stop replicating but while checking the status it appears to be in Active state. But the slave volume did not increase in size.
>> >> >> >>>>>>>>>>>>>>>>>>> So we have restarted the geo-replication session and checked the status. The status was in an active state and it was in History Crawl for a long time. We have enabled the DEBUG mode in logging and checked for any error.
>> >> >> >>>>>>>>>>>>>>>>>>> There was around 2000 file appeared for syncing candidate. The Rsync process starts but the rsync did not happen in the slave volume. Every time the rsync process appears in the "ps auxxx" list but the replication did not happen in the slave end. What would be the cause of this problem? Is there anyway to debug it?
>> >> >> >>>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>> We have also checked the strace of the rync program.
>> >> >> >>>>>>>>>>>>>>>>>>> it displays something like this
>> >> >> >>>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>> "write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128"
>> >> >> >>>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>> We are using the below specs
>> >> >> >>>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>> Gluster version - 4.1.7
>> >> >> >>>>>>>>>>>>>>>>>>> Sync mode - rsync
>> >> >> >>>>>>>>>>>>>>>>>>> Volume - 1x3 in each end (master and slave)
>> >> >> >>>>>>>>>>>>>>>>>>> Intranet Bandwidth - 10 Gig
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>>>> --
>> >> >> >>>>>>>>>>>>>>>>>> Thanks and Regards,
>> >> >> >>>>>>>>>>>>>>>>>> Kotresh H R
>> >> >> >>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>>> --
>> >> >> >>>>>>>>>>>>>>> Thanks and Regards,
>> >> >> >>>>>>>>>>>>>>> Kotresh H R
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>>
>> >> >> >>>>>>>>>>>>> --
>> >> >> >>>>>>>>>>>>> Thanks and Regards,
>> >> >> >>>>>>>>>>>>> Kotresh H R
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>>
>> >> >> >>>>>>>>>>> --
>> >> >> >>>>>>>>>>> Thanks and Regards,
>> >> >> >>>>>>>>>>> Kotresh H R
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>>
>> >> >> >>>>>>> --
>> >> >> >>>>>>> Thanks and Regards,
>> >> >> >>>>>>> Kotresh H R
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> --
>> >> >> >>>>> Thanks and Regards,
>> >> >> >>>>> Kotresh H R
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Thanks and Regards,
>> >> >> > Kotresh H R