[Gluster-users] Geo-Replication Issue

Thu Dec 11 06:13:41 UTC 2014

Hi Dave,

Two things.

1. I see the gluster has been upgraded from 3.4.2 to 3.5.3. 
   Geo-rep has undergone design changes to make it distributed 
   between these releases (https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_distributed_geo_rep.md).
   Have you followed all the upgrade steps w.r.t geo-rep 
   mentioned in the following link?

   http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5

2. Does the output of command 'gluster vol info <vol-name> --xml' proper ?
   Please paste the output.

Thanks and Regards,
Kotresh H R

----- Original Message -----
From: "David Gibbons" <david.c.gibbons at gmail.com>
To: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com>
Cc: "gluster-users" <Gluster-users at gluster.org>, vnosov at stonefly.com
Sent: Wednesday, December 10, 2014 6:12:00 PM
Subject: Re: [Gluster-users] Geo-Replication Issue

Symlinking gluster to /usr/bin/ seems to have resolved the path issue.
Thanks for the tip there.

Now there's a different error throw in the geo-rep/ssh...log:

> [2014-12-10 07:32:42.609031] E
>> [syncdutils(monitor):240:log_raise_exception] <top>: FAIL:
>
> Traceback (most recent call last):
>
>   File "/usr/local/libexec/glusterfs/python/syncdaemon/gsyncd.py", line
>> 150, in main
>
>     main_i()
>
>   File "/usr/local/libexec/glusterfs/python/syncdaemon/gsyncd.py", line
>> 530, in main_i
>
>     return monitor(*rscs)
>
>   File "/usr/local/libexec/glusterfs/python/syncdaemon/monitor.py", line
>> 243, in monitor
>
>     return Monitor().multiplex(*distribute(*resources))
>
>   File "/usr/local/libexec/glusterfs/python/syncdaemon/monitor.py", line
>> 205, in distribute
>
>     mvol = Volinfo(master.volume, master.host)
>
>   File "/usr/local/libexec/glusterfs/python/syncdaemon/monitor.py", line
>> 22, in __init__
>
>     vi = XET.fromstring(vix)
>
>   File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 963, in XML
>
>     parser.feed(text)
>
>   File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 1245, in feed
>
>     self._parser.Parse(data, 0)
>
> ExpatError: syntax error: line 2, column 0
>
> [2014-12-10 07:32:42.610858] I [syncdutils(monitor):192:finalize] <top>:
>> exiting.
>
>
I also get a bunch of these errors but have been assuming that they are
being thrown because geo-replication hasn't started successfully yet. There
is one for each brick:

> [2014-12-10 12:33:33.539737] E
>> [glusterd-geo-rep.c:2685:glusterd_gsync_read_frm_status] 0-: Unable to read
>> gsyncd status file
>
> [2014-12-10 12:33:33.539742] E
>> [glusterd-geo-rep.c:2999:glusterd_read_status_file] 0-: Unable to read the
>> statusfile for /mnt/a-3-shares-brick-4/brick brick for  shares(master),
>> gfs-a-bkp::bkpshares(slave) session
>
>
Do I have a config file error somewhere that I need to track down? This
volume *was* upgraded from 3.4.2 a few weeks ago.

Cheers,
Dave

On Wed, Dec 10, 2014 at 7:29 AM, David Gibbons <david.c.gibbons at gmail.com>
wrote:

> Hi Kotresh,
>
> Thanks for the tip. Unfortunately that does not seem to have any effect.
> The path to the gluster binaries was already in $PATH. I did try adding the
> path to the gsyncd binary, but same result. Contents of $PATH are:
>
>>
>> /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/libexec/glusterfs/
>>
>
> It seems like perhaps one of the remote gsyncd processes cannot find the
> gluster binary, because I see the following in the
> geo-replication/shares/ssh...log. Can you point me toward how I can find
> out what is throwing this log entry?
>
>> [2014-12-10 07:20:53.886676] E
>>> [syncdutils(monitor):218:log_raise_exception] <top>: execution of "gluster"
>>> failed with ENOENT (No such file or directory)
>>
>> [2014-12-10 07:20:53.886883] I [syncdutils(monitor):192:finalize] <top>:
>>> exiting.
>>
>>
> I think that whatever process is trying to use the gluster command has the
> incorrect path to access it. Do you know how I could modify *that* path?
>
> I've manually tested the ssh_command and ssh_command_tar variables in the
> relevant gsyncd.conf; both connect to the slave server successfully and
> appear to execute the command they're supposed to.
>
> gluster_command_dir in gsyncd.conf is also the correct directory
> (/usr/local/sbin).
>
> In summary: I think we're on to something with setting the path, but I
> think I need to set it somewhere other than my shell.
>
> Thanks,
> Dave
>
>
> On Tue, Dec 9, 2014 at 11:52 PM, Kotresh Hiremath Ravishankar <
> khiremat at redhat.com> wrote:
>
>> If that is the case, as a workaround, try adding 'gluster' path
>> to PATH environment variable or creating symlinks to gluster,
>> glusterd binaries.
>>
>> 1. export PATH=$PATH:<path where gluster binaries are installed>
>>
>> Above should work, let me know if doesn't.
>>
>> Thanks and Regards,
>> Kotresh H R
>>
>> ----- Original Message -----
>> From: "David Gibbons" <david.c.gibbons at gmail.com>
>> To: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com>
>> Cc: "gluster-users" <Gluster-users at gluster.org>, vnosov at stonefly.com
>> Sent: Tuesday, December 9, 2014 6:16:03 PM
>> Subject: Re: [Gluster-users] Geo-Replication Issue
>>
>> Hi Kotresh,
>>
>> Yes, I believe that I am. Can you tell me which symlinks are missing/cause
>> geo-replication to fail to start? I can create them manually.
>>
>> Thank you,
>> Dave
>>
>> On Tue, Dec 9, 2014 at 3:54 AM, Kotresh Hiremath Ravishankar <
>> khiremat at redhat.com> wrote:
>>
>> > Hi Dave,
>> >
>> > Are you hitting the below bug and so not able to sync symlinks ?
>> > https://bugzilla.redhat.com/show_bug.cgi?id=1105283
>> >
>> > Does geo-rep status say "Not Started" ?
>> >
>> > Thanks and Regards,
>> > Kotresh H R
>> >
>> > ----- Original Message -----
>> > From: "David Gibbons" <david.c.gibbons at gmail.com>
>> > To: "gluster-users" <Gluster-users at gluster.org>
>> > Cc: vnosov at stonefly.com
>> > Sent: Monday, December 8, 2014 7:03:31 PM
>> > Subject: Re: [Gluster-users] Geo-Replication Issue
>> >
>> > Apologies for sending so many messages about this! I think I may be
>> > running into this bug:
>> > https://bugzilla.redhat.com/show_bug.cgi?id=1105283
>> >
>> > Would someone be so kind as to let me know which symlinks are missing
>> when
>> > this bug manifests, so that I can create them?
>> >
>> > Thank you,
>> > Dave
>> >
>> >
>> > On Sun, Dec 7, 2014 at 11:01 AM, David Gibbons <
>> david.c.gibbons at gmail.com
>> > > wrote:
>> >
>> >
>> >
>> > Ok,
>> >
>> > I was able to get geo-replication configured by changing
>> > /usr/local/libexec/glusterfs/gverify.sh to use ssh to access the local
>> > machine, instead of accessing bash -c directly. I then found that the
>> hook
>> > script was missing for geo-replication, so I copied that over manually.
>> I
>> > now have what appears to be a "configured" geo-rep setup:
>> >
>> >
>> >
>> >
>> > # gluster volume geo-replication shares gfs-a-bkp::bkpshares status
>> >
>> >
>> >
>> >
>> > MASTER NODE MASTER VOL MASTER BRICK SLAVE STATUS CHECKPOINT STATUS CRAWL
>> > STATUS
>> >
>> >
>> >
>> --------------------------------------------------------------------------------------------------------------------------------------------------------
>> >
>> > gfs-a-3 shares /mnt/a-3-shares-brick-1/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-3 shares /mnt/a-3-shares-brick-2/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-3 shares /mnt/a-3-shares-brick-3/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-3 shares /mnt/a-3-shares-brick-4/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-2 shares /mnt/a-2-shares-brick-1/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-2 shares /mnt/a-2-shares-brick-2/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-2 shares /mnt/a-2-shares-brick-3/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-2 shares /mnt/a-2-shares-brick-4/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-4 shares /mnt/a-4-shares-brick-1/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-4 shares /mnt/a-4-shares-brick-2/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-4 shares /mnt/a-4-shares-brick-3/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-4 shares /mnt/a-4-shares-brick-4/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-1 shares /mnt/a-1-shares-brick-1/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-1 shares /mnt/a-1-shares-brick-2/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-1 shares /mnt/a-1-shares-brick-3/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > gfs-a-1 shares /mnt/a-1-shares-brick-4/brick gfs-a-bkp::bkpshares Not
>> > Started N/A N/A
>> >
>> > So that's a step in the right direction (and I can upload a patch for
>> > gverify to a bugzilla). However, gverify *should* have worked with
>> bash-c,
>> > and I was not able to figure out why it didn't work, other than it
>> didn't
>> > seem able to find some programs. I'm thinking that maybe the PATH
>> variable
>> > is wrong for Gluster, and that's why gverify didn't work out of the box.
>> >
>> > When I attempt to start geo-rep now, I get the following in the geo-rep
>> > log:
>> >
>> >
>> > [2014-12-07 10:52:40.893594] E
>> > [syncdutils(monitor):218:log_raise_exception] <top>: execution of
>> "gluster"
>> > failed with ENOENT (No such file or directory)
>> >
>> > [2014-12-07 10:52:40.893886] I [syncdutils(monitor):192:finalize] <top>:
>> > exiting.
>> >
>> > Which seems to agree that maybe gluster isn't running with the same path
>> > variable that my console session is running with. Is this possible? I
>> know
>> > I'm grasping :).
>> >
>> > Any nudge in the right direction would be very much appreciated!
>> >
>> > Cheers,
>> > Dave
>> >
>> >
>> > On Sat, Dec 6, 2014 at 10:06 AM, David Gibbons <
>> david.c.gibbons at gmail.com
>> > > wrote:
>> >
>> >
>> >
>> > Good Morning,
>> >
>> > I am having some trouble getting geo-replication started on a 3.5.3
>> volume.
>> >
>> > I have verified that password-less SSH is functional in both directions
>> > from the backup gluster server, and all nodes in the production
>> gluster. I
>> > have verified that all nodes in production and backup cluster are
>> running
>> > the same version of gluster, and that name resolution works in both
>> > directions.
>> >
>> > When I attempt to start geo-replication with this command:
>> >
>> >
>> > gluster volume geo-replication shares gfs-a-bkp::bkpshares create
>> push-pem
>> >
>> > I end up with the following in the logs:
>> >
>> >
>> > [2014-12-06 15:02:50.284426] E
>> > [glusterd-geo-rep.c:1889:glusterd_verify_slave] 0-: Not a valid slave
>> >
>> > [2014-12-06 15:02:50.284495] E
>> > [glusterd-geo-rep.c:2106:glusterd_op_stage_gsync_create] 0-:
>> > gfs-a-bkp::bkpshares is not a valid slave volume. Error: Unable to fetch
>> > master volume details. Please check the master cluster and master
>> volume.
>> >
>> > [2014-12-06 15:02:50.284509] E [glusterd-syncop.c:912:gd_stage_op_phase]
>> > 0-management: Staging of operation 'Volume Geo-replication Create'
>> failed
>> > on localhost : Unable to fetch master volume details. Please check the
>> > master cluster and master volume.
>> >
>> > Would someone be so kind as to point me in the right direction?
>> >
>> > Cheers,
>> > Dave
>> >
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> >
>>
>
>