[Gluster-users] geo-replication {error=12} on one primary node
Stefan Kania
stefan at kania-online.de
Tue Feb 13 19:11:39 UTC 2024
Hi to all,
Yes, I saw that there is a thread about geo-replication with nearly the
same problem, I read it, but I think my problem is a bit different.
I created two volumes the primary volume "privol01" and the secondary
volume "secvol01". All hosts are having the same packages installed, all
hosts are debian12 with gluster version 10.05. So even rsync is the
same on any of the hosts. (I installed one host (vm) and clone it).
I have:
Volume Name: privol01
Type: Replicate
Volume ID: 93ace064-2862-41fe-9606-af5a4af9f5ab
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: p01:/gluster/brick
Brick2: p02:/gluster/brick
Brick3: p03:/gluster/brick
and:
Volume Name: secvol01
Type: Replicate
Volume ID: 4ebb7768-51da-446c-a301-dc3ea49a9ba2
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: s01:/gluster/brick
Brick2: s02:/gluster/brick
Brick3: s03:/gluster/brick
resolving the names of the hosts is working in any direction
that's what I did:
on all secondary hosts:
groupadd geogruppe
useradd -G geogruppe -m geobenutzer
passwd geobenutzer
ln -s /usr/sbin/gluster /usr/bin
on one of the secondary hosts:
gluster-mountbroker setup /var/mountbroker geogruppe
gluster-mountbroker add secvol01 geobenutzer
on one of the primary hosts:
ssh-keygen
ssh-copy-id geobenutzer at s01.gluster
gluster-georep-sshkey generate
gluster v geo-replication privol01 geobenutzer at s01.gluster::secvol01
create push-pem
on one of the secondary hosts:
/usr/libexec/glusterfs/set_geo_rep_pem_keys.sh
All the commands exited with out an error message.
Restarted glusterd on all nodes
then on the primary host:
gluster volume geo-replication privol01
geobenutzer at s01.gluster::secvol01 start
The status is showing:
PRIMARY NODE PRIMARY VOL PRIMARY BRICK SECONDARY USER
SECONDARY SECONDARY NODE STATUS CRAWL
STATUS LAST_SYNCED
---------------------------------------------------------------------------------------------------------------------------------------------------------------
p03 privol01 /gluster/brick geobenutzer
geobenutzer at s01.gluster::secvol01 Passive N/A
N/A
p02 privol01 /gluster/brick geobenutzer
geobenutzer at s01.gluster::secvol01 Passive N/A
N/A
p01 privol01 /gluster/brick geobenutzer
geobenutzer at s01.gluster::secvol01 N/A Faulty N/A
N/A
For p01 the status is changing from "Initializing... to" "status=Active
status=History Crawl" to status=Faulty and then back to Initializing
But only for the primary host p01.
Here is the lock from p01:
--------------------------------
[2024-02-13 18:30:06.64585] I
[gsyncdstatus(monitor):247:set_worker_status] GeorepStatus: Worker
Status Change [{status=Initializing...}]
[2024-02-13 18:30:06.65004] I [monitor(monitor):158:monitor] Monitor:
starting gsyncd worker [{brick=/gluster/brick}, {secondary_node=s01}]
[2024-02-13 18:30:06.147194] I [resource(worker
/gluster/brick):1387:connect_remote] SSH: Initializing SSH connection
between primary and secondary...
[2024-02-13 18:30:07.777785] I [resource(worker
/gluster/brick):1435:connect_remote] SSH: SSH connection between primary
and secondary established. [{duration=1.6304}]
[2024-02-13 18:30:07.777971] I [resource(worker
/gluster/brick):1116:connect] GLUSTER: Mounting gluster volume locally...
[2024-02-13 18:30:08.822077] I [resource(worker
/gluster/brick):1138:connect] GLUSTER: Mounted gluster volume
[{duration=1.0438}]
[2024-02-13 18:30:08.823039] I [subcmds(worker
/gluster/brick):84:subcmd_worker] <top>: Worker spawn successful.
Acknowledging back to monitor
[2024-02-13 18:30:10.861742] I [primary(worker
/gluster/brick):1661:register] _GPrimary: Working dir
[{path=/var/lib/misc/gluster/gsyncd/privol01_s01.gluster_secvol01/gluster-brick}]
[2024-02-13 18:30:10.864432] I [resource(worker
/gluster/brick):1291:service_loop] GLUSTER: Register time
[{time=1707849010}]
[2024-02-13 18:30:10.906805] I [gsyncdstatus(worker
/gluster/brick):280:set_active] GeorepStatus: Worker Status Change
[{status=Active}]
[2024-02-13 18:30:11.7656] I [gsyncdstatus(worker
/gluster/brick):252:set_worker_crawl_status] GeorepStatus: Crawl Status
Change [{status=History Crawl}]
[2024-02-13 18:30:11.7984] I [primary(worker /gluster/brick):1572:crawl]
_GPrimary: starting history crawl [{turns=1}, {stime=(1707848760, 0)},
{etime=1707849011}, {entry_stime=None}]
[2024-02-13 18:30:12.9234] I [primary(worker /gluster/brick):1604:crawl]
_GPrimary: secondary's time [{stime=(1707848760, 0)}]
[2024-02-13 18:30:12.388528] I [primary(worker
/gluster/brick):2009:syncjob] Syncer: Sync Time Taken [{job=2},
{num_files=2}, {return_code=12}, {duration=0.0520}]
[2024-02-13 18:30:12.388745] E [syncdutils(worker
/gluster/brick):845:errlog] Popen: command returned error [{cmd=rsync
-aR0 --inplace --files-from=- --super --stats --numeric-ids
--no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e
ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto
-S /tmp/gsyncd-aux-ssh-1_kow1tp/c343d8e67535166a0d66b71865f3f3c4.sock
geobenutzer at s01:/proc/2675/cwd}, {error=12}]
[2024-02-13 18:30:12.826546] I [monitor(monitor):227:monitor] Monitor:
worker died in startup phase [{brick=/gluster/brick}]
[2024-02-13 18:30:12.845687] I
[gsyncdstatus(monitor):247:set_worker_status] GeorepStatus: Worker
Status Change [{status=Faulty}]
---------------------
The host p01 is trying to connect to s01
A look at host p02 of the primary volume is showing:
-------------------
[2024-02-13 18:25:55.179385] I
[gsyncdstatus(monitor):247:set_worker_status] GeorepStatus: Worker
Status Change [{status=Initializing...}]
[2024-02-13 18:25:55.179572] I [monitor(monitor):158:monitor] Monitor:
starting gsyncd worker [{brick=/gluster/brick}, {secondary_node=s01}]
[2024-02-13 18:25:55.258658] I [resource(worker
/gluster/brick):1387:connect_remote] SSH: Initializing SSH connection
between primary and secondary...
[2024-02-13 18:25:57.78159] I [resource(worker
/gluster/brick):1435:connect_remote] SSH: SSH connection between primary
and secondary established. [{duration=1.8194}]
[2024-02-13 18:25:57.78254] I [resource(worker
/gluster/brick):1116:connect] GLUSTER: Mounting gluster volume locally...
[2024-02-13 18:25:58.123291] I [resource(worker
/gluster/brick):1138:connect] GLUSTER: Mounted gluster volume
[{duration=1.0450}]
[2024-02-13 18:25:58.123410] I [subcmds(worker
/gluster/brick):84:subcmd_worker] <top>: Worker spawn successful.
Acknowledging back to monitor
[2024-02-13 18:26:00.135934] I [primary(worker
/gluster/brick):1661:register] _GPrimary: Working dir
[{path=/var/lib/misc/gluster/gsyncd/privol01_s01.gluster_secvol01/gluster-brick}]
[2024-02-13 18:26:00.136287] I [resource(worker
/gluster/brick):1291:service_loop] GLUSTER: Register time
[{time=1707848760}]
[2024-02-13 18:26:00.179157] I [gsyncdstatus(worker
/gluster/brick):286:set_passive] GeorepStatus: Worker Status Change
[{status=Passive}]
------------------
This is primary node is also connecting to s01 and it works.
It must have something to do with the primary host, because if I stop
the replication and restart it, the primary host is triying to connect
to a different secondary host with the same error:
----------------
Popen: command returned error [{cmd=rsync -aR0 --inplace --files-from=-
--super --stats --numeric-ids --no-implied-dirs --existing --xattrs
--acls --ignore-missing-args . -e ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto
-S /tmp/gsyncd-aux-ssh-1_kow1tp/c343d8e67535166a0d66b71865f3f3c4.sock
geobenutzer at s01:/proc/2675/cwd}, {error=12}]
----------------
So the problem must be the primary host p01. That's the host I
configured the passwordless ssh-session.
This is is test-setup I also tried it before with two other volumes with
6 Nodes each. There I had 2 faulty nodes in the primary volume.
I can start and stop the replication session from any of the primary
nodes but always p01 is faulty.
Any help ?
Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3477 bytes
Desc: Kryptografische S/MIME-Signatur
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240213/d43a1a5d/attachment.p7s>
More information about the Gluster-users
mailing list