<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Hi Shwetha, <br>
<br>
I ran the script against against the first brick in my cluster
(after shutting gluster down) and ran into a small issue: <br>
<font face="monospace"><br>
[root@storage01 ~]# ./glusterfs-georep-upgrade.py
/data/storage_a/storage/ 2>&1 | tee -a
/var/log/storage_a.txt<br>
Traceback (most recent call last):<br>
File "./glusterfs-georep-upgrade.py", line 77, in <module><br>
modify_htime_file(args.brick_path)<br>
File "./glusterfs-georep-upgrade.py", line 61, in
modify_htime_file<br>
shutil.copyfile(pth, os.path.join(path, changelog))<br>
File "/usr/lib64/python3.6/shutil.py", line 104, in copyfile<br>
raise SameFileError("{!r} and {!r} are the same
file".format(src, dst))<br>
shutil.SameFileError:
'/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474'
and
'/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474'
are the same file</font><br>
<br>
Since this script wasn't run prior to the upgrade some of the files
in the original HTIME file are already in the year/month/date format
- so the copy operation fails on those files terminating the loop
early, meaning that not all of the files make it into the
temp_htime_file, and the new htime file doesn't get put into place.
<br>
<font face="monospace"><br>
[root@storage01 changelogs]# tail -c 256 htime/HTIME.1597342860 <br>
/changelog.1616431368/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431382/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431396/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431410[root@storage01
changelogs]# <br>
[root@storage01 changelogs]# <br>
[root@storage01 changelogs]# tail -c 256 htime/temp_htime_file <br>
/changelog.1615317943/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/changelog.1615317957/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/changelog.1615317972/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474[root@storage01
changelogs]#<br>
<br>
[root@storage01 changelogs]# ls -lh<br>
total 4.0K<br>
drw-------. 7 root root 80 Mar 22 10:07 2020<br>
drw-------. 5 root root 52 Mar 22 10:07 2021<br>
-rw-r--r--. 1 root root 51 Mar 22 09:43 CHANGELOG<br>
drw-------. 2 root root 10 May 7 2020 csnap<br>
drw-------. 2 root root 65 Mar 22 09:57 htime<br>
[root@storage01 changelogs]# ls -lh 2021/03/<br>
total 24K<br>
drw-------. 2 root root 16K Mar 22 10:07 01<br>
drw-------. 2 root root 74 Mar 9 12:19 09<br>
drw-------. 2 root root 42 Mar 15 14:08 15<br>
drw-------. 2 root root 42 Mar 16 12:04 16<br>
[root@storage01 changelogs]# ls -lh htime/<br>
total 178M<br>
-rw-r--r--. 1 root root 86M Mar 22 09:43 HTIME.1597342860<br>
-rw-r--r--. 1 root root 93M Mar 22 10:07 temp_htime_file<br>
</font><br>
<br>
I assume here I can just manually add the remaining changelog files
into the temp_htime_file and then perform the backup and
replacement, but wanted to double check. <br>
<br>
It's also strange that there are files listed in the original
HTIME.1597342860 file which don't exist on the filesystem.... the
directories end at 2021/03/16, but there are files listed in the
2021/03/22 . Please advise. <br>
<br>
<font face="monospace">[root@storage01 changelogs]# find 2021/03/
-type f | egrep -v "^2021/03/01"<br>
2021/03/09/CHANGELOG.1615318474<br>
2021/03/09/CHANGELOG.1615321197<br>
2021/03/15/CHANGELOG.1615842490<br>
2021/03/16/CHANGELOG.1615921482<br>
<br>
[root@storage01 changelogs]# tail -c 2048 htime/HTIME.1597342860 <br>
hangelog.1616431028/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431043/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431058/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431073/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431088/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431103/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431118/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431133/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431148/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431163/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431178/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431193/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431208/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431223/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431238/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431253/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431268/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431283/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431298/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431312/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431326/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431340/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431354/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431368/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431382/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431396/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431410</font><br>
<br>
<br>
Thanks,<br>
-Matthew<br>
<div class="moz-signature"><font size="-1">
<p>--<br>
<br>
</p>
</font>
</div>
<div class="moz-cite-prefix">On 3/19/21 5:36 AM, Shwetha Acharya
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAERh03qtOiZg_hLAZtGjZ0Y6QbK-4R38HTf_=hs4iiUNzotCyQ@mail.gmail.com">
<div>
Notice: This message was sent from outside the University of
Victoria email system. Please be cautious with links and
sensitive information.
</div>
<br>
<div>
<div dir="ltr"> <a class="gmail_plusreply" id="plusReplyChip-1"
href="mailto:kaleb@redhat.com" tabindex="-1"
moz-do-not-send="true">@Kaleb Keithley</a> <a
class="gmail_plusreply" id="plusReplyChip-2"
href="mailto:khiremat@redhat.com" tabindex="-1"
moz-do-not-send="true">@Kotresh Hiremath Ravishankar</a>
any insights on use of rpm scriplets for this script?
Basically this script is to be run before the upgradation.<br>
<br>
Strahil,<br>
I will update the docs once we decide with respect to rpm
scriptlet part.<br>
<br>
Regrads,<br>
Shwetha <br>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Mar 19, 2021 at 5:45
PM Strahil Nikolov <<a
href="mailto:hunter86_bg@yahoo.com" moz-do-not-send="true">hunter86_bg@yahoo.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote">
Hi Shwetha,
<div><br>
</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616155936663">Is
that script mentioned in the documentation (release notes
for example) ?</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156103839"><br>
</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616155936803">Do
you think that the new glusterfs rpm can call (via rpm
scriplets) the script safely ?</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156033372"><br>
</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004182"><br>
</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004390">If
yes, then it will be nice if the rpm itself runs the
script instead of the user.</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004390"><br>
</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156077707"><br>
</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156077811">Best
Regards,</div>
<div
id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156082780">Strahil
Nikolov<br>
<br>
<blockquote>
<div>
<div>On Fri, Mar 19, 2021 at 11:44, Shwetha Acharya</div>
<div><<a href="mailto:sacharya@redhat.com"
target="_blank" moz-do-not-send="true">sacharya@redhat.com</a>>
wrote:</div>
</div>
<div>
<div id="gmail-m_634331480037335435yiv1091814412">
<div dir="ltr">Hi Mathew,<br>
<br>
The upgrade script <a rel="nofollow noopener
noreferrer"
href="https://github.com/gluster/glusterfs/commit/2857fe3fad4d2b30894847088a54b847b88a23b9"
target="_blank" moz-do-not-send="true">https://github.com/gluster/glusterfs/commit/2857fe3fad4d2b30894847088a54b847b88a23b9</a>
need to be run, to make sure that the changelogs
and htimes files created in the older version are
updated as per the new changes in the directory
structure.<br>
<br>
If not done, search algorithm during history crawl
would give wrong result.<br>
<br>
Regards,<br>
Shwetha</div>
<br>
<div>
<div dir="ltr">On Fri, Mar 19, 2021 at 3:53 AM
Strahil Nikolov <<a rel="nofollow noopener
noreferrer"
href="mailto:hunter86_bg@yahoo.com"
target="_blank" moz-do-not-send="true">hunter86_bg@yahoo.com</a>>
wrote:<br>
</div>
<blockquote>
Sadly,
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048432"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048546">I'm
out of ideas. It makes sense... if the
changelog was changed - then it won't work
after the upgrade.</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048546"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106093562">I
guess that once you delete the session, you
can remove the extended attribute for the
time.</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106136564"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106136756">Best
Regards,</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106141079">Strahil
Nikolov</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106094445"><br>
<blockquote>
<div>
<div>On Wed, Mar 17, 2021 at 23:11,
Matthew Benstead</div>
<div><<a rel="nofollow noopener
noreferrer"
href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true">matthewb@uvic.ca</a>>
wrote:</div>
</div>
<div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
<div>Yes, I've run through everything
short of regenerating the keys and
creating the session again with no
errors. Everything looks ok.
<br>
<br>
But I did notice that the changelog
format had changed, instead of them
being dumped into one directory, they
now seem to be separated in
year/month/day directories...
<br>
<br>
Looks like this change in 8.0: <a
rel="nofollow noopener noreferrer"
shape="rect"
href="https://github.com/gluster/glusterfs/issues/154"
target="_blank"
moz-do-not-send="true">
https://github.com/gluster/glusterfs/issues/154</a><br>
<br>
[root@storage01 changelogs]# ls -lh |
head<br>
total 16G<br>
drw-------. 3 root root 24 Mar 9
11:34 2021<br>
-rw-r--r--. 1 root root 51 Mar 17
13:19 CHANGELOG<br>
-rw-r--r--. 1 root root 13K Aug 13
2020 CHANGELOG.1597343197<br>
-rw-r--r--. 1 root root 51K Aug 13
2020 CHANGELOG.1597343212<br>
-rw-r--r--. 1 root root 86K Aug 13
2020 CHANGELOG.1597343227<br>
-rw-r--r--. 1 root root 99K Aug 13
2020 CHANGELOG.1597343242<br>
-rw-r--r--. 1 root root 69K Aug 13
2020 CHANGELOG.1597343257<br>
-rw-r--r--. 1 root root 69K Aug 13
2020 CHANGELOG.1597343272<br>
-rw-r--r--. 1 root root 72K Aug 13
2020 CHANGELOG.1597343287<br>
[root@storage01 changelogs]# ls -lh |
tail<br>
-rw-r--r--. 1 root root 92 Mar 1
21:33 CHANGELOG.1614663193<br>
-rw-r--r--. 1 root root 92 Mar 1
21:42 CHANGELOG.1614663731<br>
-rw-r--r--. 1 root root 92 Mar 1
21:42 CHANGELOG.1614663760<br>
-rw-r--r--. 1 root root 511 Mar 1
21:47 CHANGELOG.1614664043<br>
-rw-r--r--. 1 root root 536 Mar 1
21:48 CHANGELOG.1614664101<br>
-rw-r--r--. 1 root root 2.8K Mar 1
21:48 CHANGELOG.1614664116<br>
-rw-r--r--. 1 root root 92 Mar 1
22:20 CHANGELOG.1614666061<br>
-rw-r--r--. 1 root root 92 Mar 1
22:29 CHANGELOG.1614666554<br>
drw-------. 2 root root 10 May 7
2020 csnap<br>
drw-------. 2 root root 38 Aug 13
2020 htime<br>
<br>
<br>
[root@storage01 changelogs]# ls -lh
2021/03/09/<br>
total 8.0K<br>
-rw-r--r--. 1 root root 51 Mar 9
11:26 CHANGELOG.1615318474<br>
-rw-r--r--. 1 root root 51 Mar 9
12:19 CHANGELOG.1615321197<br>
[root@storage01 changelogs]# ls -lh
2021/03/15/<br>
total 4.0K<br>
-rw-r--r--. 1 root root 51 Mar 15
13:38 CHANGELOG.1615842490<br>
[root@storage01 changelogs]# ls -lh
2021/03/16<br>
total 4.0K<br>
-rw-r--r--. 1 root root 331 Mar 16
12:04 CHANGELOG.1615921482<br>
<br>
But it looks like the htime file still
records them...<br>
<br>
[root@storage01 changelogs]# ls -lh
htime<br>
total 84M<br>
-rw-r--r--. 1 root root 84M Mar 17
13:31 HTIME.1597342860<br>
<br>
[root@storage01 changelogs]# head -c
256 htime/HTIME.1597342860 <br>
/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342875/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342890/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342904/data/storage_a/storage/.glusterfs/changelogs/changelog[root@storage01
changelogs]# <br>
<br>
[root@storage01 changelogs]# tail -c
256 htime/HTIME.1597342860 <br>
/changelog.1616013484/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013499/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013514/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013529[root@storage01
changelogs]# <br>
<br>
<br>
And there seems to be an xattr for
time in the brick root - presumably
for when changelogs were enabled:
<br>
<br>
[root@storage01 changelogs]# getfattr
-d -m. -e hex /data/storage_a/storage
2>&1 | egrep xtime<br>
trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x60510140000ef317<br>
<br>
<br>
Reading through the changes, it looks
like there is a script to convert from
one format to the other... I didn't
see anything in the release notes for
8.0 about it... this seems like it
could be the fix, and explain why
gluster can't get through the
changelogs.... Thoughts? <br>
<ul>
<li><a rel="nofollow noopener
noreferrer" shape="rect"
href="https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964"
target="_blank"
moz-do-not-send="true">https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964</a></li>
<li><a rel="nofollow noopener
noreferrer" shape="rect"
href="https://review.gluster.org/#/c/glusterfs/+/24121/"
target="_blank"
moz-do-not-send="true">https://review.gluster.org/#/c/glusterfs/+/24121/</a><br>
</li>
</ul>
<br>
Thanks,<br>
-Matthew<br>
<div>
<p>--<br>
Matthew Benstead<br>
System Administrator<br>
<a rel="nofollow noopener
noreferrer" shape="rect"
href="https://pacificclimate.org/"
target="_blank"
moz-do-not-send="true">Pacific
Climate Impacts Consortium</a><br>
University of Victoria, UH1<br>
PO Box 1800, STN CSC<br>
Victoria, BC, V8W 2Y2<br>
Phone: +1-250-721-8432<br>
Email: <a rel="nofollow noopener
noreferrer" shape="rect"
href="mailto:matthewb@uvic.ca"
target="_blank"
moz-do-not-send="true">
matthewb@uvic.ca</a></p>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt01812">
<div>On 3/16/21 9:36 PM, Strahil
Nikolov wrote:<br>
</div>
<blockquote type="cite">
<div>Notice: This message was sent
from outside the University of
Victoria email system. Please be
cautious with links and
sensitive information.
</div>
<br>
<div>Have you verified all steps
for creating the geo-replication
?
<div><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955616619">If
yes , maybe using
"reset-sync-time + delete +
create" makes sense.Keep in
mind that it will take a long
time once the geo-rep is
established again.</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955729476"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955750689"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955750840">Best
Regards,</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955755206">Strahil
Nikolov</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955730266"><br>
<blockquote>
<div>
<div>On Tue, Mar 16, 2021
at 22:34, Matthew
Benstead</div>
<div><a rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:matthewb@uvic.ca"
target="_blank"
moz-do-not-send="true"><matthewb@uvic.ca></a>
wrote:</div>
</div>
<div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
<div>Thanks Strahil, <br>
<br>
I wanted to make sure
the issue wasn't
occurring because
there were no new
changes to sync from
the master volume. So
I created some files
and restarted the
sync, but it had no
effect.
<br>
<br>
[root@storage01 ~]# cd
/storage2/home/test/<br>
[root@storage01 test]#
for nums in
{1,2,3,4,5,6,7,8,9,0};
do touch $nums.txt;
done<br>
<br>
[root@storage01 test]#
gluster volume
geo-replication
storage <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> start<br>
Starting
geo-replication
session between
storage & <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
[root@storage01 test]#
gluster volume
geo-replication status
<br>
<br>
MASTER NODE MASTER
VOL MASTER
BRICK
SLAVE USER
SLAVE
SLAVE NODE
STATUS
CRAWL STATUS
LAST_SYNCED
<br>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
10.0.231.91
storage
/data/storage_a/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
10.0.231.91
storage
/data/storage_c/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
10.0.231.91
storage
/data/storage_b/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
10.0.231.93
storage
/data/storage_c/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
10.0.231.93
storage
/data/storage_b/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
10.0.231.93
storage
/data/storage_a/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
10.0.231.92
storage
/data/storage_b/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
10.0.231.92
storage
/data/storage_a/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
10.0.231.92
storage
/data/storage_c/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A
N/A
<br>
[root@storage01 test]#
gluster volume
geo-replication status
<br>
<br>
MASTER NODE MASTER
VOL MASTER
BRICK
SLAVE USER
SLAVE
SLAVE NODE
STATUS CRAWL
STATUS
LAST_SYNCED
<br>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
10.0.231.91
storage
/data/storage_a/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
10.0.231.91
storage
/data/storage_c/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
10.0.231.91
storage
/data/storage_b/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
10.0.231.93
storage
/data/storage_c/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
10.0.231.93
storage
/data/storage_b/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
10.0.231.93
storage
/data/storage_a/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
10.0.231.92
storage
/data/storage_b/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
10.0.231.92
storage
/data/storage_a/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
10.0.231.92
storage
/data/storage_c/storage
geoaccount <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A
N/A
<br>
[root@storage01 test]#
gluster volume
geo-replication
storage <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> stop<br>
Stopping
geo-replication
session between
storage & <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank"
moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
<br>
Still getting the same
error about the
history crawl failing:
<br>
<br>
[2021-03-16
19:05:05.227677] I
[MSGID: 132035]
[gf-history-changelog.c:837:gf_history_changelog]
0-gfchangelog:
Requesting historical
changelogs
[{start=1614666552},
{end=1615921505}]
<br>
[2021-03-16
19:05:05.227733] I
[MSGID: 132019]
[gf-history-changelog.c:755:gf_changelog_extract_min_max]
0-gfchangelog:
changelogs min max
[{min=1597342860},
{max=1615921502},
{total_changelogs=1300114}]
<br>
[2021-03-16
19:05:05.408567] E
[MSGID: 132009]
[gf-history-changelog.c:941:gf_history_changelog]
0-gfchangelog: wrong
result [{for=end},
{start=1615921502},
{idx=1300113}]
<br>
<br>
<br>
[2021-03-16
19:05:05.228092] I
[resource(worker
/data/storage_c/storage):1292:service_loop]
GLUSTER: Register time
[{time=1615921505}]<br>
[2021-03-16
19:05:05.228626] D
[repce(worker
/data/storage_c/storage):195:push]
RepceClient: call
124117:140500837320448:1615921505.23
keep_alive(None,) ...<br>
[2021-03-16
19:05:05.230076] D
[repce(worker
/data/storage_c/storage):215:__call__]
RepceClient: call
124117:140500837320448:1615921505.23
keep_alive -> 1<br>
[2021-03-16
19:05:05.230693] D
[master(worker
/data/storage_c/storage):540:crawlwrap]
_GMaster: primary
master with volume id
cf94a8f2-324b-40b3-bf72-c3766100ea99 ...<br>
[2021-03-16
19:05:05.237607] I
[gsyncdstatus(worker
/data/storage_c/storage):281:set_active]
GeorepStatus: Worker
Status Change
[{status=Active}]<br>
[2021-03-16
19:05:05.242046] I
[gsyncdstatus(worker
/data/storage_c/storage):253:set_worker_crawl_status]
GeorepStatus: Crawl
Status Change
[{status=History
Crawl}]<br>
[2021-03-16
19:05:05.242450] I
[master(worker
/data/storage_c/storage):1559:crawl]
_GMaster: starting
history crawl
[{turns=1},
{stime=(1614666552,
0)},
{entry_stime=(1614664108,
0)},
{etime=1615921505}]<br>
[2021-03-16
19:05:05.244151] E
[resource(worker
/data/storage_c/storage):1312:service_loop]
GLUSTER: Changelog
History Crawl failed
[{error=[Errno 0]
Success}]<br>
[2021-03-16
19:05:05.394129] E
[resource(worker
/data/storage_a/storage):1312:service_loop]
GLUSTER: Changelog
History Crawl failed
[{error=[Errno 0]
Success}]<br>
[2021-03-16
19:05:05.408759] E
[resource(worker
/data/storage_b/storage):1312:service_loop]
GLUSTER: Changelog
History Crawl failed
[{error=[Errno 0]
Success}]<br>
[2021-03-16
19:05:06.158694] I
[monitor(monitor):228:monitor]
Monitor: worker died
in startup phase
[{brick=/data/storage_a/storage}]<br>
[2021-03-16
19:05:06.163052] I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus: Worker
Status Change
[{status=Faulty}]<br>
[2021-03-16
19:05:06.204464] I
[monitor(monitor):228:monitor]
Monitor: worker died
in startup phase
[{brick=/data/storage_b/storage}]<br>
[2021-03-16
19:05:06.208961] I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus: Worker
Status Change
[{status=Faulty}]<br>
[2021-03-16
19:05:06.220495] I
[monitor(monitor):228:monitor]
Monitor: worker died
in startup phase
[{brick=/data/storage_c/storage}]<br>
[2021-03-16
19:05:06.223947] I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus: Worker
Status Change
[{status=Faulty}]<br>
<br>
I confirmed NTP is
working: <br>
<br>
<br>
pcic-backup02 |
CHANGED | rc=0
>><br>
remote
refid st t when
poll reach delay
offset jitter<br>
==============================================================================<br>
+s216-232-132-95
68.69.221.61 2 u
29 1024 377
24.141 2.457
1.081<br>
*<a rel="nofollow
noopener noreferrer"
href="http://yyz-1.ip.0xt.ca" target="_blank" moz-do-not-send="true">yyz-1.ip.0xt.ca</a>
206.108.0.131 2 u
257 1024 377
57.119 -0.084
5.625<br>
+ip102.ip-198-27
192.168.10.254 2 u
189 1024 377
64.227 -3.012
8.867<br>
<br>
storage03 | CHANGED |
rc=0 >><br>
remote
refid st t when
poll reach delay
offset jitter<br>
==============================================================================<br>
*198.161.203.36
128.233.150.93 2 u
36 1024 377
16.055 -0.381
0.318<br>
+s206-75-147-25.
192.168.10.254 2 u
528 1024 377
23.648 -6.196
4.803<br>
+time.cloudflare
10.69.8.80 3 u
121 1024 377
2.408 0.507 0.791<br>
<br>
storage02 | CHANGED |
rc=0 >><br>
remote
refid st t when
poll reach delay
offset jitter<br>
==============================================================================<br>
*198.161.203.36
128.233.150.93 2 u
918 1024 377
15.952 0.226
0.197<br>
+linuxgeneration
16.164.40.197 2 u
88 1024 377
62.692 -1.160
2.007<br>
+<a rel="nofollow
noopener noreferrer"
href="http://dns3.switch.ca" target="_blank" moz-do-not-send="true">dns3.switch.ca</a>
206.108.0.131 2 u
857 1024 377
27.315 0.778
0.483<br>
<br>
storage01 | CHANGED |
rc=0 >><br>
remote
refid st t when
poll reach delay
offset jitter<br>
==============================================================================<br>
+198.161.203.36
128.233.150.93 2 u
121 1024 377
16.069 1.016
0.195<br>
+<a rel="nofollow
noopener noreferrer"
href="http://zero.gotroot.ca" target="_blank" moz-do-not-send="true">zero.gotroot.ca</a>
30.114.5.31 2 u
543 1024 377
5.106 -2.462 4.923<br>
*<a rel="nofollow
noopener noreferrer"
href="http://ntp3.torix.ca" target="_blank" moz-do-not-send="true">ntp3.torix.ca</a>
.PTP0. 1 u
300 1024 377
54.010 2.421
15.182<br>
<br>
pcic-backup01 |
CHANGED | rc=0
>><br>
remote
refid st t when
poll reach delay
offset jitter<br>
==============================================================================<br>
*<a rel="nofollow
noopener noreferrer"
href="http://dns3.switch.ca" target="_blank" moz-do-not-send="true">dns3.switch.ca</a>
206.108.0.131 2 u
983 1024 377
26.990 0.523
1.389<br>
+<a rel="nofollow
noopener noreferrer"
href="http://dns2.switch.ca" target="_blank" moz-do-not-send="true">dns2.switch.ca</a>
206.108.0.131 2 u
689 1024 377
26.975 -0.257
0.467<br>
+64.ip-54-39-23.
214.176.184.39 2 u
909 1024 377
64.262 -0.604
6.129<br>
<br>
And everything is
working on the same
version of gluster: <br>
<br>
pcic-backup02 |
CHANGED | rc=0
>><br>
glusterfs 8.3<br>
pcic-backup01 |
CHANGED | rc=0
>><br>
glusterfs 8.3<br>
storage02 | CHANGED |
rc=0 >><br>
glusterfs 8.3<br>
storage01 | CHANGED |
rc=0 >><br>
glusterfs 8.3<br>
storage03 | CHANGED |
rc=0 >><br>
glusterfs 8.3<br>
<br>
SSH works, and the
backup user/group is
configured with
mountbroker: <br>
<br>
[root@storage01 ~]#
ssh -i
/root/.ssh/id_rsa <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81"
target="_blank"
moz-do-not-send="true">
geoaccount@10.0.231.81</a> uname -a<br>
Linux pcic-backup01
3.10.0-1160.15.2.el7.x86_64
#1 SMP Wed Feb 3
15:06:38 UTC 2021
x86_64 x86_64 x86_64
GNU/Linux<br>
[root@storage01 ~]#
ssh -i
/root/.ssh/id_rsa <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.82"
target="_blank"
moz-do-not-send="true">
geoaccount@10.0.231.82</a> uname -a<br>
Linux pcic-backup02
3.10.0-1160.15.2.el7.x86_64
#1 SMP Wed Feb 3
15:06:38 UTC 2021
x86_64 x86_64 x86_64
GNU/Linux<br>
<br>
<br>
[root@pcic-backup01
~]# grep geo
/etc/passwd<br>
geoaccount:x:1000:1000::/home/geoaccount:/bin/bash<br>
[root@pcic-backup01
~]# grep geo
/etc/group<br>
geogroup:x:1000:geoaccount<br>
geoaccount:x:1001:geoaccount<br>
<br>
[root@pcic-backup01
~]#
gluster-mountbroker
status<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
| NODE | NODE
STATUS | MOUNT
ROOT |
GROUP |
USERS |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
| 10.0.231.82
| UP |
/var/mountbroker-root(OK)
| geogroup(OK) |
geoaccount(pcic-backup)
|<br>
| localhost
| UP |
/var/mountbroker-root(OK)
| geogroup(OK) |
geoaccount(pcic-backup)
|<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
<br>
<br>
<br>
<br>
So, then if I'm going
to have to resync,
what is the best way
to do this? <br>
<br>
With delete or delete
reset-sync-time ? <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session"
target="_blank"
moz-do-not-send="true">
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session</a><br>
<br>
<br>
<br>
Erasing the index? So
I don't have to
transfer the files
again that are already
on the backup?
<br>
<ul>
<li><a rel="nofollow
noopener
noreferrer"
shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete"
target="_blank"
moz-do-not-send="true">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete</a>
</li>
<li><a rel="nofollow
noopener
noreferrer"
shape="rect"
href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices"
target="_blank"
moz-do-not-send="true">https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices</a><br>
</li>
</ul>
<br>
<br>
<br>
Is it possible to use
the special-sync-mode
option from here: <a
rel="nofollow
noopener noreferrer"
shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery"
target="_blank"
moz-do-not-send="true">
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery</a><br>
<br>
<br>
<br>
Thoughts? <br>
<br>
Thanks,<br>
-Matthew<br>
--<br>
<br>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt93720">
<div>On 3/12/21 3:31
PM, Strahil
Nikolov wrote:<br>
</div>
<blockquote
type="cite">
<div>Notice: This
message was sent
from outside the
University of
Victoria email
system. Please
be cautious with
links and
sensitive
information.
</div>
<br>
<div>Usually, when
I'm stuck - I
just start over.
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591681347">For
example, check
the
prerequisites:</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591695583">-
Is ssh
available (no
firewall
blocking)</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591712278">-
Is time sync
enabled
(ntp/chrony)</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591726239">-
Is DNS ok on
all hosts
(including PTR
records)</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591754957">-
Is the gluster
version the
same on all
nodes (primary
&
secondary)</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591754957"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591782389">Then
start over as
if the geo rep
was never
existing. For
example , stop
it and start
over with the
secondary
nodes's checks
(mountbroker,
user, group) .</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591782389"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591841464">Most
probably
somwthing will
come up and
you will fix
it.</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591841464"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591864006">In
worst case
scenario, you
will need to
clean ip the
geo-rep and
start fresh.</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591864006"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591889862"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591889925">Best
Regards,</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591895692">Strahil
Nikolov<br>
<br>
<blockquote>
<div>
<div>On Fri,
Mar 12, 2021
at 20:01,
Matthew
Benstead</div>
<div><a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true"><matthewb@uvic.ca></a>
wrote:</div>
</div>
<div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
<div>Hi
Strahil, <br>
<br>
Yes, SELinux
was put into
permissive
mode on the
secondary
nodes as well:
<br>
<br>
[root@pcic-backup01 ~]# sestatus | egrep -i "^SELinux status|mode"<br>
SELinux
status:
enabled<br>
Current
mode:
permissive<br>
Mode from
config
file:
enforcing<br>
<br>
[root@pcic-backup02 ~]# sestatus | egrep -i "^SELinux status|mode"<br>
SELinux
status:
enabled<br>
Current
mode:
permissive<br>
Mode from
config
file:
enforcing<br>
<br>
The secondary
server logs
didn't show
anything
interesting: <br>
<br>
gsyncd.log: <br>
<br>
[2021-03-11
19:15:28.81820]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.92/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:28.101819]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.91/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:28.107012]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:28.124567]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_b/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:28.128145]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:29.425739]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3184}]<br>
[2021-03-11
19:15:29.427448]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:29.433340]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_b/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3083}]<br>
[2021-03-11
19:15:29.434452]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.91/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3321}]<br>
[2021-03-11
19:15:29.434314]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_b/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:29.435575]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.91/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:29.439769]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.92/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3576}]<br>
[2021-03-11
19:15:29.440998]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.92/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:29.454745]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3262}]<br>
[2021-03-11
19:15:29.456192]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:32.594865]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.92/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:32.607815]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:32.647663]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_b/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:32.656280]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.91/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:32.668299]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:44.260689]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.92/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:44.271457]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:44.271883]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_b/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:44.279670]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.91/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:44.284261]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
gluster volume
locally...<br>
[2021-03-11
19:15:45.614280]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_b/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3419}]<br>
[2021-03-11
19:15:45.615622]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_b/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:45.617986]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3461}]<br>
[2021-03-11
19:15:45.618180]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.91/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3380}]<br>
[2021-03-11
19:15:45.619539]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.91/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:45.618999]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:45.620843]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3361}]<br>
[2021-03-11
19:15:45.621347]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.92/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
gluster volume
[{duration=1.3604}]<br>
[2021-03-11
19:15:45.622179]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:45.622541]
I
[resource(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.92/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
slave
listening<br>
[2021-03-11
19:15:47.626054]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.91/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:48.778399]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:48.778491]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.92/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:48.796854]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
[2021-03-11
19:15:48.800697]
I [repce(slave
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.93/data/storage_b/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):96:service_loop</a>] RepceServer:
terminating on
reaching EOF.<br>
<br>
The mnt
geo-rep files
were also
uninteresting:
<br>
[2021-03-11
19:15:28.250150]
I [MSGID:
100030]
[glusterfsd.c:2689:main]
0-/usr/sbin/glusterfs: Started running version
[{arg=/usr/sbin/glusterfs},
{version=8.3},
{cmdlinestr=/usr/sbin/glusterfs --user-map-root=g<br>
eoaccount
--aux-gfid-mount
--acl
--log-level=INFO
--log-file=/var/log/glusterfs/geo-replication-slaves/storage_10.0.231.81_pcic-backup/mnt-10.0.231.93-data-storage_b-storage.log
--volfile-server=localhost --volf<br>
ile-id=pcic-backup --client-pid=-1
/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI}]
<br>
[2021-03-11
19:15:28.253485]
I
[glusterfsd.c:2424:daemonize]
0-glusterfs:
Pid of current
running
process is
157484<br>
[2021-03-11
19:15:28.267911]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=0}]
<br>
[2021-03-11
19:15:28.267984]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=1}]
<br>
[2021-03-11
19:15:28.268371]
I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk]
0-glusterfs:
Received list
of available
volfile
servers:
<a
rel="nofollow
noopener
noreferrer"
href="http://10.0.231.82:24007"
target="_blank" moz-do-not-send="true">
10.0.231.82:24007</a> <br>
[2021-03-11
19:15:28.271729]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=2}]
<br>
[2021-03-11
19:15:28.271762]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=3}]
<br>
[2021-03-11
19:15:28.272223]
I [MSGID:
114020]
[client.c:2315:notify]
0-pcic-backup-client-0: parent translators are ready, attempting connect
on transport
[]
<br>
[2021-03-11
19:15:28.275883]
I [MSGID:
114020]
[client.c:2315:notify]
0-pcic-backup-client-1: parent translators are ready, attempting connect
on transport
[]
<br>
[2021-03-11
19:15:28.276154]
I
[rpc-clnt.c:1975:rpc_clnt_reconfig]
0-pcic-backup-client-0: changing port to 49153 (from 0)<br>
[2021-03-11
19:15:28.276193]
I
[socket.c:849:__socket_shutdown]
0-pcic-backup-client-0: intentional socket shutdown(13)<br>
Final graph:<br>
...<br>
+------------------------------------------------------------------------------+<br>
[2021-03-11
19:15:28.282144]
I
[socket.c:849:__socket_shutdown]
0-pcic-backup-client-1: intentional socket shutdown(15)<br>
[2021-03-11
19:15:28.286536]
I [MSGID:
114057]
[client-handshake.c:1128:select_server_supported_programs]
0-pcic-backup-client-0: Using Program [{Program-name=GlusterFS 4.x v1},
{Num=1298437},
{Version=400}]
<br>
[2021-03-11
19:15:28.287208]
I [MSGID:
114046]
[client-handshake.c:857:client_setvolume_cbk]
0-pcic-backup-client-0: Connected, attached to remote volume
[{conn-name=pcic-backup-client-0},
{remote_subvol=/data/brick}]
<br>
[2021-03-11
19:15:28.290162]
I [MSGID:
114057]
[client-handshake.c:1128:select_server_supported_programs]
0-pcic-backup-client-1: Using Program [{Program-name=GlusterFS 4.x v1},
{Num=1298437},
{Version=400}]
<br>
[2021-03-11
19:15:28.291122]
I [MSGID:
114046]
[client-handshake.c:857:client_setvolume_cbk]
0-pcic-backup-client-1: Connected, attached to remote volume
[{conn-name=pcic-backup-client-1},
{remote_subvol=/data/brick}]
<br>
[2021-03-11
19:15:28.292703]
I
[fuse-bridge.c:5300:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24
kernel 7.23<br>
[2021-03-11
19:15:28.292730]
I
[fuse-bridge.c:5926:fuse_graph_sync]
0-fuse:
switched to
graph 0<br>
[2021-03-11
19:15:32.809518]
I
[fuse-bridge.c:6242:fuse_thread_proc]
0-fuse:
initiating
unmount of
/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI<br>
[2021-03-11
19:15:32.810216]
W
[glusterfsd.c:1439:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7ea5) [0x7ff56b175ea5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
[0x55664e67db45] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
[0x55664e67d9ab]
) 0-: received
signum (15),
shutting down
<br>
[2021-03-11
19:15:32.810253]
I
[fuse-bridge.c:7074:fini]
0-fuse:
Unmounting
'/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.<br>
[2021-03-11
19:15:32.810268]
I
[fuse-bridge.c:7079:fini]
0-fuse:
Closing fuse
connection to
'/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.<br>
<br>
<br>
I'm really at
a loss for
where to go
from here, it
seems like
everything is
set up
correctly, and
it has been
working well
through the
7.x minor
versions, but
the jump to 8
has broken
something...<br>
<br>
There
definitely are
lots of
changelogs on
the servers
that fit into
the
timeframe.....
I haven't made
any writes to
the source
volume.... do
you think
that's the
problem? That
it needs some
new changelog
info to sync?
<br>
I had been
holding off
making any
writes in case
I needed to go
back to
Gluster7.9 -
not sure if
that's really
a good option
or not.
<br>
<br>
[root@storage01 changelogs]# for dirs in {a,b,c}; do echo
"/data/storage_$dirs/storage/.glusterfs/changelogs";
ls -lh
/data/storage_$dirs/storage/.glusterfs/changelogs
| head; echo
""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
total 16G<br>
drw-------. 3
root root 24
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:50
CHANGELOG<br>
-rw-r--r--. 1
root root 13K
Aug 13 2020
CHANGELOG.1597343197<br>
-rw-r--r--. 1
root root 51K
Aug 13 2020
CHANGELOG.1597343212<br>
-rw-r--r--. 1
root root 86K
Aug 13 2020
CHANGELOG.1597343227<br>
-rw-r--r--. 1
root root 99K
Aug 13 2020
CHANGELOG.1597343242<br>
-rw-r--r--. 1
root root 69K
Aug 13 2020
CHANGELOG.1597343257<br>
-rw-r--r--. 1
root root 69K
Aug 13 2020
CHANGELOG.1597343272<br>
-rw-r--r--. 1
root root 72K
Aug 13 2020
CHANGELOG.1597343287<br>
<br>
/data/storage_b/storage/.glusterfs/changelogs<br>
total 3.3G<br>
drw-------. 3
root root 24
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:50
CHANGELOG<br>
-rw-r--r--. 1
root root 13K
Aug 13 2020
CHANGELOG.1597343197<br>
-rw-r--r--. 1
root root 53K
Aug 13 2020
CHANGELOG.1597343212<br>
-rw-r--r--. 1
root root 89K
Aug 13 2020
CHANGELOG.1597343227<br>
-rw-r--r--. 1
root root 89K
Aug 13 2020
CHANGELOG.1597343242<br>
-rw-r--r--. 1
root root 69K
Aug 13 2020
CHANGELOG.1597343257<br>
-rw-r--r--. 1
root root 71K
Aug 13 2020
CHANGELOG.1597343272<br>
-rw-r--r--. 1
root root 86K
Aug 13 2020
CHANGELOG.1597343287<br>
<br>
/data/storage_c/storage/.glusterfs/changelogs<br>
total 9.6G<br>
drw-------. 3
root root 16
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:50
CHANGELOG<br>
-rw-r--r--. 1
root root 16K
Aug 13 2020
CHANGELOG.1597343199<br>
-rw-r--r--. 1
root root 71K
Aug 13 2020
CHANGELOG.1597343214<br>
-rw-r--r--. 1
root root 122K
Aug 13 2020
CHANGELOG.1597343229<br>
-rw-r--r--. 1
root root 73K
Aug 13 2020
CHANGELOG.1597343244<br>
-rw-r--r--. 1
root root 100K
Aug 13 2020
CHANGELOG.1597343259<br>
-rw-r--r--. 1
root root 95K
Aug 13 2020
CHANGELOG.1597343274<br>
-rw-r--r--. 1
root root 92K
Aug 13 2020
CHANGELOG.1597343289<br>
<br>
[root@storage01 changelogs]# for dirs in {a,b,c}; do echo
"/data/storage_$dirs/storage/.glusterfs/changelogs";
ls -lh
/data/storage_$dirs/storage/.glusterfs/changelogs
| tail; echo
""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 92
Mar 1 21:33
CHANGELOG.1614663193<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663731<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663760<br>
-rw-r--r--. 1
root root 511
Mar 1 21:47
CHANGELOG.1614664043<br>
-rw-r--r--. 1
root root 536
Mar 1 21:48
CHANGELOG.1614664101<br>
-rw-r--r--. 1
root root 2.8K
Mar 1 21:48
CHANGELOG.1614664116<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666061<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666554<br>
drw-------. 2
root root 10
May 7 2020
csnap<br>
drw-------. 2
root root 38
Aug 13 2020
htime<br>
<br>
/data/storage_b/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663731<br>
-rw-r--r--. 1
root root 480
Mar 1 21:42
CHANGELOG.1614663745<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663760<br>
-rw-r--r--. 1
root root 524
Mar 1 21:47
CHANGELOG.1614664043<br>
-rw-r--r--. 1
root root 495
Mar 1 21:48
CHANGELOG.1614664100<br>
-rw-r--r--. 1
root root 1.6K
Mar 1 21:48
CHANGELOG.1614664114<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666060<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666553<br>
drw-------. 2
root root 10
May 7 2020
csnap<br>
drw-------. 2
root root 38
Aug 13 2020
htime<br>
<br>
/data/storage_c/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663738<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663753<br>
-rw-r--r--. 1
root root 395
Mar 1 21:47
CHANGELOG.1614664051<br>
-rw-r--r--. 1
root root 316
Mar 1 21:48
CHANGELOG.1614664094<br>
-rw-r--r--. 1
root root 1.2K
Mar 1 21:48
CHANGELOG.1614664109<br>
-rw-r--r--. 1
root root 174
Mar 1 21:48
CHANGELOG.1614664123<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666061<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666553<br>
drw-------. 2
root root 6
May 7 2020
csnap<br>
drw-------. 2
root root 30
Aug 13 2020
htime<br>
<br>
[root@storage02 ~]# for dirs in {a,b,c}; do echo
"/data/storage_$dirs/storage/.glusterfs/changelogs";
ls -lh
/data/storage_$dirs/storage/.glusterfs/changelogs
| head; echo
""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
total 9.6G<br>
drw-------. 3
root root 24
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:50
CHANGELOG<br>
-rw-r--r--. 1
root root 4.2K
Aug 13 2020
CHANGELOG.1597343193<br>
-rw-r--r--. 1
root root 32K
Aug 13 2020
CHANGELOG.1597343208<br>
-rw-r--r--. 1
root root 107K
Aug 13 2020
CHANGELOG.1597343223<br>
-rw-r--r--. 1
root root 120K
Aug 13 2020
CHANGELOG.1597343238<br>
-rw-r--r--. 1
root root 72K
Aug 13 2020
CHANGELOG.1597343253<br>
-rw-r--r--. 1
root root 111K
Aug 13 2020
CHANGELOG.1597343268<br>
-rw-r--r--. 1
root root 91K
Aug 13 2020
CHANGELOG.1597343283<br>
<br>
/data/storage_b/storage/.glusterfs/changelogs<br>
total 16G<br>
drw-------. 3
root root 24
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:50
CHANGELOG<br>
-rw-r--r--. 1
root root 3.9K
Aug 13 2020
CHANGELOG.1597343193<br>
-rw-r--r--. 1
root root 35K
Aug 13 2020
CHANGELOG.1597343208<br>
-rw-r--r--. 1
root root 85K
Aug 13 2020
CHANGELOG.1597343223<br>
-rw-r--r--. 1
root root 103K
Aug 13 2020
CHANGELOG.1597343238<br>
-rw-r--r--. 1
root root 70K
Aug 13 2020
CHANGELOG.1597343253<br>
-rw-r--r--. 1
root root 72K
Aug 13 2020
CHANGELOG.1597343268<br>
-rw-r--r--. 1
root root 73K
Aug 13 2020
CHANGELOG.1597343283<br>
<br>
/data/storage_c/storage/.glusterfs/changelogs<br>
total 3.3G<br>
drw-------. 3
root root 16
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:51
CHANGELOG<br>
-rw-r--r--. 1
root root 21K
Aug 13 2020
CHANGELOG.1597343202<br>
-rw-r--r--. 1
root root 75K
Aug 13 2020
CHANGELOG.1597343217<br>
-rw-r--r--. 1
root root 92K
Aug 13 2020
CHANGELOG.1597343232<br>
-rw-r--r--. 1
root root 77K
Aug 13 2020
CHANGELOG.1597343247<br>
-rw-r--r--. 1
root root 66K
Aug 13 2020
CHANGELOG.1597343262<br>
-rw-r--r--. 1
root root 84K
Aug 13 2020
CHANGELOG.1597343277<br>
-rw-r--r--. 1
root root 81K
Aug 13 2020
CHANGELOG.1597343292<br>
<br>
[root@storage02 ~]# for dirs in {a,b,c}; do echo
"/data/storage_$dirs/storage/.glusterfs/changelogs";
ls -lh
/data/storage_$dirs/storage/.glusterfs/changelogs
| tail; echo
""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663734<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663749<br>
-rw-r--r--. 1
root root 395
Mar 1 21:47
CHANGELOG.1614664052<br>
-rw-r--r--. 1
root root 316
Mar 1 21:48
CHANGELOG.1614664096<br>
-rw-r--r--. 1
root root 1.2K
Mar 1 21:48
CHANGELOG.1614664111<br>
-rw-r--r--. 1
root root 174
Mar 1 21:48
CHANGELOG.1614664126<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666056<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666560<br>
drw-------. 2
root root 10
May 7 2020
csnap<br>
drw-------. 2
root root 38
Aug 13 2020
htime<br>
<br>
/data/storage_b/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663735<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663749<br>
-rw-r--r--. 1
root root 511
Mar 1 21:47
CHANGELOG.1614664052<br>
-rw-r--r--. 1
root root 316
Mar 1 21:48
CHANGELOG.1614664096<br>
-rw-r--r--. 1
root root 1.8K
Mar 1 21:48
CHANGELOG.1614664111<br>
-rw-r--r--. 1
root root 1.4K
Mar 1 21:48
CHANGELOG.1614664126<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666060<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666556<br>
drw-------. 2
root root 10
May 7 2020
csnap<br>
drw-------. 2
root root 38
Aug 13 2020
htime<br>
<br>
/data/storage_c/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663738<br>
-rw-r--r--. 1
root root 521
Mar 1 21:42
CHANGELOG.1614663752<br>
-rw-r--r--. 1
root root 524
Mar 1 21:47
CHANGELOG.1614664042<br>
-rw-r--r--. 1
root root 92
Mar 1 21:47
CHANGELOG.1614664057<br>
-rw-r--r--. 1
root root 536
Mar 1 21:48
CHANGELOG.1614664102<br>
-rw-r--r--. 1
root root 1.6K
Mar 1 21:48
CHANGELOG.1614664117<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666057<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666550<br>
drw-------. 2
root root 6
May 7 2020
csnap<br>
drw-------. 2
root root 30
Aug 13 2020
htime<br>
<br>
<br>
[root@storage03 ~]# for dirs in {a,b,c}; do echo
"/data/storage_$dirs/storage/.glusterfs/changelogs";
ls -lh
/data/storage_$dirs/storage/.glusterfs/changelogs
| head; echo
""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
total 3.4G<br>
drw-------. 3
root root 24
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:50
CHANGELOG<br>
-rw-r--r--. 1
root root 19K
Aug 13 2020
CHANGELOG.1597343201<br>
-rw-r--r--. 1
root root 66K
Aug 13 2020
CHANGELOG.1597343215<br>
-rw-r--r--. 1
root root 91K
Aug 13 2020
CHANGELOG.1597343230<br>
-rw-r--r--. 1
root root 82K
Aug 13 2020
CHANGELOG.1597343245<br>
-rw-r--r--. 1
root root 64K
Aug 13 2020
CHANGELOG.1597343259<br>
-rw-r--r--. 1
root root 75K
Aug 13 2020
CHANGELOG.1597343274<br>
-rw-r--r--. 1
root root 81K
Aug 13 2020
CHANGELOG.1597343289<br>
<br>
/data/storage_b/storage/.glusterfs/changelogs<br>
total 9.6G<br>
drw-------. 3
root root 24
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:51
CHANGELOG<br>
-rw-r--r--. 1
root root 19K
Aug 13 2020
CHANGELOG.1597343201<br>
-rw-r--r--. 1
root root 80K
Aug 13 2020
CHANGELOG.1597343215<br>
-rw-r--r--. 1
root root 119K
Aug 13 2020
CHANGELOG.1597343230<br>
-rw-r--r--. 1
root root 65K
Aug 13 2020
CHANGELOG.1597343244<br>
-rw-r--r--. 1
root root 100K
Aug 13 2020
CHANGELOG.1597343259<br>
-rw-r--r--. 1
root root 95K
Aug 13 2020
CHANGELOG.1597343274<br>
-rw-r--r--. 1
root root 92K
Aug 13 2020
CHANGELOG.1597343289<br>
<br>
/data/storage_c/storage/.glusterfs/changelogs<br>
total 16G<br>
drw-------. 3
root root 16
Mar 9 11:34
2021<br>
-rw-r--r--. 1
root root 51
Mar 12 09:51
CHANGELOG<br>
-rw-r--r--. 1
root root 3.9K
Aug 13 2020
CHANGELOG.1597343193<br>
-rw-r--r--. 1
root root 35K
Aug 13 2020
CHANGELOG.1597343208<br>
-rw-r--r--. 1
root root 85K
Aug 13 2020
CHANGELOG.1597343223<br>
-rw-r--r--. 1
root root 103K
Aug 13 2020
CHANGELOG.1597343238<br>
-rw-r--r--. 1
root root 70K
Aug 13 2020
CHANGELOG.1597343253<br>
-rw-r--r--. 1
root root 71K
Aug 13 2020
CHANGELOG.1597343268<br>
-rw-r--r--. 1
root root 73K
Aug 13 2020
CHANGELOG.1597343283<br>
<br>
[root@storage03 ~]# for dirs in {a,b,c}; do echo
"/data/storage_$dirs/storage/.glusterfs/changelogs";
ls -lh
/data/storage_$dirs/storage/.glusterfs/changelogs
| tail; echo
""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 92
Mar 1 21:33
CHANGELOG.1614663183<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663740<br>
-rw-r--r--. 1
root root 521
Mar 1 21:42
CHANGELOG.1614663755<br>
-rw-r--r--. 1
root root 524
Mar 1 21:47
CHANGELOG.1614664049<br>
-rw-r--r--. 1
root root 1.9K
Mar 1 21:48
CHANGELOG.1614664106<br>
-rw-r--r--. 1
root root 174
Mar 1 21:48
CHANGELOG.1614664121<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666051<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666559<br>
drw-------. 2
root root 10
May 7 2020
csnap<br>
drw-------. 2
root root 38
Aug 13 2020
htime<br>
<br>
/data/storage_b/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 474
Mar 1 21:33
CHANGELOG.1614663182<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663739<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663753<br>
-rw-r--r--. 1
root root 395
Mar 1 21:47
CHANGELOG.1614664049<br>
-rw-r--r--. 1
root root 1.4K
Mar 1 21:48
CHANGELOG.1614664106<br>
-rw-r--r--. 1
root root 174
Mar 1 21:48
CHANGELOG.1614664120<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666063<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666557<br>
drw-------. 2
root root 10
May 7 2020
csnap<br>
drw-------. 2
root root 38
Aug 13 2020
htime<br>
<br>
/data/storage_c/storage/.glusterfs/changelogs<br>
-rw-r--r--. 1
root root 468
Mar 1 21:33
CHANGELOG.1614663183<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663740<br>
-rw-r--r--. 1
root root 92
Mar 1 21:42
CHANGELOG.1614663754<br>
-rw-r--r--. 1
root root 511
Mar 1 21:47
CHANGELOG.1614664048<br>
-rw-r--r--. 1
root root 2.0K
Mar 1 21:48
CHANGELOG.1614664105<br>
-rw-r--r--. 1
root root 1.4K
Mar 1 21:48
CHANGELOG.1614664120<br>
-rw-r--r--. 1
root root 92
Mar 1 22:20
CHANGELOG.1614666063<br>
-rw-r--r--. 1
root root 92
Mar 1 22:29
CHANGELOG.1614666556<br>
drw-------. 2
root root 6
May 7 2020
csnap<br>
drw-------. 2
root root 30
Aug 13 2020
htime<br>
<br>
Thanks,<br>
-Matthew<br>
<div>
<p>--<br>
Matthew
Benstead<br>
System
Administrator<br>
<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="https://pacificclimate.org/"
target="_blank" moz-do-not-send="true">Pacific Climate Impacts
Consortium</a><br>
University of
Victoria, UH1<br>
PO Box 1800,
STN CSC<br>
Victoria, BC,
V8W 2Y2<br>
Phone:
+1-250-721-8432<br>
Email: <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true">
matthewb@uvic.ca</a></p>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt16014">
<div>On
3/11/21 11:37
PM, Strahil
Nikolov wrote:<br>
</div>
<blockquote
type="cite">
<div>Notice:
This message
was sent from
outside the
University of
Victoria email
system. Please
be cautious
with links and
sensitive
information.
</div>
<br>
<div>Have you
checked the
secondary
volume nodes'
logs &
SELINUX status
?
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534613030"><br>
</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534613170">Best
Regards,</div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534618885">Strahil
Nikolov<br>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534604587"><br>
<blockquote>
<div>
<div>On Thu,
Mar 11, 2021
at 21:36,
Matthew
Benstead</div>
<div><a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true"><matthewb@uvic.ca></a>
wrote:</div>
</div>
<div>
<div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
<div>Hi
Strahil, <br>
<br>
It looks like
perhaps the
changelog_log_level
and log_level
options? I've
set them to
debug:
<br>
<br>
[root@storage01 ~]# gluster volume geo-replication storage <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config | egrep -i "log_level"<br>
changelog_log_level:INFO<br>
cli_log_level:INFO<br>
gluster_log_level:INFO<br>
log_level:INFO<br>
slave_gluster_log_level:INFO<br>
slave_log_level:INFO<br>
<br>
[root@storage01 ~]# gluster volume geo-replication storage <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config changelog_log_level DEBUG<br>
geo-replication config updated successfully<br>
<br>
[root@storage01 ~]# gluster volume geo-replication storage <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config log_level DEBUG<br>
geo-replication config updated successfully<br>
<br>
<br>
Then I
restarted
geo-replication:
<br>
<br>
[root@storage01 ~]# gluster volume geo-replication storage <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> start<br>
Starting
geo-replication
session
between
storage &
<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
[root@storage01 ~]# gluster volume geo-replication status <br>
<br>
MASTER NODE
MASTER VOL
MASTER
BRICK
SLAVE USER
SLAVE SLAVE NODE
STATUS
CRAWL
STATUS
LAST_SYNCED
<br>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
10.0.231.91
storage
/data/storage_a/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
10.0.231.91
storage
/data/storage_c/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
10.0.231.91
storage
/data/storage_b/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
10.0.231.92
storage
/data/storage_b/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
10.0.231.92
storage
/data/storage_a/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
10.0.231.92
storage
/data/storage_c/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
10.0.231.93
storage
/data/storage_c/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
10.0.231.93
storage
/data/storage_b/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
10.0.231.93
storage
/data/storage_a/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A
Initializing...
N/A N/A
<br>
[root@storage01 ~]# gluster volume geo-replication status <br>
<br>
MASTER NODE
MASTER VOL
MASTER
BRICK
SLAVE USER
SLAVE SLAVE NODE STATUS
CRAWL
STATUS
LAST_SYNCED
<br>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
10.0.231.91
storage
/data/storage_a/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
10.0.231.91
storage
/data/storage_c/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
10.0.231.91
storage
/data/storage_b/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
10.0.231.92
storage
/data/storage_b/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
10.0.231.92
storage
/data/storage_a/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
10.0.231.92
storage
/data/storage_c/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
10.0.231.93
storage
/data/storage_c/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
10.0.231.93
storage
/data/storage_b/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
10.0.231.93
storage
/data/storage_a/storage geoaccount <a rel="nofollow noopener
noreferrer"
shape="rect"
href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty
N/A N/A
<br>
<br>
[root@storage01 ~]# gluster volume geo-replication storage <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> stop<br>
Stopping
geo-replication
session
between
storage &
<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
<br>
<br>
The changelogs
didn't really
show anything
new around
changelog
selection: <br>
<br>
[root@storage01 storage_10.0.231.81_pcic-backup]# cat
changes-data-storage_a-storage.log
| egrep
"2021-03-11"<br>
[2021-03-11
19:15:30.552889]
I [MSGID:
132028]
[gf-changelog.c:577:gf_changelog_register_generic]
0-gfchangelog:
Registering
brick
[{brick=/data/storage_a/storage},
{notify_filter=1}]
<br>
[2021-03-11
19:15:30.552893]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=0}]
<br>
[2021-03-11
19:15:30.552894]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=1}]
<br>
[2021-03-11
19:15:30.553633]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=3}]
<br>
[2021-03-11
19:15:30.553634]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=2}]
<br>
[2021-03-11
19:15:30.554236]
D
[rpcsvc.c:2831:rpcsvc_init]
0-rpc-service:
RPC service
inited.<br>
[2021-03-11
19:15:30.554403]
D
[rpcsvc.c:2342:rpcsvc_program_register]
0-rpc-service:
New program
registered:
GF-DUMP, Num:
123451501,
Ver: 1, Port:
0<br>
[2021-03-11
19:15:30.554420]
D
[rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
[2021-03-11
19:15:30.554933]
D
[socket.c:4485:socket_init]
0-socket.gfchangelog: disabling nodelay<br>
[2021-03-11
19:15:30.554944]
D
[socket.c:4523:socket_init]
0-socket.gfchangelog: Configured transport.tcp-user-timeout=42<br>
[2021-03-11
19:15:30.554949]
D
[socket.c:4543:socket_init]
0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9<br>
[2021-03-11
19:15:30.555002]
I
[socket.c:929:__socket_server_bind]
0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23<br>
[2021-03-11
19:15:30.555324]
D
[rpcsvc.c:2342:rpcsvc_program_register]
0-rpc-service:
New program
registered:
LIBGFCHANGELOG
REBORP, Num:
1886350951,
Ver: 1, Port:
0<br>
[2021-03-11
19:15:30.555345]
D
[rpc-clnt.c:1020:rpc_clnt_connection_init]
0-gfchangelog:
defaulting
frame-timeout
to 30mins<br>
[2021-03-11
19:15:30.555351]
D
[rpc-clnt.c:1032:rpc_clnt_connection_init]
0-gfchangelog:
disable
ping-timeout<br>
[2021-03-11
19:15:30.555358]
D
[rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
[2021-03-11
19:15:30.555399]
D
[socket.c:4485:socket_init]
0-gfchangelog:
disabling
nodelay<br>
[2021-03-11
19:15:30.555406]
D
[socket.c:4523:socket_init]
0-gfchangelog:
Configured
transport.tcp-user-timeout=42<br>
[2021-03-11
19:15:32.555711]
D
[rpc-clnt-ping.c:298:rpc_clnt_start_ping]
0-gfchangelog:
ping timeout
is 0,
returning<br>
[2021-03-11
19:15:32.572157]
I [MSGID:
132035]
[gf-history-changelog.c:837:gf_history_changelog]
0-gfchangelog:
Requesting
historical
changelogs
[{start=1614666553},
{end=1615490132}]
<br>
[2021-03-11
19:15:32.572436]
I [MSGID:
132019]
[gf-history-changelog.c:755:gf_changelog_extract_min_max]
0-gfchangelog:
changelogs min
max
[{min=1597342860},
{max=1615490121}, {total_changelogs=1256897}]
<br>
[2021-03-11
19:15:32.621244]
E [MSGID:
132009]
[gf-history-changelog.c:941:gf_history_changelog]
0-gfchangelog:
wrong result
[{for=end},
{start=1615490121},
{idx=1256896}]
<br>
[2021-03-11
19:15:46.733182]
I [MSGID:
132028]
[gf-changelog.c:577:gf_changelog_register_generic]
0-gfchangelog:
Registering
brick
[{brick=/data/storage_a/storage},
{notify_filter=1}]
<br>
[2021-03-11
19:15:46.733316]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=0}]
<br>
[2021-03-11
19:15:46.733348]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=1}]
<br>
[2021-03-11
19:15:46.734031]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=2}]
<br>
[2021-03-11
19:15:46.734085]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=3}]
<br>
[2021-03-11
19:15:46.734591]
D
[rpcsvc.c:2831:rpcsvc_init]
0-rpc-service:
RPC service
inited.<br>
[2021-03-11
19:15:46.734755]
D
[rpcsvc.c:2342:rpcsvc_program_register]
0-rpc-service:
New program
registered:
GF-DUMP, Num:
123451501,
Ver: 1, Port:
0<br>
[2021-03-11
19:15:46.734772]
D
[rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
[2021-03-11
19:15:46.735256]
D
[socket.c:4485:socket_init]
0-socket.gfchangelog: disabling nodelay<br>
[2021-03-11
19:15:46.735266]
D
[socket.c:4523:socket_init]
0-socket.gfchangelog: Configured transport.tcp-user-timeout=42<br>
[2021-03-11
19:15:46.735271]
D
[socket.c:4543:socket_init]
0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9<br>
[2021-03-11
19:15:46.735325]
I
[socket.c:929:__socket_server_bind]
0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21<br>
[2021-03-11
19:15:46.735704]
D
[rpcsvc.c:2342:rpcsvc_program_register]
0-rpc-service:
New program
registered:
LIBGFCHANGELOG
REBORP, Num:
1886350951,
Ver: 1, Port:
0<br>
[2021-03-11
19:15:46.735721]
D
[rpc-clnt.c:1020:rpc_clnt_connection_init]
0-gfchangelog:
defaulting
frame-timeout
to 30mins<br>
[2021-03-11
19:15:46.735726]
D
[rpc-clnt.c:1032:rpc_clnt_connection_init]
0-gfchangelog:
disable
ping-timeout<br>
[2021-03-11
19:15:46.735733]
D
[rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
[2021-03-11
19:15:46.735771]
D
[socket.c:4485:socket_init]
0-gfchangelog:
disabling
nodelay<br>
[2021-03-11
19:15:46.735778]
D
[socket.c:4523:socket_init]
0-gfchangelog:
Configured
transport.tcp-user-timeout=42<br>
[2021-03-11
19:15:47.618464]
D
[rpc-clnt-ping.c:298:rpc_clnt_start_ping]
0-gfchangelog:
ping timeout
is 0,
returning<br>
<br>
<br>
[root@storage01 storage_10.0.231.81_pcic-backup]# cat
changes-data-storage_b-storage.log
| egrep
"2021-03-11"<br>
[2021-03-11
19:15:30.611457]
I [MSGID:
132028]
[gf-changelog.c:577:gf_changelog_register_generic]
0-gfchangelog:
Registering
brick
[{brick=/data/storage_b/storage},
{notify_filter=1}]
<br>
[2021-03-11
19:15:30.611574]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=1}]
<br>
[2021-03-11
19:15:30.611641]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=3}]
<br>
[2021-03-11
19:15:30.611645]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=2}]
<br>
[2021-03-11
19:15:30.612325]
D
[rpcsvc.c:2831:rpcsvc_init]
0-rpc-service:
RPC service
inited.<br>
[2021-03-11
19:15:30.612488]
D
[rpcsvc.c:2342:rpcsvc_program_register]
0-rpc-service:
New program
registered:
GF-DUMP, Num:
123451501,
Ver: 1, Port:
0<br>
[2021-03-11
19:15:30.612507]
D
[rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
[2021-03-11
19:15:30.613005]
D
[socket.c:4485:socket_init]
0-socket.gfchangelog: disabling nodelay<br>
[2021-03-11
19:15:30.613130]
D
[socket.c:4523:socket_init]
0-socket.gfchangelog: Configured transport.tcp-user-timeout=42<br>
[2021-03-11
19:15:30.613142]
D
[socket.c:4543:socket_init]
0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9<br>
[2021-03-11
19:15:30.613208]
I
[socket.c:929:__socket_server_bind]
0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 22<br>
[2021-03-11
19:15:30.613545]
D
[rpcsvc.c:2342:rpcsvc_program_register]
0-rpc-service:
New program
registered:
LIBGFCHANGELOG
REBORP, Num:
1886350951,
Ver: 1, Port:
0<br>
[2021-03-11
19:15:30.613567]
D
[rpc-clnt.c:1020:rpc_clnt_connection_init]
0-gfchangelog:
defaulting
frame-timeout
to 30mins<br>
[2021-03-11
19:15:30.613574]
D
[rpc-clnt.c:1032:rpc_clnt_connection_init]
0-gfchangelog:
disable
ping-timeout<br>
[2021-03-11
19:15:30.613582]
D
[rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
[2021-03-11
19:15:30.613637]
D
[socket.c:4485:socket_init]
0-gfchangelog:
disabling
nodelay<br>
[2021-03-11
19:15:30.613654]
D
[socket.c:4523:socket_init]
0-gfchangelog:
Configured
transport.tcp-user-timeout=42<br>
[2021-03-11
19:15:32.614273]
D
[rpc-clnt-ping.c:298:rpc_clnt_start_ping]
0-gfchangelog:
ping timeout
is 0,
returning<br>
[2021-03-11
19:15:32.643628]
I [MSGID:
132035]
[gf-history-changelog.c:837:gf_history_changelog]
0-gfchangelog:
Requesting
historical
changelogs
[{start=1614666552},
{end=1615490132}]
<br>
[2021-03-11
19:15:32.643716]
I [MSGID:
132019]
[gf-history-changelog.c:755:gf_changelog_extract_min_max]
0-gfchangelog:
changelogs min
max
[{min=1597342860},
{max=1615490123}, {total_changelogs=1264296}]
<br>
[2021-03-11
19:15:32.700397]
E [MSGID:
132009]
[gf-history-changelog.c:941:gf_history_changelog]
0-gfchangelog:
wrong result
[{for=end},
{start=1615490123},
{idx=1264295}]
<br>
[2021-03-11
19:15:46.832322]
I [MSGID:
132028]
[gf-changelog.c:577:gf_changelog_register_generic]
0-gfchangelog:
Registering
brick
[{brick=/data/storage_b/storage},
{notify_filter=1}]
<br>
[2021-03-11
19:15:46.832394]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=0}]
<br>
[2021-03-11
19:15:46.832465]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=1}]
<br>
[2021-03-11
19:15:46.832531]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=2}]
<br>
[2021-03-11
19:15:46.833086]
I [MSGID:
101190]
[event-epoll.c:670:event_dispatch_epoll_worker]
0-epoll:
Started thread
with index
[{index=3}]
<br>
[2021-03-11
19:15:46.833648]
D
[rpcsvc.c:2831:rpcsvc_init]
0-rpc-service:
RPC service
inited.<br>
[2021-03-11
19:15:46.833817]
D
[rpcsvc.c:2342:rpcsvc_program_register]
0-rpc-service:
New program
registered:
GF-DUMP, Num:
123451501,
Ver: 1, Port:
0<br>
[2021-03-11
19:15:46.833835]
D
[rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
[2021-03-11
19:15:46.834368]
D
[socket.c:4485:socket_init]
0-socket.gfchangelog: disabling nodelay<br>
[2021-03-11
19:15:46.834380]
D
[socket.c:4523:socket_init]
0-socket.gfchangelog: Configured transport.tcp-user-timeout=42<br>
[2021-03-11
19:15:46.834386]
D
[socket.c:4543:socket_init]
0-socket.gfchangelog: Reconfigured transport.keepalivecnt=9<br>
[2021-03-11
19:15:46.834441]
I
[socket.c:929:__socket_server_bind]
0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23<br>
[2021-03-11
19:15:46.834768]
D
[rpcsvc.c:2342:rpcsvc_program_register]
0-rpc-service:
New program
registered:
LIBGFCHANGELOG
REBORP, Num:
1886350951,
Ver: 1, Port:
0<br>
[2021-03-11
19:15:46.834789]
D
[rpc-clnt.c:1020:rpc_clnt_connection_init]
0-gfchangelog:
defaulting
frame-timeout
to 30mins<br>
[2021-03-11
19:15:46.834795]
D
[rpc-clnt.c:1032:rpc_clnt_connection_init]
0-gfchangelog:
disable
ping-timeout<br>
[2021-03-11
19:15:46.834802]
D
[rpc-transport.c:278:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/8.3/rpc-transport/socket.so<br>
[2021-03-11
19:15:46.834845]
D
[socket.c:4485:socket_init]
0-gfchangelog:
disabling
nodelay<br>
[2021-03-11
19:15:46.834853]
D
[socket.c:4523:socket_init]
0-gfchangelog:
Configured
transport.tcp-user-timeout=42<br>
[2021-03-11
19:15:47.618476]
D
[rpc-clnt-ping.c:298:rpc_clnt_start_ping]
0-gfchangelog:
ping timeout
is 0,
returning<br>
<br>
<br>
gsyncd logged
a lot but I'm
not sure if
it's helpful:
<br>
<br>
[2021-03-11
19:15:00.41898]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:21.551302]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:21.631470]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:21.718386]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:21.804991]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:26.203999]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:26.284775]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:26.573355]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:26.653752]
D
[gsyncd(monitor):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:26.756994]
D
[monitor(monitor):304:distribute]
<top>:
master bricks:
[{'host':
'10.0.231.91',
'uuid':
'afc24654-2887-41f6-a9c2-8e835de243b6',
'dir':
'/data/storage_a/storage'},
{'host':
'10.0.2<br>
31.92',
'uuid':
'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d',
'dir':
'/data/storage_b/storage'},
{'host':
'10.0.231.93',
'uuid':
'8b28b331-3780-46bc-9da3-fb27de4ab57b',
'dir':
'/data/storage_c/storage'},
{'host': '10.<br>
0.231.92',
'uuid':
'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d',
'dir':
'/data/storage_a/storage'},
{'host':
'10.0.231.93',
'uuid':
'8b28b331-3780-46bc-9da3-fb27de4ab57b',
'dir':
'/data/storage_b/storage'},
{'host': '<br>
10.0.231.91',
'uuid':
'afc24654-2887-41f6-a9c2-8e835de243b6',
'dir':
'/data/storage_c/storage'},
{'host':
'10.0.231.93',
'uuid':
'8b28b331-3780-46bc-9da3-fb27de4ab57b',
'dir':
'/data/storage_a/storage'},
{'host'<br>
:
'10.0.231.91',
'uuid':
'afc24654-2887-41f6-a9c2-8e835de243b6',
'dir':
'/data/storage_b/storage'},
{'host':
'10.0.231.92',
'uuid':
'ebbd7b74-3cf8-4752-a71c-b0f0ca86c97d',
'dir':
'/data/storage_c/storage'}]<br>
[2021-03-11
19:15:26.757252]
D
[monitor(monitor):314:distribute]
<top>:
slave SSH
gateway:
<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81</a><br>
[2021-03-11
19:15:27.416235]
D
[monitor(monitor):334:distribute]
<top>:
slave bricks:
[{'host':
'10.0.231.81',
'uuid':
'b88dea4f-31ec-416a-9110-3ccdc3910acd',
'dir':
'/data/brick'},
{'host':
'10.0.231.82',
'uuid<br>
':
'be50a8de-3934-4fee-a80d-8e2e99017902',
'dir':
'/data/brick'}]<br>
[2021-03-11
19:15:27.416825]
D
[syncdutils(monitor):932:is_hot]
Volinfo:
brickpath:
'10.0.231.91:/data/storage_a/storage'<br>
[2021-03-11
19:15:27.417273]
D
[syncdutils(monitor):932:is_hot]
Volinfo:
brickpath:
'10.0.231.91:/data/storage_c/storage'<br>
[2021-03-11
19:15:27.417515]
D
[syncdutils(monitor):932:is_hot]
Volinfo:
brickpath:
'10.0.231.91:/data/storage_b/storage'<br>
[2021-03-11
19:15:27.417763]
D
[monitor(monitor):348:distribute]
<top>:
worker specs:
[({'host':
'10.0.231.91',
'uuid':
'afc24654-2887-41f6-a9c2-8e835de243b6',
'dir':
'/data/storage_a/storage'},
('geoaccount@10.<br>
0.231.81',
'b88dea4f-31ec-416a-9110-3ccdc3910acd'),
'1', False),
({'host':
'10.0.231.91',
'uuid':
'afc24654-2887-41f6-a9c2-8e835de243b6',
'dir':
'/data/storage_c/storage'},
('<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.82"
target="_blank" moz-do-not-send="true">geoaccount@10.0.231.82</a>',
'be50a8de-3<br>
934-4fee-a80d-8e2e99017902'), '2', False), ({'host': '10.0.231.91',
'uuid':
'afc24654-2887-41f6-a9c2-8e835de243b6',
'dir':
'/data/storage_b/storage'},
('<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.82"
target="_blank" moz-do-not-send="true">geoaccount@10.0.231.82</a>',
'be50a8de-3934-4fee-a80d-8e2e9901<br>
7902'), '3',
False)]<br>
[2021-03-11
19:15:27.425009]
I
[monitor(monitor):160:monitor]
Monitor:
starting
gsyncd worker
[{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]<br>
[2021-03-11
19:15:27.426764]
I
[monitor(monitor):160:monitor]
Monitor:
starting
gsyncd worker
[{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]<br>
[2021-03-11
19:15:27.429208]
I
[monitor(monitor):160:monitor]
Monitor:
starting
gsyncd worker
[{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]<br>
[2021-03-11
19:15:27.432280]
D
[monitor(monitor):195:monitor]
Monitor:
Worker would
mount volume
privately<br>
[2021-03-11
19:15:27.434195]
D
[monitor(monitor):195:monitor]
Monitor:
Worker would
mount volume
privately<br>
[2021-03-11
19:15:27.436584]
D
[monitor(monitor):195:monitor]
Monitor:
Worker would
mount volume
privately<br>
[2021-03-11
19:15:27.478806]
D
[gsyncd(worker
/data/storage_c/storage):303:main] <top>: Using session config
file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:27.478852]
D
[gsyncd(worker
/data/storage_b/storage):303:main] <top>: Using session config
file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:27.480104]
D
[gsyncd(worker
/data/storage_a/storage):303:main] <top>: Using session config
file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:27.500456]
I
[resource(worker
/data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH
connection
between master
and slave...<br>
[2021-03-11
19:15:27.501375]
I
[resource(worker
/data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH
connection
between master
and slave...<br>
[2021-03-11
19:15:27.502003]
I
[resource(worker
/data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH
connection
between master
and slave...<br>
[2021-03-11
19:15:27.525511]
D
[repce(worker
/data/storage_a/storage):195:push] RepceClient: call
192117:140572692309824:1615490127.53
__repce_version__() ...<br>
[2021-03-11
19:15:27.525582]
D
[repce(worker
/data/storage_b/storage):195:push] RepceClient: call
192115:139891296405312:1615490127.53
__repce_version__() ...<br>
[2021-03-11
19:15:27.526089]
D
[repce(worker
/data/storage_c/storage):195:push] RepceClient: call
192114:140388828780352:1615490127.53
__repce_version__() ...<br>
[2021-03-11
19:15:29.435985]
D
[repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
192117:140572692309824:1615490127.53
__repce_version__ -> 1.0<br>
[2021-03-11
19:15:29.436213]
D
[repce(worker
/data/storage_a/storage):195:push] RepceClient: call
192117:140572692309824:1615490129.44
version() ...<br>
[2021-03-11
19:15:29.437136]
D
[repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
192117:140572692309824:1615490129.44
version ->
1.0<br>
[2021-03-11
19:15:29.437268]
D
[repce(worker
/data/storage_a/storage):195:push] RepceClient: call
192117:140572692309824:1615490129.44
pid() ...<br>
[2021-03-11
19:15:29.437915]
D
[repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
192117:140572692309824:1615490129.44
pid ->
157321<br>
[2021-03-11
19:15:29.438004]
I
[resource(worker
/data/storage_a/storage):1436:connect_remote] SSH: SSH connection
between master
and slave
established.
[{duration=1.9359}]<br>
[2021-03-11
19:15:29.438072]
I
[resource(worker
/data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume
locally...<br>
[2021-03-11
19:15:29.494538]
D
[repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
192115:139891296405312:1615490127.53
__repce_version__ -> 1.0<br>
[2021-03-11
19:15:29.494748]
D
[repce(worker
/data/storage_b/storage):195:push] RepceClient: call
192115:139891296405312:1615490129.49
version() ...<br>
[2021-03-11
19:15:29.495290]
D
[repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
192115:139891296405312:1615490129.49
version ->
1.0<br>
[2021-03-11
19:15:29.495400]
D
[repce(worker
/data/storage_b/storage):195:push] RepceClient: call
192115:139891296405312:1615490129.5
pid() ...<br>
[2021-03-11
19:15:29.495872]
D
[repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
192115:139891296405312:1615490129.5
pid ->
88110<br>
[2021-03-11
19:15:29.495960]
I
[resource(worker
/data/storage_b/storage):1436:connect_remote] SSH: SSH connection
between master
and slave
established.
[{duration=1.9944}]<br>
[2021-03-11
19:15:29.496028]
I
[resource(worker
/data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume
locally...<br>
[2021-03-11
19:15:29.501255]
D
[repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
192114:140388828780352:1615490127.53
__repce_version__ -> 1.0<br>
[2021-03-11
19:15:29.501454]
D
[repce(worker
/data/storage_c/storage):195:push] RepceClient: call
192114:140388828780352:1615490129.5
version() ...<br>
[2021-03-11
19:15:29.502258]
D
[repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
192114:140388828780352:1615490129.5
version ->
1.0<br>
[2021-03-11
19:15:29.502444]
D
[repce(worker
/data/storage_c/storage):195:push] RepceClient: call
192114:140388828780352:1615490129.5
pid() ...<br>
[2021-03-11
19:15:29.503140]
D
[repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
192114:140388828780352:1615490129.5
pid ->
88111<br>
[2021-03-11
19:15:29.503232]
I
[resource(worker
/data/storage_c/storage):1436:connect_remote] SSH: SSH connection
between master
and slave
established.
[{duration=2.0026}]<br>
[2021-03-11
19:15:29.503302]
I
[resource(worker
/data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume
locally...<br>
[2021-03-11
19:15:29.533899]
D
[resource(worker
/data/storage_a/storage):880:inhibit] DirectMounter: auxiliary glusterfs
mount in place<br>
[2021-03-11
19:15:29.595736]
D
[resource(worker
/data/storage_b/storage):880:inhibit] DirectMounter: auxiliary glusterfs
mount in place<br>
[2021-03-11
19:15:29.601110]
D
[resource(worker
/data/storage_c/storage):880:inhibit] DirectMounter: auxiliary glusterfs
mount in place<br>
[2021-03-11
19:15:30.541542]
D
[resource(worker
/data/storage_a/storage):964:inhibit] DirectMounter: auxiliary glusterfs
mount prepared<br>
[2021-03-11
19:15:30.541816]
I
[resource(worker
/data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume
[{duration=1.1037}]<br>
[2021-03-11
19:15:30.541887]
I
[subcmds(worker
/data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn
successful.
Acknowledging
back to
monitor<br>
[2021-03-11
19:15:30.542042]
D
[master(worker
/data/storage_a/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=xsync}]<br>
[2021-03-11
19:15:30.542125]
D
[monitor(monitor):222:monitor]
Monitor:
worker(/data/storage_a/storage)
connected<br>
[2021-03-11
19:15:30.543323]
D
[master(worker
/data/storage_a/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changelog}]<br>
[2021-03-11
19:15:30.544460]
D
[master(worker
/data/storage_a/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changeloghistory}]<br>
[2021-03-11
19:15:30.552103]
D
[master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
[2021-03-11
19:15:30.602937]
D
[resource(worker
/data/storage_b/storage):964:inhibit] DirectMounter: auxiliary glusterfs
mount prepared<br>
[2021-03-11
19:15:30.603117]
I
[resource(worker
/data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume
[{duration=1.1070}]<br>
[2021-03-11
19:15:30.603197]
I
[subcmds(worker
/data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn
successful.
Acknowledging
back to
monitor<br>
[2021-03-11
19:15:30.603353]
D
[master(worker
/data/storage_b/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=xsync}]<br>
[2021-03-11
19:15:30.603338]
D
[monitor(monitor):222:monitor]
Monitor:
worker(/data/storage_b/storage)
connected<br>
[2021-03-11
19:15:30.604620]
D
[master(worker
/data/storage_b/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changelog}]<br>
[2021-03-11
19:15:30.605600]
D
[master(worker
/data/storage_b/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changeloghistory}]<br>
[2021-03-11
19:15:30.608365]
D
[resource(worker
/data/storage_c/storage):964:inhibit] DirectMounter: auxiliary glusterfs
mount prepared<br>
[2021-03-11
19:15:30.608534]
I
[resource(worker
/data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume
[{duration=1.1052}]<br>
[2021-03-11
19:15:30.608612]
I
[subcmds(worker
/data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn
successful.
Acknowledging
back to
monitor<br>
[2021-03-11
19:15:30.608762]
D
[master(worker
/data/storage_c/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=xsync}]<br>
[2021-03-11
19:15:30.608779]
D
[monitor(monitor):222:monitor]
Monitor:
worker(/data/storage_c/storage)
connected<br>
[2021-03-11
19:15:30.610033]
D
[master(worker
/data/storage_c/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changelog}]<br>
[2021-03-11
19:15:30.610637]
D
[master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
[2021-03-11
19:15:30.610970]
D
[master(worker
/data/storage_c/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changeloghistory}]<br>
[2021-03-11
19:15:30.616197]
D
[master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
[2021-03-11
19:15:31.371265]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:31.451000]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:31.537257]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:31.623800]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:32.555840]
D
[master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
[2021-03-11
19:15:32.556051]
D
[master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
[2021-03-11
19:15:32.556122]
D
[master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
[2021-03-11
19:15:32.556179]
I
[master(worker
/data/storage_a/storage):1645:register] _GMaster: Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]<br>
[2021-03-11
19:15:32.556359]
I
[resource(worker
/data/storage_a/storage):1292:service_loop] GLUSTER: Register time
[{time=1615490132}]<br>
[2021-03-11
19:15:32.556823]
D
[repce(worker
/data/storage_a/storage):195:push] RepceClient: call
192117:140570487928576:1615490132.56
keep_alive(None,) ...<br>
[2021-03-11
19:15:32.558429]
D
[repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
192117:140570487928576:1615490132.56
keep_alive
-> 1<br>
[2021-03-11
19:15:32.558974]
D
[master(worker
/data/storage_a/storage):540:crawlwrap] _GMaster: primary master with
volume id
cf94a8f2-324b-40b3-bf72-c3766100ea99
...<br>
[2021-03-11
19:15:32.567478]
I
[gsyncdstatus(worker
/data/storage_a/storage):281:set_active] GeorepStatus: Worker Status
Change
[{status=Active}]<br>
[2021-03-11
19:15:32.571824]
I
[gsyncdstatus(worker
/data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus:
Crawl Status
Change
[{status=History
Crawl}]<br>
[2021-03-11
19:15:32.572052]
I
[master(worker
/data/storage_a/storage):1559:crawl] _GMaster: starting history crawl
[{turns=1},
{stime=(1614666553,
0)},
{entry_stime=(1614664115,
0)},
{etime=1615490132}]<br>
[2021-03-11
19:15:32.614506]
D
[master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
[2021-03-11
19:15:32.614701]
D
[master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
[2021-03-11
19:15:32.614788]
D
[master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
[2021-03-11
19:15:32.614845]
I
[master(worker
/data/storage_b/storage):1645:register] _GMaster: Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]<br>
[2021-03-11
19:15:32.615000]
I
[resource(worker
/data/storage_b/storage):1292:service_loop] GLUSTER: Register time
[{time=1615490132}]<br>
[2021-03-11
19:15:32.615586]
D
[repce(worker
/data/storage_b/storage):195:push] RepceClient: call
192115:139889215526656:1615490132.62
keep_alive(None,) ...<br>
[2021-03-11
19:15:32.617373]
D
[repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
192115:139889215526656:1615490132.62
keep_alive
-> 1<br>
[2021-03-11
19:15:32.618144]
D
[master(worker
/data/storage_b/storage):540:crawlwrap] _GMaster: primary master with
volume id
cf94a8f2-324b-40b3-bf72-c3766100ea99
...<br>
[2021-03-11
19:15:32.619323]
D
[master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
[2021-03-11
19:15:32.619491]
D
[master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
[2021-03-11
19:15:32.619739]
D
[master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
[2021-03-11
19:15:32.619863]
I
[master(worker
/data/storage_c/storage):1645:register] _GMaster: Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]<br>
[2021-03-11
19:15:32.620040]
I
[resource(worker
/data/storage_c/storage):1292:service_loop] GLUSTER: Register time
[{time=1615490132}]<br>
[2021-03-11
19:15:32.620599]
D
[repce(worker
/data/storage_c/storage):195:push] RepceClient: call
192114:140386886469376:1615490132.62
keep_alive(None,) ...<br>
[2021-03-11
19:15:32.621397]
E
[resource(worker
/data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History
Crawl failed
[{error=[Errno
0] Success}]<br>
[2021-03-11
19:15:32.622035]
D
[repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
192114:140386886469376:1615490132.62
keep_alive
-> 1<br>
[2021-03-11
19:15:32.622701]
D
[master(worker
/data/storage_c/storage):540:crawlwrap] _GMaster: primary master with
volume id
cf94a8f2-324b-40b3-bf72-c3766100ea99
...<br>
[2021-03-11
19:15:32.627031]
I
[gsyncdstatus(worker
/data/storage_b/storage):281:set_active] GeorepStatus: Worker Status
Change
[{status=Active}]<br>
[2021-03-11
19:15:32.643184]
I
[gsyncdstatus(worker
/data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus:
Crawl Status
Change
[{status=History
Crawl}]<br>
[2021-03-11
19:15:32.643528]
I
[master(worker
/data/storage_b/storage):1559:crawl] _GMaster: starting history crawl
[{turns=1},
{stime=(1614666552,
0)},
{entry_stime=(1614664113,
0)},
{etime=1615490132}]<br>
[2021-03-11
19:15:32.645148]
I
[gsyncdstatus(worker
/data/storage_c/storage):281:set_active] GeorepStatus: Worker Status
Change
[{status=Active}]<br>
[2021-03-11
19:15:32.649631]
I
[gsyncdstatus(worker
/data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus:
Crawl Status
Change
[{status=History
Crawl}]<br>
[2021-03-11
19:15:32.649882]
I
[master(worker
/data/storage_c/storage):1559:crawl] _GMaster: starting history crawl
[{turns=1},
{stime=(1614666552,
0)},
{entry_stime=(1614664108,
0)},
{etime=1615490132}]<br>
[2021-03-11
19:15:32.650907]
E
[resource(worker
/data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History
Crawl failed
[{error=[Errno
0] Success}]<br>
[2021-03-11
19:15:32.700489]
E
[resource(worker
/data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History
Crawl failed
[{error=[Errno
0] Success}]<br>
[2021-03-11
19:15:33.545886]
I
[monitor(monitor):228:monitor]
Monitor:
worker died in
startup phase
[{brick=/data/storage_a/storage}]<br>
[2021-03-11
19:15:33.550487]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Faulty}]<br>
[2021-03-11
19:15:33.606991]
I
[monitor(monitor):228:monitor]
Monitor:
worker died in
startup phase
[{brick=/data/storage_b/storage}]<br>
[2021-03-11
19:15:33.611573]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Faulty}]<br>
[2021-03-11
19:15:33.612337]
I
[monitor(monitor):228:monitor]
Monitor:
worker died in
startup phase
[{brick=/data/storage_c/storage}]<br>
[2021-03-11
19:15:33.615777]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Faulty}]<br>
[2021-03-11
19:15:34.684247]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:34.764971]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:34.851174]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:34.937166]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:36.994502]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:37.73805]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:37.159288]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:37.244153]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:38.916510]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:38.997649]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:39.84816]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:39.172045]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:40.896359]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:40.976135]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:41.62052]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:41.147902]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:42.791997]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:42.871239]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:42.956609]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:43.42473]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:43.566190]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Initializing...}]<br>
[2021-03-11
19:15:43.566400]
I
[monitor(monitor):160:monitor]
Monitor:
starting
gsyncd worker
[{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]<br>
[2021-03-11
19:15:43.572240]
D
[monitor(monitor):195:monitor]
Monitor:
Worker would
mount volume
privately<br>
[2021-03-11
19:15:43.612744]
D
[gsyncd(worker
/data/storage_a/storage):303:main] <top>: Using session config
file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:43.625689]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Initializing...}]<br>
[2021-03-11
19:15:43.626060]
I
[monitor(monitor):160:monitor]
Monitor:
starting
gsyncd worker
[{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]<br>
[2021-03-11
19:15:43.632287]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Initializing...}]<br>
[2021-03-11
19:15:43.632137]
D
[monitor(monitor):195:monitor]
Monitor:
Worker would
mount volume
privately<br>
[2021-03-11
19:15:43.632508]
I
[monitor(monitor):160:monitor]
Monitor:
starting
gsyncd worker
[{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]<br>
[2021-03-11
19:15:43.635565]
I
[resource(worker
/data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH
connection
between master
and slave...<br>
[2021-03-11
19:15:43.637835]
D
[monitor(monitor):195:monitor]
Monitor:
Worker would
mount volume
privately<br>
[2021-03-11
19:15:43.661304]
D
[repce(worker
/data/storage_a/storage):195:push] RepceClient: call
192535:140367272073024:1615490143.66
__repce_version__() ...<br>
[2021-03-11
19:15:43.674499]
D
[gsyncd(worker
/data/storage_b/storage):303:main] <top>: Using session config
file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:43.680706]
D
[gsyncd(worker
/data/storage_c/storage):303:main] <top>: Using session config
file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:43.693773]
I
[resource(worker
/data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH
connection
between master
and slave...<br>
[2021-03-11
19:15:43.700957]
I
[resource(worker
/data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH
connection
between master
and slave...<br>
[2021-03-11
19:15:43.717686]
D
[repce(worker
/data/storage_b/storage):195:push] RepceClient: call
192539:139907321804608:1615490143.72
__repce_version__() ...<br>
[2021-03-11
19:15:43.725369]
D
[repce(worker
/data/storage_c/storage):195:push] RepceClient: call
192541:140653101852480:1615490143.73
__repce_version__() ...<br>
[2021-03-11
19:15:44.289117]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:44.375693]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:44.472251]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:44.558429]
D
[gsyncd(status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:45.619694]
D
[repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
192535:140367272073024:1615490143.66
__repce_version__ -> 1.0<br>
[2021-03-11
19:15:45.619930]
D
[repce(worker
/data/storage_a/storage):195:push] RepceClient: call
192535:140367272073024:1615490145.62
version() ...<br>
[2021-03-11
19:15:45.621191]
D
[repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
192535:140367272073024:1615490145.62
version ->
1.0<br>
[2021-03-11
19:15:45.621332]
D
[repce(worker
/data/storage_a/storage):195:push] RepceClient: call
192535:140367272073024:1615490145.62
pid() ...<br>
[2021-03-11
19:15:45.621859]
D
[repce(worker
/data/storage_a/storage):215:__call__] RepceClient: call
192535:140367272073024:1615490145.62
pid ->
158229<br>
[2021-03-11
19:15:45.621939]
I
[resource(worker
/data/storage_a/storage):1436:connect_remote] SSH: SSH connection
between master
and slave
established.
[{duration=1.9862}]<br>
[2021-03-11
19:15:45.622000]
I
[resource(worker
/data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume
locally...<br>
[2021-03-11
19:15:45.714468]
D
[resource(worker
/data/storage_a/storage):880:inhibit] DirectMounter: auxiliary glusterfs
mount in place<br>
[2021-03-11
19:15:45.718441]
D
[repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
192541:140653101852480:1615490143.73
__repce_version__ -> 1.0<br>
[2021-03-11
19:15:45.718643]
D
[repce(worker
/data/storage_c/storage):195:push] RepceClient: call
192541:140653101852480:1615490145.72
version() ...<br>
[2021-03-11
19:15:45.719492]
D
[repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
192541:140653101852480:1615490145.72
version ->
1.0<br>
[2021-03-11
19:15:45.719772]
D
[repce(worker
/data/storage_c/storage):195:push] RepceClient: call
192541:140653101852480:1615490145.72
pid() ...<br>
[2021-03-11
19:15:45.720202]
D
[repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
192539:139907321804608:1615490143.72
__repce_version__ -> 1.0<br>
[2021-03-11
19:15:45.720381]
D
[repce(worker
/data/storage_b/storage):195:push] RepceClient: call
192539:139907321804608:1615490145.72
version() ...<br>
[2021-03-11
19:15:45.720463]
D
[repce(worker
/data/storage_c/storage):215:__call__] RepceClient: call
192541:140653101852480:1615490145.72
pid ->
88921<br>
[2021-03-11
19:15:45.720694]
I
[resource(worker
/data/storage_c/storage):1436:connect_remote] SSH: SSH connection
between master
and slave
established.
[{duration=2.0196}]<br>
[2021-03-11
19:15:45.720882]
I
[resource(worker
/data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume
locally...<br>
[2021-03-11
19:15:45.721146]
D
[repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
192539:139907321804608:1615490145.72
version ->
1.0<br>
[2021-03-11
19:15:45.721271]
D
[repce(worker
/data/storage_b/storage):195:push] RepceClient: call
192539:139907321804608:1615490145.72
pid() ...<br>
[2021-03-11
19:15:45.721795]
D
[repce(worker
/data/storage_b/storage):215:__call__] RepceClient: call
192539:139907321804608:1615490145.72
pid ->
88924<br>
[2021-03-11
19:15:45.721911]
I
[resource(worker
/data/storage_b/storage):1436:connect_remote] SSH: SSH connection
between master
and slave
established.
[{duration=2.0280}]<br>
[2021-03-11
19:15:45.721993]
I
[resource(worker
/data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume
locally...<br>
[2021-03-11
19:15:45.816891]
D
[resource(worker
/data/storage_b/storage):880:inhibit] DirectMounter: auxiliary glusterfs
mount in place<br>
[2021-03-11
19:15:45.816960]
D
[resource(worker
/data/storage_c/storage):880:inhibit] DirectMounter: auxiliary glusterfs
mount in place<br>
[2021-03-11
19:15:46.721534]
D
[resource(worker
/data/storage_a/storage):964:inhibit] DirectMounter: auxiliary glusterfs
mount prepared<br>
[2021-03-11
19:15:46.721726]
I
[resource(worker
/data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume
[{duration=1.0997}]<br>
[2021-03-11
19:15:46.721796]
I
[subcmds(worker
/data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn
successful.
Acknowledging
back to
monitor<br>
[2021-03-11
19:15:46.721971]
D
[master(worker
/data/storage_a/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=xsync}]<br>
[2021-03-11
19:15:46.722122]
D
[monitor(monitor):222:monitor]
Monitor:
worker(/data/storage_a/storage)
connected<br>
[2021-03-11
19:15:46.723871]
D
[master(worker
/data/storage_a/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changelog}]<br>
[2021-03-11
19:15:46.725100]
D
[master(worker
/data/storage_a/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changeloghistory}]<br>
[2021-03-11
19:15:46.732400]
D
[master(worker
/data/storage_a/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage<br>
[2021-03-11
19:15:46.823477]
D
[resource(worker
/data/storage_c/storage):964:inhibit] DirectMounter: auxiliary glusterfs
mount prepared<br>
[2021-03-11
19:15:46.823645]
I
[resource(worker
/data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume
[{duration=1.1027}]<br>
[2021-03-11
19:15:46.823754]
I
[subcmds(worker
/data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn
successful.
Acknowledging
back to
monitor<br>
[2021-03-11
19:15:46.823932]
D
[master(worker
/data/storage_c/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=xsync}]<br>
[2021-03-11
19:15:46.823904]
D
[resource(worker
/data/storage_b/storage):964:inhibit] DirectMounter: auxiliary glusterfs
mount prepared<br>
[2021-03-11
19:15:46.823930]
D
[monitor(monitor):222:monitor]
Monitor:
worker(/data/storage_c/storage)
connected<br>
[2021-03-11
19:15:46.824103]
I
[resource(worker
/data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume
[{duration=1.1020}]<br>
[2021-03-11
19:15:46.824184]
I
[subcmds(worker
/data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn
successful.
Acknowledging
back to
monitor<br>
[2021-03-11
19:15:46.824340]
D
[master(worker
/data/storage_b/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=xsync}]<br>
[2021-03-11
19:15:46.824321]
D
[monitor(monitor):222:monitor]
Monitor:
worker(/data/storage_b/storage)
connected<br>
[2021-03-11
19:15:46.825100]
D
[master(worker
/data/storage_c/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changelog}]<br>
[2021-03-11
19:15:46.825414]
D
[master(worker
/data/storage_b/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changelog}]<br>
[2021-03-11
19:15:46.826375]
D
[master(worker
/data/storage_b/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changeloghistory}]<br>
[2021-03-11
19:15:46.826574]
D
[master(worker
/data/storage_c/storage):105:gmaster_builder] <top>: setting up
change
detection mode
[{mode=changeloghistory}]<br>
[2021-03-11
19:15:46.831506]
D
[master(worker
/data/storage_b/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage<br>
[2021-03-11
19:15:46.833168]
D
[master(worker
/data/storage_c/storage):778:setup_working_dir] _GMaster: changelog
working dir
/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage<br>
[2021-03-11
19:15:47.275141]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:47.320247]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:47.570877]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:47.615571]
D
[gsyncd(config-get):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:47.620893]
E
[syncdutils(worker
/data/storage_a/storage):325:log_raise_exception] <top>:
connection to
peer is broken<br>
[2021-03-11
19:15:47.620939]
E
[syncdutils(worker
/data/storage_c/storage):325:log_raise_exception] <top>:
connection to
peer is broken<br>
[2021-03-11
19:15:47.621668]
E
[syncdutils(worker
/data/storage_a/storage):847:errlog] Popen: command returned error
[{cmd=ssh
-oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem
-p 22
-oControlMaster=auto
-S
/tmp/gsyncd-aux-ssh-_AyCOc/79fa3dc75e30f532b4a40bc08c2b10a1.sock
<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81</a> /nonexistent/gsyncd slave storage <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> --master-node 10.0.231.91
--master-node-id
afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
/data/storage_a/storage
--local-node
10.0.231.81
--local-node-id
b88dea4f-31ec-416a-9110-3ccdc3910acd --slave-timeout 120
--slave-log-level
INFO
--slave-gluster-log-level
INFO
--slave-gluster-command-dir
/usr/sbin
--master-dist-count
3},
{error=255}]<br>
[2021-03-11
19:15:47.621685]
E
[syncdutils(worker
/data/storage_c/storage):847:errlog] Popen: command returned error
[{cmd=ssh
-oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem
-p 22
-oControlMaster=auto
-S
/tmp/gsyncd-aux-ssh-WOgOEu/e15fc58bb13552de0710eaf018209548.sock
<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.82"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.82</a> /nonexistent/gsyncd slave storage <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> --master-node 10.0.231.91
--master-node-id
afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
/data/storage_c/storage
--local-node
10.0.231.82
--local-node-id
be50a8de-3934-4fee-a80d-8e2e99017902 --slave-timeout 120
--slave-log-level
INFO
--slave-gluster-log-level
INFO
--slave-gluster-command-dir
/usr/sbin
--master-dist-count
3},
{error=255}]<br>
[2021-03-11
19:15:47.621776]
E
[syncdutils(worker
/data/storage_a/storage):851:logerr] Popen: ssh> Killed by signal 15.<br>
[2021-03-11
19:15:47.621819]
E
[syncdutils(worker
/data/storage_c/storage):851:logerr] Popen: ssh> Killed by signal 15.<br>
[2021-03-11
19:15:47.621850]
E
[syncdutils(worker
/data/storage_b/storage):325:log_raise_exception] <top>:
connection to
peer is broken<br>
[2021-03-11
19:15:47.622437]
E
[syncdutils(worker
/data/storage_b/storage):847:errlog] Popen: command returned error
[{cmd=ssh
-oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem
-p 22
-oControlMaster=auto
-S
/tmp/gsyncd-aux-ssh-Vy935W/e15fc58bb13552de0710eaf018209548.sock
<a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.82"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.82</a> /nonexistent/gsyncd slave storage <a
rel="nofollow
noopener
noreferrer"
shape="rect"
href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> --master-node 10.0.231.91
--master-node-id
afc24654-2887-41f6-a9c2-8e835de243b6 --master-brick
/data/storage_b/storage
--local-node
10.0.231.82
--local-node-id
be50a8de-3934-4fee-a80d-8e2e99017902 --slave-timeout 120
--slave-log-level
INFO
--slave-gluster-log-level
INFO
--slave-gluster-command-dir
/usr/sbin
--master-dist-count
3},
{error=255}]<br>
[2021-03-11
19:15:47.622556]
E
[syncdutils(worker
/data/storage_b/storage):851:logerr] Popen: ssh> Killed by signal 15.<br>
[2021-03-11
19:15:47.723756]
I
[monitor(monitor):228:monitor]
Monitor:
worker died in
startup phase
[{brick=/data/storage_a/storage}]<br>
[2021-03-11
19:15:47.731405]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Faulty}]<br>
[2021-03-11
19:15:47.825223]
I
[monitor(monitor):228:monitor]
Monitor:
worker died in
startup phase
[{brick=/data/storage_c/storage}]<br>
[2021-03-11
19:15:47.825685]
I
[monitor(monitor):228:monitor]
Monitor:
worker died in
startup phase
[{brick=/data/storage_b/storage}]<br>
[2021-03-11
19:15:47.829011]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Faulty}]<br>
[2021-03-11
19:15:47.830965]
I
[gsyncdstatus(monitor):248:set_worker_status]
GeorepStatus:
Worker Status
Change
[{status=Faulty}]<br>
[2021-03-11
19:15:48.669634]
D
[gsyncd(monitor-status):303:main]
<top>:
Using session
config file
[{path=/var/lib/glusterd/geo-replication/storage_10.0.231.81_pcic-backup/gsyncd.conf}]<br>
[2021-03-11
19:15:48.683784]
I
[subcmds(monitor-status):29:subcmd_monitor_status]
<top>:
Monitor Status
Change
[{status=Stopped}]<br>
<br>
<br>
Thanks,<br>
-Matthew<br>
<div>
<p><br>
</p>
</div>
<div>On
3/11/21 9:37
AM, Strahil
Nikolov wrote:<br>
</div>
<blockquote
type="cite">
<pre>Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.
I think you have to increase the debug logs for geo-rep session.
I will try to find the command necessary to increase it.
Best Regards,
Strahil Nikolov
В четвъртък, 11 март 2021 г., 00:38:41 ч. Гринуич+2, Matthew Benstead <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:matthewb@uvic.ca" target="_blank" moz-do-not-send="true"><matthewb@uvic.ca></a> написа:
Thanks Strahil,
Right - I had come across your message in early January that v8 from the CentOS Sig was missing the SELinux rules, and had put SELinux into permissive mode after the upgrade when I saw denied messages in the audit logs.
[root@storage01 ~]# sestatus | egrep "^SELinux status|[mM]ode"
SELinux status: enabled
Current mode: permissive
Mode from config file: enforcing
Yes - I am using an unprivileged user for georep:
[root@pcic-backup01 ~]# gluster-mountbroker status
+-------------+-------------+---------------------------+--------------+--------------------------+
| NODE | NODE STATUS | MOUNT ROOT | GROUP | USERS |
+-------------+-------------+---------------------------+--------------+--------------------------+
| 10.0.231.82 | UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup) |
| localhost | UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup) |
+-------------+-------------+---------------------------+--------------+--------------------------+
[root@pcic-backup02 ~]# gluster-mountbroker status
+-------------+-------------+---------------------------+--------------+--------------------------+
| NODE | NODE STATUS | MOUNT ROOT | GROUP | USERS |
+-------------+-------------+---------------------------+--------------+--------------------------+
| 10.0.231.81 | UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup) |
| localhost | UP | /var/mountbroker-root(OK) | geogroup(OK) | geoaccount(pcic-backup) |
+-------------+-------------+---------------------------+--------------+--------------------------+
Thanks,
-Matthew
--
Matthew Benstead
System AdministratorPacific Climate Impacts ConsortiumUniversity of Victoria, UH1PO Box 1800, STN CSCVictoria, BC, V8W 2Y2Phone: +1-250-721-8432Email: <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:matthewb@uvic.ca" target="_blank" moz-do-not-send="true">matthewb@uvic.ca</a>
On 3/10/21 2:11 PM, Strahil Nikolov wrote:
</pre>
<blockquote
type="cite">
<pre>
Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.
I have tested georep on v8.3 and it was running quite well untill you involve SELINUX.
Are you using SELINUX ?
Are you using unprivileged user for the georep ?
Also, you can check <a rel="nofollow noopener noreferrer" shape="rect" href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication" target="_blank" moz-do-not-send="true">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication</a> .
Best Regards,
Strahil Nikolov
</pre>
<blockquote
type="cite">
<pre>
On Thu, Mar 11, 2021 at 0:03, Matthew Benstead
<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:matthewb@uvic.ca" target="_blank" moz-do-not-send="true"><matthewb@uvic.ca></a> wrote:
Hello,
I recently upgraded my Distributed-Replicate cluster from Gluster 7.9 to 8.3 on CentOS7 using the CentOS Storage SIG packages. I had geo-replication syncing properly before the upgrade, but not it is not working after.
After I had upgraded both master and slave clusters I attempted to start geo-replication again, but it goes to faulty quickly:
[root@storage01 ~]# gluster volume geo-replication storage <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">geoaccount@10.0.231.81::pcic-backup</a> start
Starting geo-replication session between storage & <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">geoaccount@10.0.231.81::pcic-backup</a> has been successful\
[root@storage01 ~]# gluster volume geo-replication status
MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.0.231.91 storage /data/storage_a/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
10.0.231.91 storage /data/storage_c/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
10.0.231.91 storage /data/storage_b/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
10.0.231.92 storage /data/storage_b/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
10.0.231.92 storage /data/storage_a/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
10.0.231.92 storage /data/storage_c/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
10.0.231.93 storage /data/storage_c/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
10.0.231.93 storage /data/storage_b/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
10.0.231.93 storage /data/storage_a/storage geoaccount <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">ssh://geoaccount@10.0.231.81::pcic-backup</a> N/A Faulty N/A N/A
[root@storage01 ~]# gluster volume geo-replication storage <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">geoaccount@10.0.231.81::pcic-backup</a> stop
Stopping geo-replication session between storage & <a rel="nofollow noopener noreferrer" shape="rect" href="mailto:geoaccount@10.0.231.81::pcic-backup" target="_blank" moz-do-not-send="true">geoaccount@10.0.231.81::pcic-backup</a> has been successful
I went through the gsyncd logs and see it attempts to go back through the changlogs - which would make sense - but fails:
[2021-03-10 19:18:42.165807] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:42.166136] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
[2021-03-10 19:18:42.167829] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:42.172343] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:42.172580] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:42.235574] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:42.236613] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:42.238614] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:44.144856] I [resource(worker /data/storage_b/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9059}]
[2021-03-10 19:18:44.145065] I [resource(worker /data/storage_b/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:44.162873] I [resource(worker /data/storage_a/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9259}]
[2021-03-10 19:18:44.163412] I [resource(worker /data/storage_a/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:44.167506] I [resource(worker /data/storage_c/storage):1436:connect_remote] SSH: SSH connection between master and slave established. [{duration=1.9316}]
[2021-03-10 19:18:44.167746] I [resource(worker /data/storage_c/storage):1116:connect] GLUSTER: Mounting gluster volume locally...
[2021-03-10 19:18:45.251372] I [resource(worker /data/storage_b/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1062}]
[2021-03-10 19:18:45.251583] I [subcmds(worker /data/storage_b/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:45.271950] I [resource(worker /data/storage_c/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1041}]
[2021-03-10 19:18:45.272118] I [subcmds(worker /data/storage_c/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:45.275180] I [resource(worker /data/storage_a/storage):1139:connect] GLUSTER: Mounted gluster volume [{duration=1.1116}]
[2021-03-10 19:18:45.275361] I [subcmds(worker /data/storage_a/storage):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2021-03-10 19:18:47.265618] I [master(worker /data/storage_b/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]
[2021-03-10 19:18:47.265954] I [resource(worker /data/storage_b/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.276746] I [gsyncdstatus(worker /data/storage_b/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.281194] I [gsyncdstatus(worker /data/storage_b/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.281404] I [master(worker /data/storage_b/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664113, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.285340] I [master(worker /data/storage_c/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]
[2021-03-10 19:18:47.285579] I [resource(worker /data/storage_c/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.287383] I [master(worker /data/storage_a/storage):1645:register] _GMaster: Working dir [{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]
[2021-03-10 19:18:47.287697] I [resource(worker /data/storage_a/storage):1292:service_loop] GLUSTER: Register time [{time=1615403927}]
[2021-03-10 19:18:47.298415] I [gsyncdstatus(worker /data/storage_c/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.301342] I [gsyncdstatus(worker /data/storage_a/storage):281:set_active] GeorepStatus: Worker Status Change [{status=Active}]
[2021-03-10 19:18:47.304183] I [gsyncdstatus(worker /data/storage_c/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.304418] I [master(worker /data/storage_c/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666552, 0)}, {entry_stime=(1614664108, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.305294] E [resource(worker /data/storage_c/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:47.308124] I [gsyncdstatus(worker /data/storage_a/storage):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change [{status=History Crawl}]
[2021-03-10 19:18:47.308509] I [master(worker /data/storage_a/storage):1559:crawl] _GMaster: starting history crawl [{turns=1}, {stime=(1614666553, 0)}, {entry_stime=(1614664115, 0)}, {etime=1615403927}]
[2021-03-10 19:18:47.357470] E [resource(worker /data/storage_b/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:47.383949] E [resource(worker /data/storage_a/storage):1312:service_loop] GLUSTER: Changelog History Crawl failed [{error=[Errno 0] Success}]
[2021-03-10 19:18:48.255340] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_b/storage}]
[2021-03-10 19:18:48.260052] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:48.275651] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_c/storage}]
[2021-03-10 19:18:48.278064] I [monitor(monitor):228:monitor] Monitor: worker died in startup phase [{brick=/data/storage_a/storage}]
[2021-03-10 19:18:48.280453] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:48.282274] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Faulty}]
[2021-03-10 19:18:58.275702] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.276041] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_b/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:58.296252] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.296506] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_c/storage}, {slave_node=10.0.231.82}]
[2021-03-10 19:18:58.301290] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change [{status=Initializing...}]
[2021-03-10 19:18:58.301521] I [monitor(monitor):160:monitor] Monitor: starting gsyncd worker [{brick=/data/storage_a/storage}, {slave_node=10.0.231.81}]
[2021-03-10 19:18:58.345817] I [resource(worker /data/storage_b/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:58.361268] I [resource(worker /data/storage_c/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:58.367985] I [resource(worker /data/storage_a/storage):1387:connect_remote] SSH: Initializing SSH connection between master and slave...
[2021-03-10 19:18:59.115143] I [subcmds(monitor-status):29:subcmd_monitor_status] <top>: Monitor Status Change [{status=Stopped}]
It seems like there is an issue selecting the changelogs - perhaps similar to this issue? <a rel="nofollow noopener noreferrer" shape="rect" href="https://github.com/gluster/glusterfs/issues/1766" target="_blank" moz-do-not-send="true">https://github.com/gluster/glusterfs/issues/1766</a>
[root@storage01 storage_10.0.231.81_pcic-backup]# cat changes-data-storage_a-storage.log
[2021-03-10 19:18:45.284764] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_a/storage}, {notify_filter=1}]
[2021-03-10 19:18:45.285275] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
[2021-03-10 19:18:45.285269] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
[2021-03-10 19:18:45.286615] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 21
[2021-03-10 19:18:47.308607] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666553}, {end=1615403927}]
[2021-03-10 19:18:47.308659] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1250878}]
[2021-03-10 19:18:47.383774] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1250877}]
[root@storage01 storage_10.0.231.81_pcic-backup]# tail -7 changes-data-storage_b-storage.log
[2021-03-10 19:18:45.263211] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
[2021-03-10 19:18:45.263151] I [MSGID: 132028] [gf-changelog.c:577:gf_changelog_register_generic] 0-gfchangelog: Registering brick [{brick=/data/storage_b/storage}, {notify_filter=1}]
[2021-03-10 19:18:45.263294] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
[2021-03-10 19:18:45.264598] I [socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing (AF_UNIX) reuse check socket 23
[2021-03-10 19:18:47.281499] I [MSGID: 132035] [gf-history-changelog.c:837:gf_history_changelog] 0-gfchangelog: Requesting historical changelogs [{start=1614666552}, {end=1615403927}]
[2021-03-10 19:18:47.281551] I [MSGID: 132019] [gf-history-changelog.c:755:gf_changelog_extract_min_max] 0-gfchangelog: changelogs min max [{min=1597342860}, {max=1615403927}, {total_changelogs=1258258}]
[2021-03-10 19:18:47.357244] E [MSGID: 132009] [gf-history-changelog.c:941:gf_history_changelog] 0-gfchangelog: wrong result [{for=end}, {start=1615403927}, {idx=1258257}]
Any ideas on where to debug this? I'd prefer not to have to remove and re-sync everything as there is about 240TB on the cluster...
Thanks,
-Matthew
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: <a rel="nofollow noopener noreferrer" shape="rect" href="https://meet.google.com/cpu-eiue-hvk" target="_blank" moz-do-not-send="true">https://meet.google.com/cpu-eiue-hvk</a>
Gluster-users mailing list
<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a rel="nofollow noopener noreferrer" shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00
UTC<br>
Bridge: <a rel="nofollow noopener noreferrer"
href="https://meet.google.com/cpu-eiue-hvk"
target="_blank" moz-do-not-send="true">
https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a rel="nofollow noopener noreferrer"
href="mailto:Gluster-users@gluster.org"
target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a rel="nofollow noopener noreferrer"
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</body>
</html>