<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office"><head><!--[if gte mso 9]><xml><o:OfficeDocumentSettings><o:AllowPNG/><o:PixelsPerInch>96</o:PixelsPerInch></o:OfficeDocumentSettings></xml><![endif]--></head><body>
As I mentioned, I’ve never had such a case.<div>Give it a try in a test environment and if it works - go ahead.</div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov <br><br><p class="yahoo-quoted-begin" style="font-size: 15px; color: #715FFA; padding-top: 15px; margin-top: 0">On Wednesday, August 16, 2023, 1:21 PM, Sebastian Neustein <sebastian.neustein@arc-aachen.de> wrote:</p><blockquote class="iosymail"><div id="yiv2370176778"><div>
<div class="yiv2370176778moz-cite-prefix">Strahil Nikolov:<br clear="none">
</div>
<blockquote type="cite">
I’ve never had such situation and I don’t recall someone sharing
something similar.</blockquote>
<br clear="none">
<br clear="none">
That's strange, it is really easy to reproduce. This is from a fresh
test environment:<br clear="none">
<br clear="none">
summary:<br clear="none">
- There is one snapshot present. <br clear="none">
- On one node glusterd is stopped. <br clear="none">
- During the stop, one snapshot is deleted. <br clear="none">
- The node is brought up again<br clear="none">
- On that node there is an orphaned snapshot<br clear="none">
<br clear="none">
<br clear="none">
detailed version:<br clear="none">
# on node 1:<br clear="none">
root@gl1:~# cat /etc/debian_version<br clear="none">
11.7<br clear="none">
<br clear="none">
root@gl1:~# gluster --version<br clear="none">
glusterfs 10.4<br clear="none">
<br clear="none">
root@gl1:~# gluster volume info<br clear="none">
Volume Name: glvol_samba<br clear="none">
Type: Replicate<br clear="none">
Volume ID: 91cb059e-10e4-4439-92ea-001065652749<br clear="none">
Status: Started<br clear="none">
Snapshot Count: 1<br clear="none">
Number of Bricks: 1 x 3 = 3<br clear="none">
Transport-type: tcp<br clear="none">
Bricks:<br clear="none">
Brick1: gl1:/data/glusterfs/glvol_samba/brick0/brick<br clear="none">
Brick2: gl2:/data/glusterfs/glvol_samba/brick0/brick<br clear="none">
Brick3: gl3:/data/glusterfs/glvol_samba/brick0/brick<br clear="none">
Options Reconfigured:<br clear="none">
cluster.granular-entry-heal: on<br clear="none">
storage.fips-mode-rchecksum: on<br clear="none">
transport.address-family: inet<br clear="none">
nfs.disable: on<br clear="none">
performance.client-io-threads: off<br clear="none">
features.barrier: disable<br clear="none">
<br clear="none">
root@gl1:~# gluster snapshot list<br clear="none">
snaps_GMT-2023.08.15-13.05.28<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
# on node 3:<br clear="none">
root@gl3:~# systemctl stop glusterd.service<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
# on node 1:<br clear="none">
root@gl1:~# gluster snapshot deactivate
snaps_GMT-2023.08.15-13.05.28<br clear="none">
Deactivating snap will make its data inaccessible. Do you want to
continue? (y/n) y<br clear="none">
Snapshot deactivate: snaps_GMT-2023.08.15-13.05.28: Snap deactivated
successfully<br clear="none">
<br clear="none">
root@gl1:~# gluster snapshot delete snaps_GMT-2023.08.15-13.05.28<br clear="none">
Deleting snap will erase all the information about the snap. Do you
still want to continue? (y/n) y<br clear="none">
snapshot delete: snaps_GMT-2023.08.15-13.05.28: snap removed
successfully<br clear="none">
<br clear="none">
root@gl1:~# gluster snapshot list<br clear="none">
No snapshots present<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
# on node 3:<br clear="none">
root@gl3:~# systemctl start glusterd.service<br clear="none">
<br clear="none">
root@gl3:~# gluster snapshot list<br clear="none">
snaps_GMT-2023.08.15-13.05.28<br clear="none">
<br clear="none">
root@gl3:~# gluster snapshot deactivate
snaps_GMT-2023.08.15-13.05.28<br clear="none">
Deactivating snap will make its data inaccessible. Do you want to
continue? (y/n) y<br clear="none">
snapshot deactivate: failed: Pre Validation failed on gl1.ad.arc.de.
Snapshot (snaps_GMT-2023.08.15-13.05.28) does not exist.<br clear="none">
Pre Validation failed on gl2. Snapshot
(snaps_GMT-2023.08.15-13.05.28) does not exist.<br clear="none">
Snapshot command failed<br clear="none">
<br clear="none">
root@gl3:~# lvs -a<br clear="none">
LV VG Attr LSize
Pool Origin Data% Meta% Move Log Cpy%Sync Convert<br clear="none">
669cbc14fa7542acafb2995666284583_0 vg_brick0 Vwi-aotz-- 15,00g
tp_brick0 lv_brick0 0,08<br clear="none">
lv_brick0 vg_brick0 Vwi-aotz-- 15,00g
tp_brick0 0,08<br clear="none">
[lvol0_pmspare] vg_brick0 ewi------- 20,00m<br clear="none">
tp_brick0 vg_brick0 twi-aotz--
18,00g 0,12 10,57<br clear="none">
[tp_brick0_tdata] vg_brick0 Twi-ao---- 18,00g<br clear="none">
[tp_brick0_tmeta] vg_brick0 ewi-ao---- 20,00m<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
Would it be dangerous to just delete following items on node 3 while
gluster is down:<br clear="none">
- the orphaned directories in /var/lib/glusterd/snaps/<br clear="none">
- the orphaned lvm, here 669cbc14fa7542acafb2995666284583_0 <br clear="none">
<br clear="none">
Or is there a self-heal command?<br clear="none">
<br clear="none">
Regards<br clear="none">
Sebastian<br clear="none">
<br clear="none">
<div class="yiv2370176778moz-cite-prefix">Am 10.08.2023 um 20:33 schrieb Strahil
Nikolov:<br clear="none">
</div>
<blockquote type="cite">
</blockquote></div><div id="yiv2370176778yqt07030" class="yiv2370176778yqt4182361365"><div>
I’ve never had such situation and I don’t recall someone sharing
something similar.
<div><span style=""><br clear="none">
</span></div>
<div><span style="">Most probably
it’s easier to remove the node from the TSP and re-add it.</span></div>
<div>
<div>Of course , test the case in VMs just to validate that it’s
possible to add a mode to a cluster with snapshots.</div>
<div><br clear="none">
</div>
<div>I have a vague feeling that you will need to delete all
snapshots.</div>
<div>
<div><br clear="none">
</div>
<div>Best Regards,</div>
<div>Strahil Nikolov </div>
<div><br clear="none">
<p style="font-size:15px;color:#715FFA;padding-top:15px;margin-top:0;" class="yiv2370176778yahoo-quoted-begin">On
Thursday, August 10, 2023, 4:36 AM, Sebastian Neustein
<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:sebastian.neustein@arc-aachen.de" target="_blank" href="mailto:sebastian.neustein@arc-aachen.de" class="yiv2370176778moz-txt-link-rfc2396E"><sebastian.neustein@arc-aachen.de></a> wrote:</p>
<blockquote class="yiv2370176778iosymail">
<div id="yiv2370176778">
<div>
<div class="yiv2370176778c-message_kit__blocks yiv2370176778c-message_kit__blocks--rich_text">
<div class="yiv2370176778c-message__message_blocks yiv2370176778c-message__message_blocks--rich_text">
<div class="yiv2370176778p-block_kit_renderer">
<div class="yiv2370176778p-block_kit_renderer__block_wrapper yiv2370176778p-block_kit_renderer__block_wrapper--first">
<div class="yiv2370176778p-rich_text_block">
<div class="yiv2370176778p-rich_text_section">Hi<br clear="none">
<br clear="none">
Due to an outage of one node, after
bringing it up again, the node has some
orphaned snapshosts, which are already
deleted on the other nodes. <br clear="none">
<br clear="none">
<span class="yiv2370176778c-mrkdwn__br"></span>How
can I delete these orphaned snapshots?
Trying the normal way produceses these
errors:<br clear="none">
<code class="yiv2370176778c-mrkdwn__code">[2023-08-08
19:34:03.667109 +0000] E [MSGID: 106115]
[glusterd-mgmt.c:118:gd_mgmt_v3_collate_errors] 0-management: Pre
Validation failed on B742. Please check
log file for details.</code><br clear="none">
<code class="yiv2370176778c-mrkdwn__code">[2023-08-08
19:34:03.667184 +0000] E [MSGID: 106115]
[glusterd-mgmt.c:118:gd_mgmt_v3_collate_errors] 0-management: Pre
Validation failed on B741. Please check
log file for details.</code><br clear="none">
<code class="yiv2370176778c-mrkdwn__code">[2023-08-08
19:34:03.667210 +0000] E [MSGID: 106121]
[glusterd-mgmt.c:1083:glusterd_mgmt_v3_pre_validate] 0-management: Pre
Validation failed on peers</code><br clear="none">
<code class="yiv2370176778c-mrkdwn__code">[2023-08-08
19:34:03.667236 +0000] E [MSGID: 106121]
[glusterd-mgmt.c:2875:glusterd_mgmt_v3_initiate_snap_phases]
0-management: Pre Validation Failed</code><span class="yiv2370176778c-mrkdwn__br"></span><br clear="none">
<br clear="none">
Even worse: I followed <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/troubleshooting_snapshots" class="yiv2370176778c-link">read hat gluster
snapshot trouble guide</a> and deleted
one of those directories defining a
snapshot. Now I receive this on the cli:<br clear="none">
<code class="yiv2370176778c-mrkdwn__code">run-gluster-snaps-e4dcd4166538414c849fa91b0b3934d7-brick6-brick[297342]:
[2023-08-09 08:59:41.107243 +0000] M
[MSGID: 113075]
[posix-helpers.c:2161:posix_health_check_thread_proc]
0-e4dcd4166538414c849fa91b0b3934d7-posix: health-check failed, going
down</code><br clear="none">
<code class="yiv2370176778c-mrkdwn__code">run-gluster-snaps-e4dcd4166538414c849fa91b0b3934d7-brick6-brick[297342]:
[2023-08-09 08:59:41.107243 +0000] M
[MSGID: 113075]
[posix-helpers.c:2161:posix_health_check_thread_proc]
0-e4dcd4166538414c849fa91b0b3934d7-posix: health-check failed, going
down</code><br clear="none">
<code class="yiv2370176778c-mrkdwn__code">run-gluster-snaps-e4dcd4166538414c849fa91b0b3934d7-brick6-brick[297342]:
[2023-08-09 08:59:41.107292 +0000] M
[MSGID: 113075]
[posix-helpers.c:2179:posix_health_check_thread_proc]
0-e4dcd4166538414c849fa91b0b3934d7-posix: still alive! -> SIGTERM</code><br clear="none">
<code class="yiv2370176778c-mrkdwn__code">run-gluster-snaps-e4dcd4166538414c849fa91b0b3934d7-brick6-brick[297342]:
[2023-08-09 08:59:41.107292 +0000] M
[MSGID: 113075]
[posix-helpers.c:2179:posix_health_check_thread_proc]
0-e4dcd4166538414c849fa91b0b3934d7-posix: still alive! -> SIGTERM</code><span class="yiv2370176778c-mrkdwn__br"></span><br clear="none">
<br clear="none">
What are my options? <br clear="none">
- is there an easy way to remove all those
snapshots?<br clear="none">
- or would it be easier to remove and
rejoin the node to the gluster cluster?<br clear="none">
<br clear="none">
Thank you for any help!<br clear="none">
<br clear="none">
Seb<br clear="none">
<span dir="ltr" class="yiv2370176778c-message__edited_label"></span></div>
</div>
</div>
</div>
</div>
</div>
<pre class="yiv2370176778moz-signature">--
Sebastian Neustein
Airport Research Center GmbH
Bismarckstraße 61
52066 Aachen
Germany
Phone: +49 241 16843-23
Fax: +49 241 16843-19
e-mail: <a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:sebastian.neustein@arc-aachen.de" target="_blank" href="mailto:sebastian.neustein@arc-aachen.de" class="yiv2370176778moz-txt-link-abbreviated yiv2370176778moz-txt-link-freetext">sebastian.neustein@arc-aachen.de</a>
Website: <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="http://www.airport-consultants.com" class="yiv2370176778moz-txt-link-freetext yiv2370176778moz-txt-link-freetext">http://www.airport-consultants.com</a>
Register Court: Amtsgericht Aachen HRB 7313
Ust-Id-No.: DE196450052
Managing Director:
Dipl.-Ing. Tom Alexander Heuer</pre>
</div>
</div>
________<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
Community Meeting Calendar:<br clear="none">
<br clear="none">
Schedule -<br clear="none">
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">
Bridge: <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://meet.google.com/cpu-eiue-hvk" class="yiv2370176778moz-txt-link-freetext">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">
Gluster-users mailing list<br clear="none">
<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:Gluster-users@gluster.org" target="_blank" href="mailto:Gluster-users@gluster.org" class="yiv2370176778moz-txt-link-freetext">Gluster-users@gluster.org</a><br clear="none">
<a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://lists.gluster.org/mailman/listinfo/gluster-users" class="yiv2370176778moz-txt-link-freetext">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none">
</blockquote>
</div>
</div>
</div>
<br clear="none">
</div></div></div><blockquote></blockquote></blockquote></div>
</body></html>