[Gluster-users] Filled up file system and glusterd not working anymore. To probe or not to probe?

[Ilias Chasapakis forumZFD]

Dear all,

first of all thanks to Stefan (Kania) for the feedback. So detaching the 
peer, removing brick and downgrading to replica 2 until you add a new 
brick as you would do with any otherwise faulty node.

So here is how we proceeded:

  * check mounts where the faulty (ex-filled-up) node is present and
    remove it from there
  * detach "bad" peer with:  gluster peer detach <your_faulty_gluster_node>
  * remove brick: gluster volume remove-brick <vol> replica 2
    <your_faulty_gluster_node>:<the_brick_path> force

So in our case we are ready to add a new brick to restore our replica 3.

This should be safer (and in the end as simple) than just copying the 
missing files probably bringing along meta-data inconsistencies. Which 
was also the doubt expressed on this post.

Am 08.01.24 um 11:13 schrieb Ilias Chasapakis forumZFD:
> Dear all,
>
> we have a replica 3 configuration an issue after that the file system 
> filled up. The gluster daemon is not starting anymore on the affected 
> node and the inconsistency that we noticed is that the 
> /var/lib/glusterd/peers does contain only one "good" node and the 
> other one is missing.
>
> Now what we would like to try is to probe from the affected peer the 
> missing one. We also thought about just copying the missing node 
> config file /var/lib/glusterd/peers here but we think this might be 
> more inconsistent because perhaps of the time passed between finding 
> out the issue and when the fill up had occurred (if this has 
> relevance). We assume probing will fill that missing entry in the 
> affected gluster and will sync to it. What we want to avoid is that 
> this happens the other way around.
>
> So in brief the question is if in our situation probing is too risky 
> and we should try other methods (removing-adding brick) or anything 
> that you might want to suggest from your experience.
>
> Many thanks in advance
>
> Ilias
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge:https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

-- 
forumZFD
Entschieden für Frieden | Committed to Peace

Ilias Chasapakis
Referent IT | IT Consultant

Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am Kölner Brett 8 | 50825 Köln | Germany

Tel 0221 91273243 | Fax 0221 91273299 |http://www.forumZFD.de

Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht Köln

Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00   BIC GENODEM1GLS

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240109/159de7a9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 665 bytes
Desc: OpenPGP digital signature
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240109/159de7a9/attachment.sig>