This is your problem : bad server has only 3 clients.<div><br></div><div>I remember there is another gluster volume command to list the IPs of the clients. Find it and run it to find which clients are actually OK (those 3) and the remaining 17 are not. </div><div><br></div><div>Then try to remount those 17 clients and if the situation persistes - work with your Network Team to identify why the 17 clients can't reach the brick.</div><div><br></div><div>Do you have selfheal enabled?</div><div><pre><i>cluster.data-self-heal
</i><i>cluster.entry-self-heal
</i>cluster.metadata-self-heal</pre><pre><br></pre><pre>Best Regards,</pre><pre>Strahil Nikolov</pre></div><div> <blockquote style="margin: 0 0 20px 0;"> <div style="font-family:Roboto, sans-serif; color:#6D00F6;"> <div>On Mon, Jan 29, 2024 at 10:26, Hu Bert</div><div><revirii@googlemail.com> wrote:</div> </div> <div style="padding: 10px 0 0 20px; margin: 10px 0 0 0; border-left: 1px solid #6D00F6;"> <div dir="ltr">Hi,<br></div><div dir="ltr">not sure what you mean with "clients" - do you mean the clients that<br></div><div dir="ltr">mount the volume?<br></div><div dir="ltr"><br></div><div dir="ltr">gluster volume status workdata clients<br></div><div dir="ltr">----------------------------------------------<br></div><div dir="ltr">Brick : glusterpub2:/gluster/md3/workdata<br></div><div dir="ltr">Clients connected : 20<br></div><div dir="ltr">Hostname BytesRead<br></div><div dir="ltr">BytesWritten OpVersion<br></div><div dir="ltr">-------- ---------<br></div><div dir="ltr">------------ ---------<br></div><div dir="ltr">192.168.0.222:49140 43698212<br></div><div dir="ltr"> 41152108 110000<br></div><div dir="ltr">[...shortened...]<br></div><div dir="ltr">192.168.0.126:49123 8362352021<br></div><div dir="ltr">16445401205 110000<br></div><div dir="ltr">----------------------------------------------<br></div><div dir="ltr">Brick : glusterpub3:/gluster/md3/workdata<br></div><div dir="ltr">Clients connected : 3<br></div><div dir="ltr">Hostname BytesRead<br></div><div dir="ltr">BytesWritten OpVersion<br></div><div dir="ltr">-------- ---------<br></div><div dir="ltr">------------ ---------<br></div><div dir="ltr">192.168.0.44:49150 5855740279<br></div><div dir="ltr">63649538575 110000<br></div><div dir="ltr">192.168.0.44:49137 308958200<br></div><div dir="ltr">319216608 110000<br></div><div dir="ltr">192.168.0.126:49120 7524915770<br></div><div dir="ltr">15489813449 110000<br></div><div dir="ltr"><br></div><div dir="ltr">192.168.0.44 (glusterpub3) is the "bad" server. Not sure what you mean<br></div><div dir="ltr">by "old" - probably not the age of the server, but rather the gluster<br></div><div dir="ltr">version. op-version is 110000 on all servers+clients, upgraded from<br></div><div dir="ltr">10.4 -> 11.1<br></div><div dir="ltr"><br></div><div dir="ltr">"Have you checked if a client is not allowed to update all 3 copies ?"<br></div><div dir="ltr">-> are there special log messages for that?<br></div><div dir="ltr"><br></div><div dir="ltr">"If it's only 1 system, you can remove the brick, reinitialize it and<br></div><div dir="ltr">then bring it back for a full sync."<br></div><div dir="ltr">-> <a href="https://docs.gluster.org/en/v3/Administrator%20Guide/Managing%20Volumes/#replace-brick" target="_blank">https://docs.gluster.org/en/v3/Administrator%20Guide/Managing%20Volumes/#replace-brick</a><br></div><div dir="ltr">-> Replacing bricks in Replicate/Distributed Replicate volumes<br></div><div dir="ltr"><br></div><div dir="ltr">this part, right? Well, can't do this right now, as there are ~33TB of<br></div><div dir="ltr">data (many small files) to copy, that would slow down the servers /<br></div><div dir="ltr">the volume. But if the replacement is running i could do it<br></div><div dir="ltr">afterwards, just to see what happens.<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Hubert<br></div><div dir="ltr"><br></div><div dir="ltr">Am Mo., 29. Jan. 2024 um 08:21 Uhr schrieb Strahil Nikolov<br></div><div dir="ltr"><<a ymailto="mailto:hunter86_bg@yahoo.com" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>>:<br></div><div dir="ltr">><br></div><div dir="ltr">> 2800 is too much. Most probably you are affected by a bug. How old are the clients ? Is only 1 server affected ?<br></div><div dir="ltr">> Have you checked if a client is not allowed to update all 3 copies ?<br></div><div dir="ltr">><br></div><div dir="ltr">> If it's only 1 system, you can remove the brick, reinitialize it and then bring it back for a full sync.<br></div><div dir="ltr">><br></div><div dir="ltr">> Best Regards,<br></div><div dir="ltr">> Strahil Nikolov<br></div><div dir="ltr">><br></div><div dir="ltr">> On Mon, Jan 29, 2024 at 8:44, Hu Bert<br></div><div dir="ltr">> <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>> wrote:<br></div><div dir="ltr">> Morning,<br></div><div dir="ltr">> a few bad apples - but which ones? Checked glustershd.log on the "bad"<br></div><div dir="ltr">> server and counted todays "gfid mismatch" entries (2800 in total):<br></div><div dir="ltr">><br></div><div dir="ltr">> 44 <gfid:faeea007-2f41-4a72-959f-e9e14e6a9ea4>/212>,<br></div><div dir="ltr">> 44 <gfid:faeea007-2f41-4a72-959f-e9e14e6a9ea4>/174>,<br></div><div dir="ltr">> 44 <gfid:d5c6d7b9-f217-4cc9-a664-448d034e74c2>/94037803>,<br></div><div dir="ltr">> 44 <gfid:d263ecc2-9c21-455c-9ba9-5a999c03adce>/94066216>,<br></div><div dir="ltr">> 44 <gfid:cbfd5d46-d580-4845-a544-e46fd82c1758>/249771609>,<br></div><div dir="ltr">> 44 <gfid:aecf217a-0797-43d1-9481-422a8ac8a5d0>/64235523>,<br></div><div dir="ltr">> 44 <gfid:a701d47b-b3fb-4e7e-bbfb-bc3e19632867>/185>,<br></div><div dir="ltr">><br></div><div dir="ltr">> etc. But as i said, these are pretty new and didn't appear when the<br></div><div dir="ltr">> volume/servers started missbehaving. Are there scripts/snippets<br></div><div dir="ltr">> available how one could handle this?<br></div><div dir="ltr">><br></div><div dir="ltr">> Healing would be very painful for the running system (still connected,<br></div><div dir="ltr">> but not very long anymore), as there surely are 4-5 million entries to<br></div><div dir="ltr">> be healed. I can't do this now - maybe, when the replacement is in<br></div><div dir="ltr">> productive state, one could give it a try.<br></div><div dir="ltr">><br></div><div dir="ltr">> Thx,<br></div><div dir="ltr">> Hubert<br></div><div dir="ltr">><br></div><div dir="ltr">> Am So., 28. Jan. 2024 um 23:12 Uhr schrieb Strahil Nikolov<br></div><div dir="ltr">> <<a ymailto="mailto:hunter86_bg@yahoo.com" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>>:<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > From gfid mismatch a manual effort is needed but you can script it.<br></div><div dir="ltr">> > I think that a few bad "apples" can break the healing and if you fix them the healing might be recovered.<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Also, check why the client is not updating all copies. Most probably you have a client that is not able to connect to a brick.<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > gluster volume status VOLUME_NAME clients<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Best Regards,<br></div><div dir="ltr">> > Strahil Nikolov<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > On Sun, Jan 28, 2024 at 20:55, Hu Bert<br></div><div dir="ltr">> > <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>> wrote:<br></div><div dir="ltr">> > Hi Strahil,<br></div><div dir="ltr">> > there's no arbiter: 3 servers with 5 bricks each.<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Volume Name: workdata<br></div><div dir="ltr">> > Type: Distributed-Replicate<br></div><div dir="ltr">> > Volume ID: 7d1e23e5-0308-4443-a832-d36f85ff7959<br></div><div dir="ltr">> > Status: Started<br></div><div dir="ltr">> > Snapshot Count: 0<br></div><div dir="ltr">> > Number of Bricks: 5 x 3 = 15<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > The "problem" is: the number of files/entries to-be-healed has<br></div><div dir="ltr">> > continuously grown since the beginning, and now we're talking about<br></div><div dir="ltr">> > way too many files to do this manually. Last time i checked: 700K per<br></div><div dir="ltr">> > brick, should be >900K at the moment. The command 'gluster volume heal<br></div><div dir="ltr">> > workdata statistics heal-count' is unable to finish. Doesn't look that<br></div><div dir="ltr">> > good :D<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Interesting, the glustershd.log on the "bad" server now shows errors like these:<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > [2024-01-28 18:48:33.734053 +0000] E [MSGID: 108008]<br></div><div dir="ltr">> > [afr-self-heal-common.c:399:afr_gfid_split_brain_source]<br></div><div dir="ltr">> > 0-workdata-replicate-3: Gfid mismatch detected for<br></div><div dir="ltr">> > <gfid:70ab3d57-bd82-4932-86bf-d613db32c1ab>/803620716>,<br></div><div dir="ltr">> > 82d7939a-8919-40ea-<br></div><div dir="ltr">> > 9459-7b8af23d3b72 on workdata-client-11 and<br></div><div dir="ltr">> > bb9399a3-0a5c-4cd1-b2b1-3ee787ec835a on workdata-client-9<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Shouldn't the heals happen on the 2 "good" servers?<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Anyway... we're currently preparing a different solution for our data<br></div><div dir="ltr">> > and we'll throw away this gluster volume - no critical data will be<br></div><div dir="ltr">> > lost, as these are derived from source data (on a different volume on<br></div><div dir="ltr">> > different servers). Will be a hard time (calculating tons of data),<br></div><div dir="ltr">> > but the chosen solution should have a way better performance.<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Well... thx to all for your efforts, really appreciate that :-)<br></div><div dir="ltr">> ><br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Hubert<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > Am So., 28. Jan. 2024 um 08:35 Uhr schrieb Strahil Nikolov<br></div><div dir="ltr">> > <<a ymailto="mailto:hunter86_bg@yahoo.com" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>>:<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > What about the arbiter node ?<br></div><div dir="ltr">> > > Actually, check on all nodes and script it - you might need it in the future.<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > Simplest way to resolve is to make the file didappear (rename to something else and then rename it back). Another easy trick is to read thr whole file: dd if=file of=/dev/null status=progress<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > Best Regards,<br></div><div dir="ltr">> > > Strahil Nikolov<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > On Sat, Jan 27, 2024 at 8:24, Hu Bert<br></div><div dir="ltr">> > > <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>> wrote:<br></div><div dir="ltr">> > > Morning,<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > gfid1:<br></div><div dir="ltr">> > > getfattr -d -e hex -m.<br></div><div dir="ltr">> > > /gluster/md{3,4,5,6,7}/workdata/.glusterfs/fa/f5/faf59566-10f5-4ddd-8b0c-a87bc6a334fb<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > glusterpub1 (good one):<br></div><div dir="ltr">> > > getfattr: Removing leading '/' from absolute path names<br></div><div dir="ltr">> > > # file: gluster/md6/workdata/.glusterfs/fa/f5/faf59566-10f5-4ddd-8b0c-a87bc6a334fb<br></div><div dir="ltr">> > > trusted.afr.dirty=0x000000000000000000000000<br></div><div dir="ltr">> > > trusted.afr.workdata-client-11=0x000000020000000100000000<br></div><div dir="ltr">> > > trusted.gfid=0xfaf5956610f54ddd8b0ca87bc6a334fb<br></div><div dir="ltr">> > > trusted.gfid2path.c2845024cc9b402e=0x38633139626234612d396236382d343532652d623434652d3664616331666434616465652f31323878313238732e6a7067<br></div><div dir="ltr">> > > trusted.glusterfs.mdata=0x0100000000000000000000000065aaecff000000002695ebb70000000065aaecff000000002695ebb70000000065aaecff000000002533f110<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > glusterpub3 (bad one):<br></div><div dir="ltr">> > > getfattr: /gluster/md6/workdata/.glusterfs/fa/f5/faf59566-10f5-4ddd-8b0c-a87bc6a334fb:<br></div><div dir="ltr">> > > No such file or directory<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > gfid 2:<br></div><div dir="ltr">> > > getfattr -d -e hex -m.<br></div><div dir="ltr">> > > /gluster/md{3,4,5,6,7}/workdata/.glusterfs/60/46/60465723-5dc0-4ebe-aced-9f2c12e52642<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > glusterpub1 (good one):<br></div><div dir="ltr">> > > getfattr: Removing leading '/' from absolute path names<br></div><div dir="ltr">> > > # file: gluster/md5/workdata/.glusterfs/60/46/60465723-5dc0-4ebe-aced-9f2c12e52642<br></div><div dir="ltr">> > > trusted.afr.dirty=0x000000000000000000000000<br></div><div dir="ltr">> > > trusted.afr.workdata-client-8=0x000000020000000100000000<br></div><div dir="ltr">> > > trusted.gfid=0x604657235dc04ebeaced9f2c12e52642<br></div><div dir="ltr">> > > trusted.gfid2path.ac4669e3c4faf926=0x33366463366137392d666135642d343238652d613738642d6234376230616662316562642f31323878313238732e6a7067<br></div><div dir="ltr">> > > trusted.glusterfs.mdata=0x0100000000000000000000000065aaecfe000000000c5403bd0000000065aaecfe000000000c5403bd0000000065aaecfe000000000ad61ee4<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > glusterpub3 (bad one):<br></div><div dir="ltr">> > > getfattr: /gluster/md5/workdata/.glusterfs/60/46/60465723-5dc0-4ebe-aced-9f2c12e52642:<br></div><div dir="ltr">> > > No such file or directory<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > thx,<br></div><div dir="ltr">> > > Hubert<br></div><div dir="ltr">> > ><br></div><div dir="ltr">> > > Am Sa., 27. Jan. 2024 um 06:13 Uhr schrieb Strahil Nikolov<br></div><div dir="ltr">> > > <<a ymailto="mailto:hunter86_bg@yahoo.com" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>>:<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > You don't need to mount it.<br></div><div dir="ltr">> > > > Like this :<br></div><div dir="ltr">> > > > # getfattr -d -e hex -m. /path/to/brick/.glusterfs/00/46/00462be8-3e61-4931-8bda-dae1645c639e<br></div><div dir="ltr">> > > > # file: 00/46/00462be8-3e61-4931-8bda-dae1645c639e<br></div><div dir="ltr">> > > > trusted.gfid=0x00462be83e6149318bdadae1645c639e<br></div><div dir="ltr">> > > > trusted.gfid2path.05fcbdafdeea18ab=0x30326333373930632d386637622d346436652d393464362d3936393132313930643131312f66696c656c6f636b696e672e7079<br></div><div dir="ltr">> > > > trusted.glusterfs.mdata=0x010000000000000000000000006170340c0000000025b6a745000000006170340c0000000020efb577000000006170340c0000000020d42b07<br></div><div dir="ltr">> > > > trusted.glusterfs.shard.block-size=0x0000000004000000<br></div><div dir="ltr">> > > > trusted.glusterfs.shard.file-size=0x00000000000000cd000000000000000000000000000000010000000000000000<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > Best Regards,<br></div><div dir="ltr">> > > > Strahil Nikolov<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > В четвъртък, 25 януари 2024 г. в 09:42:46 ч. Гринуич+2, Hu Bert <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>> написа:<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > Good morning,<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > hope i got it right... using:<br></div><div dir="ltr">> > > > <a href="https://access.redhat.com/documentation/de-de/red_hat_gluster_storage/3.1/html/administration_guide/ch27s02" target="_blank">https://access.redhat.com/documentation/de-de/red_hat_gluster_storage/3.1/html/administration_guide/ch27s02</a><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > mount -t glusterfs -o aux-gfid-mount glusterpub1:/workdata /mnt/workdata<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > gfid 1:<br></div><div dir="ltr">> > > > getfattr -n trusted.glusterfs.pathinfo -e text<br></div><div dir="ltr">> > > > /mnt/workdata/.gfid/faf59566-10f5-4ddd-8b0c-a87bc6a334fb<br></div><div dir="ltr">> > > > getfattr: Removing leading '/' from absolute path names<br></div><div dir="ltr">> > > > # file: mnt/workdata/.gfid/faf59566-10f5-4ddd-8b0c-a87bc6a334fb<br></div><div dir="ltr">> > > > trusted.glusterfs.pathinfo="(<DISTRIBUTE:workdata-dht><br></div><div dir="ltr">> > > > (<REPLICATE:workdata-replicate-3><br></div><div dir="ltr">> > > > <POSIX(/gluster/md6/workdata):glusterpub1:/gluster/md6/workdata/images/133/283/13328349/128x128s.jpg><br></div><div dir="ltr">> > > > <POSIX(/gluster/md6/workdata):glusterpub2:/gl<br></div><div dir="ltr">> > > > uster/md6/workdata/images/133/283/13328349/128x128s.jpg>))"<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > gfid 2:<br></div><div dir="ltr">> > > > getfattr -n trusted.glusterfs.pathinfo -e text<br></div><div dir="ltr">> > > > /mnt/workdata/.gfid/60465723-5dc0-4ebe-aced-9f2c12e52642<br></div><div dir="ltr">> > > > getfattr: Removing leading '/' from absolute path names<br></div><div dir="ltr">> > > > # file: mnt/workdata/.gfid/60465723-5dc0-4ebe-aced-9f2c12e52642<br></div><div dir="ltr">> > > > trusted.glusterfs.pathinfo="(<DISTRIBUTE:workdata-dht><br></div><div dir="ltr">> > > > (<REPLICATE:workdata-replicate-2><br></div><div dir="ltr">> > > > <POSIX(/gluster/md5/workdata):glusterpub2:/gluster/md5/workdata/.glusterfs/60/46/60465723-5dc0-4ebe-aced-9f2c12e52642><br></div><div dir="ltr">> > > > <POSIX(/gluster/md5/workdata<br></div><div dir="ltr">> > > > ):glusterpub1:/gluster/md5/workdata/.glusterfs/60/46/60465723-5dc0-4ebe-aced-9f2c12e52642>))"<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > glusterpub1 + glusterpub2 are the good ones, glusterpub3 is the<br></div><div dir="ltr">> > > > misbehaving (not healing) one.<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > The file with gfid 1 is available under<br></div><div dir="ltr">> > > > /gluster/md6/workdata/images/133/283/13328349/ on glusterpub1+2<br></div><div dir="ltr">> > > > bricks, but missing on glusterpub3 brick.<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > gfid 2: /gluster/md5/workdata/.glusterfs/60/46/60465723-5dc0-4ebe-aced-9f2c12e52642<br></div><div dir="ltr">> > > > is present on glusterpub1+2, but not on glusterpub3.<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > Thx,<br></div><div dir="ltr">> > > > Hubert<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > Am Mi., 24. Jan. 2024 um 17:36 Uhr schrieb Strahil Nikolov<br></div><div dir="ltr">> > > > <<a ymailto="mailto:hunter86_bg@yahoo.com" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>>:<br></div><div dir="ltr">> > > ><br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > Hi,<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > Can you find and check the files with gfids:<br></div><div dir="ltr">> > > > > 60465723-5dc0-4ebe-aced-9f2c12e52642<br></div><div dir="ltr">> > > > > faf59566-10f5-4ddd-8b0c-a87bc6a334fb<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > Use 'getfattr -d -e hex -m. ' command from <a href="https://docs.gluster.org/en/main/Troubleshooting/resolving-splitbrain/#analysis-of-the-output" target="_blank">https://docs.gluster.org/en/main/Troubleshooting/resolving-splitbrain/#analysis-of-the-output</a> .<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > Best Regards,<br></div><div dir="ltr">> > > > > Strahil Nikolov<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > On Sat, Jan 20, 2024 at 9:44, Hu Bert<br></div><div dir="ltr">> > > > > <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>> wrote:<br></div><div dir="ltr">> > > > > Good morning,<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > thx Gilberto, did the first three (set to WARNING), but the last one<br></div><div dir="ltr">> > > > > doesn't work. Anyway, with setting these three some new messages<br></div><div dir="ltr">> > > > > appear:<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > [2024-01-20 07:23:58.561106 +0000] W [MSGID: 114061]<br></div><div dir="ltr">> > > > > [client-common.c:796:client_pre_lk_v2] 0-workdata-client-11: remote_fd<br></div><div dir="ltr">> > > > > is -1. EBADFD [{gfid=faf59566-10f5-4ddd-8b0c-a87bc6a334fb},<br></div><div dir="ltr">> > > > > {errno=77}, {error=File descriptor in bad state}]<br></div><div dir="ltr">> > > > > [2024-01-20 07:23:58.561177 +0000] E [MSGID: 108028]<br></div><div dir="ltr">> > > > > [afr-open.c:361:afr_is_reopen_allowed_cbk] 0-workdata-replicate-3:<br></div><div dir="ltr">> > > > > Failed getlk for faf59566-10f5-4ddd-8b0c-a87bc6a334fb [File descriptor<br></div><div dir="ltr">> > > > > in bad state]<br></div><div dir="ltr">> > > > > [2024-01-20 07:23:58.562151 +0000] W [MSGID: 114031]<br></div><div dir="ltr">> > > > > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-11:<br></div><div dir="ltr">> > > > > remote operation failed.<br></div><div dir="ltr">> > > > > [{path=<gfid:faf59566-10f5-4ddd-8b0c-a87bc6a334fb>},<br></div><div dir="ltr">> > > > > {gfid=faf59566-10f5-4ddd-8b0c-a87b<br></div><div dir="ltr">> > > > > c6a334fb}, {errno=2}, {error=No such file or directory}]<br></div><div dir="ltr">> > > > > [2024-01-20 07:23:58.562296 +0000] W [MSGID: 114061]<br></div><div dir="ltr">> > > > > [client-common.c:530:client_pre_flush_v2] 0-workdata-client-11:<br></div><div dir="ltr">> > > > > remote_fd is -1. EBADFD [{gfid=faf59566-10f5-4ddd-8b0c-a87bc6a334fb},<br></div><div dir="ltr">> > > > > {errno=77}, {error=File descriptor in bad state}]<br></div><div dir="ltr">> > > > > [2024-01-20 07:23:58.860552 +0000] W [MSGID: 114061]<br></div><div dir="ltr">> > > > > [client-common.c:796:client_pre_lk_v2] 0-workdata-client-8: remote_fd<br></div><div dir="ltr">> > > > > is -1. EBADFD [{gfid=60465723-5dc0-4ebe-aced-9f2c12e52642},<br></div><div dir="ltr">> > > > > {errno=77}, {error=File descriptor in bad state}]<br></div><div dir="ltr">> > > > > [2024-01-20 07:23:58.860608 +0000] E [MSGID: 108028]<br></div><div dir="ltr">> > > > > [afr-open.c:361:afr_is_reopen_allowed_cbk] 0-workdata-replicate-2:<br></div><div dir="ltr">> > > > > Failed getlk for 60465723-5dc0-4ebe-aced-9f2c12e52642 [File descriptor<br></div><div dir="ltr">> > > > > in bad state]<br></div><div dir="ltr">> > > > > [2024-01-20 07:23:58.861520 +0000] W [MSGID: 114031]<br></div><div dir="ltr">> > > > > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-8:<br></div><div dir="ltr">> > > > > remote operation failed.<br></div><div dir="ltr">> > > > > [{path=<gfid:60465723-5dc0-4ebe-aced-9f2c12e52642>},<br></div><div dir="ltr">> > > > > {gfid=60465723-5dc0-4ebe-aced-9f2c1<br></div><div dir="ltr">> > > > > 2e52642}, {errno=2}, {error=No such file or directory}]<br></div><div dir="ltr">> > > > > [2024-01-20 07:23:58.861640 +0000] W [MSGID: 114061]<br></div><div dir="ltr">> > > > > [client-common.c:530:client_pre_flush_v2] 0-workdata-client-8:<br></div><div dir="ltr">> > > > > remote_fd is -1. EBADFD [{gfid=60465723-5dc0-4ebe-aced-9f2c12e52642},<br></div><div dir="ltr">> > > > > {errno=77}, {error=File descriptor in bad state}]<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > Not many log entries appear, only a few. Has someone seen error<br></div><div dir="ltr">> > > > > messages like these? Setting diagnostics.brick-sys-log-level to DEBUG<br></div><div dir="ltr">> > > > > shows way more log entries, uploaded it to:<br></div><div dir="ltr">> > > > > <a href="https://file.io/spLhlcbMCzr8" target="_blank">https://file.io/spLhlcbMCzr8</a> - not sure if that helps.<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > Thx,<br></div><div dir="ltr">> > > > > Hubert<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > Am Fr., 19. Jan. 2024 um 16:24 Uhr schrieb Gilberto Ferreira<br></div><div dir="ltr">> > > > > <<a ymailto="mailto:gilberto.nunes32@gmail.com" href="mailto:gilberto.nunes32@gmail.com">gilberto.nunes32@gmail.com</a>>:<br></div><div dir="ltr">> > > > ><br></div><div dir="ltr">> > > > > ><br></div><div dir="ltr">> > > > > > gluster volume set testvol diagnostics.brick-log-level WARNING<br></div><div dir="ltr">> > > > > > gluster volume set testvol diagnostics.brick-sys-log-level WARNING<br></div><div dir="ltr">> > > > > > gluster volume set testvol diagnostics.client-log-level ERROR<br></div><div dir="ltr">> > > > > > gluster --log-level=ERROR volume status<br></div><div dir="ltr">> > > > > ><br></div><div dir="ltr">> > > > > > ---<br></div><div dir="ltr">> > > > > > Gilberto Nunes Ferreira<br></div><div dir="ltr">> > > > > ><br></div><div dir="ltr">> > > > > ><br></div><div dir="ltr">> > > > > ><br></div><div dir="ltr">> > > > > ><br></div><div dir="ltr">> > > > > ><br></div><div dir="ltr">> > > > > ><br></div><div dir="ltr">> > > > > > Em sex., 19 de jan. de 2024 às 05:49, Hu Bert <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>> escreveu:<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> Hi Strahil,<br></div><div dir="ltr">> > > > > >> hm, don't get me wrong, it may sound a bit stupid, but... where do i<br></div><div dir="ltr">> > > > > >> set the log level? Using debian...<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> <a href="https://access.redhat.com/documentation/de-de/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level" target="_blank">https://access.redhat.com/documentation/de-de/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level</a><br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> ls /etc/glusterfs/<br></div><div dir="ltr">> > > > > >> eventsconfig.json glusterfs-georep-logrotate<br></div><div dir="ltr">> > > > > >> gluster-rsyslog-5.8.conf group-db-workload group-gluster-block<br></div><div dir="ltr">> > > > > >> group-nl-cache group-virt.example logger.conf.example<br></div><div dir="ltr">> > > > > >> glusterd.vol glusterfs-logrotate<br></div><div dir="ltr">> > > > > >> gluster-rsyslog-7.2.conf group-distributed-virt group-metadata-cache<br></div><div dir="ltr">> > > > > >> group-samba gsyncd.conf thin-arbiter.vol<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> checked: /etc/glusterfs/logger.conf.example<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> # To enable enhanced logging capabilities,<br></div><div dir="ltr">> > > > > >> #<br></div><div dir="ltr">> > > > > >> # 1. rename this file to /etc/glusterfs/logger.conf<br></div><div dir="ltr">> > > > > >> #<br></div><div dir="ltr">> > > > > >> # 2. rename /etc/rsyslog.d/gluster.conf.example to<br></div><div dir="ltr">> > > > > >> # /etc/rsyslog.d/gluster.conf<br></div><div dir="ltr">> > > > > >> #<br></div><div dir="ltr">> > > > > >> # This change requires restart of all gluster services/volumes and<br></div><div dir="ltr">> > > > > >> # rsyslog.<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> tried (to test): /etc/glusterfs/logger.conf with " LOG_LEVEL='WARNING' "<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> restart glusterd on that node, but this doesn't work, log-level stays<br></div><div dir="ltr">> > > > > >> on INFO. /etc/rsyslog.d/gluster.conf.example does not exist. Probably<br></div><div dir="ltr">> > > > > >> /etc/rsyslog.conf on debian. But first it would be better to know<br></div><div dir="ltr">> > > > > >> where to set the log-level for glusterd.<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> Depending on how much the DEBUG log-level talks ;-) i could assign up<br></div><div dir="ltr">> > > > > >> to 100G to /var<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> Thx & best regards,<br></div><div dir="ltr">> > > > > >> Hubert<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> Am Do., 18. Jan. 2024 um 22:58 Uhr schrieb Strahil Nikolov<br></div><div dir="ltr">> > > > > >> <<a ymailto="mailto:hunter86_bg@yahoo.com" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>>:<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > Are you able to set the logs to debug level ?<br></div><div dir="ltr">> > > > > >> > It might provide a clue what it is going on.<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > Best Regards,<br></div><div dir="ltr">> > > > > >> > Strahil Nikolov<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > On Thu, Jan 18, 2024 at 13:08, Diego Zuccato<br></div><div dir="ltr">> > > > > >> > <<a ymailto="mailto:diego.zuccato@unibo.it" href="mailto:diego.zuccato@unibo.it">diego.zuccato@unibo.it</a>> wrote:<br></div><div dir="ltr">> > > > > >> > That's the same kind of errors I keep seeing on my 2 clusters,<br></div><div dir="ltr">> > > > > >> > regenerated some months ago. Seems a pseudo-split-brain that should be<br></div><div dir="ltr">> > > > > >> > impossible on a replica 3 cluster but keeps happening.<br></div><div dir="ltr">> > > > > >> > Sadly going to ditch Gluster ASAP.<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > Diego<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > Il 18/01/2024 07:11, Hu Bert ha scritto:<br></div><div dir="ltr">> > > > > >> > > Good morning,<br></div><div dir="ltr">> > > > > >> > > heal still not running. Pending heals now sum up to 60K per brick.<br></div><div dir="ltr">> > > > > >> > > Heal was starting instantly e.g. after server reboot with version<br></div><div dir="ltr">> > > > > >> > > 10.4, but doesn't with version 11. What could be wrong?<br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > > I only see these errors on one of the "good" servers in glustershd.log:<br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > > [2024-01-18 06:08:57.328480 +0000] W [MSGID: 114031]<br></div><div dir="ltr">> > > > > >> > > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-0:<br></div><div dir="ltr">> > > > > >> > > remote operation failed.<br></div><div dir="ltr">> > > > > >> > > [{path=<gfid:cb39a1e4-2a4c-4727-861d-3ed9ef00681b>},<br></div><div dir="ltr">> > > > > >> > > {gfid=cb39a1e4-2a4c-4727-861d-3ed9e<br></div><div dir="ltr">> > > > > >> > > f00681b}, {errno=2}, {error=No such file or directory}]<br></div><div dir="ltr">> > > > > >> > > [2024-01-18 06:08:57.594051 +0000] W [MSGID: 114031]<br></div><div dir="ltr">> > > > > >> > > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-1:<br></div><div dir="ltr">> > > > > >> > > remote operation failed.<br></div><div dir="ltr">> > > > > >> > > [{path=<gfid:3e9b178c-ae1f-4d85-ae47-fc539d94dd11>},<br></div><div dir="ltr">> > > > > >> > > {gfid=3e9b178c-ae1f-4d85-ae47-fc539<br></div><div dir="ltr">> > > > > >> > > d94dd11}, {errno=2}, {error=No such file or directory}]<br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > > About 7K today. Any ideas? Someone?<br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > > Best regards,<br></div><div dir="ltr">> > > > > >> > > Hubert<br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > > Am Mi., 17. Jan. 2024 um 11:24 Uhr schrieb Hu Bert <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>>:<br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >> ok, finally managed to get all servers, volumes etc runnung, but took<br></div><div dir="ltr">> > > > > >> > >> a couple of restarts, cksum checks etc.<br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >> One problem: a volume doesn't heal automatically or doesn't heal at all.<br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >> gluster volume status<br></div><div dir="ltr">> > > > > >> > >> Status of volume: workdata<br></div><div dir="ltr">> > > > > >> > >> Gluster process TCP Port RDMA Port Online Pid<br></div><div dir="ltr">> > > > > >> > >> ------------------------------------------------------------------------------<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub1:/gluster/md3/workdata 58832 0 Y 3436<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub2:/gluster/md3/workdata 59315 0 Y 1526<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub3:/gluster/md3/workdata 56917 0 Y 1952<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub1:/gluster/md4/workdata 59688 0 Y 3755<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub2:/gluster/md4/workdata 60271 0 Y 2271<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub3:/gluster/md4/workdata 49461 0 Y 2399<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub1:/gluster/md5/workdata 54651 0 Y 4208<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub2:/gluster/md5/workdata 49685 0 Y 2751<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub3:/gluster/md5/workdata 59202 0 Y 2803<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub1:/gluster/md6/workdata 55829 0 Y 4583<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub2:/gluster/md6/workdata 50455 0 Y 3296<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub3:/gluster/md6/workdata 50262 0 Y 3237<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub1:/gluster/md7/workdata 52238 0 Y 5014<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub2:/gluster/md7/workdata 52474 0 Y 3673<br></div><div dir="ltr">> > > > > >> > >> Brick glusterpub3:/gluster/md7/workdata 57966 0 Y 3653<br></div><div dir="ltr">> > > > > >> > >> Self-heal Daemon on localhost N/A N/A Y 4141<br></div><div dir="ltr">> > > > > >> > >> Self-heal Daemon on glusterpub1 N/A N/A Y 5570<br></div><div dir="ltr">> > > > > >> > >> Self-heal Daemon on glusterpub2 N/A N/A Y 4139<br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >> "gluster volume heal workdata info" lists a lot of files per brick.<br></div><div dir="ltr">> > > > > >> > >> "gluster volume heal workdata statistics heal-count" shows thousands<br></div><div dir="ltr">> > > > > >> > >> of files per brick.<br></div><div dir="ltr">> > > > > >> > >> "gluster volume heal workdata enable" has no effect.<br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >> gluster volume heal workdata full<br></div><div dir="ltr">> > > > > >> > >> Launching heal operation to perform full self heal on volume workdata<br></div><div dir="ltr">> > > > > >> > >> has been successful<br></div><div dir="ltr">> > > > > >> > >> Use heal info commands to check status.<br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >> -> not doing anything at all. And nothing happening on the 2 "good"<br></div><div dir="ltr">> > > > > >> > >> servers in e.g. glustershd.log. Heal was working as expected on<br></div><div dir="ltr">> > > > > >> > >> version 10.4, but here... silence. Someone has an idea?<br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >> Best regards,<br></div><div dir="ltr">> > > > > >> > >> Hubert<br></div><div dir="ltr">> > > > > >> > >><br></div><div dir="ltr">> > > > > >> > >> Am Di., 16. Jan. 2024 um 13:44 Uhr schrieb Gilberto Ferreira<br></div><div dir="ltr">> > > > > >> > >> <<a ymailto="mailto:gilberto.nunes32@gmail.com" href="mailto:gilberto.nunes32@gmail.com">gilberto.nunes32@gmail.com</a>>:<br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>> Ah! Indeed! You need to perform an upgrade in the clients as well.<br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>><br></div><div dir="ltr">> > > > > >> > >>> Em ter., 16 de jan. de 2024 às 03:12, Hu Bert <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>> escreveu:<br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>> morning to those still reading :-)<br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>> i found this: <a href="https://docs.gluster.org/en/main/Troubleshooting/troubleshooting-glusterd/#common-issues-and-how-to-resolve-them" target="_blank">https://docs.gluster.org/en/main/Troubleshooting/troubleshooting-glusterd/#common-issues-and-how-to-resolve-them</a><br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>> there's a paragraph about "peer rejected" with the same error message,<br></div><div dir="ltr">> > > > > >> > >>>> telling me: "Update the cluster.op-version" - i had only updated the<br></div><div dir="ltr">> > > > > >> > >>>> server nodes, but not the clients. So upgrading the cluster.op-version<br></div><div dir="ltr">> > > > > >> > >>>> wasn't possible at this time. So... upgrading the clients to version<br></div><div dir="ltr">> > > > > >> > >>>> 11.1 and then the op-version should solve the problem?<br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>> Thx,<br></div><div dir="ltr">> > > > > >> > >>>> Hubert<br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>> Am Mo., 15. Jan. 2024 um 09:16 Uhr schrieb Hu Bert <<a ymailto="mailto:revirii@googlemail.com" href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>>:<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> Hi,<br></div><div dir="ltr">> > > > > >> > >>>>> just upgraded some gluster servers from version 10.4 to version 11.1.<br></div><div dir="ltr">> > > > > >> > >>>>> Debian bullseye & bookworm. When only installing the packages: good,<br></div><div dir="ltr">> > > > > >> > >>>>> servers, volumes etc. work as expected.<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> But one needs to test if the systems work after a daemon and/or server<br></div><div dir="ltr">> > > > > >> > >>>>> restart. Well, did a reboot, and after that the rebooted/restarted<br></div><div dir="ltr">> > > > > >> > >>>>> system is "out". Log message from working node:<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> [2024-01-15 08:02:21.585694 +0000] I [MSGID: 106163]<br></div><div dir="ltr">> > > > > >> > >>>>> [glusterd-handshake.c:1501:__glusterd_mgmt_hndsk_versions_ack]<br></div><div dir="ltr">> > > > > >> > >>>>> 0-management: using the op-version 100000<br></div><div dir="ltr">> > > > > >> > >>>>> [2024-01-15 08:02:21.589601 +0000] I [MSGID: 106490]<br></div><div dir="ltr">> > > > > >> > >>>>> [glusterd-handler.c:2546:__glusterd_handle_incoming_friend_req]<br></div><div dir="ltr">> > > > > >> > >>>>> 0-glusterd: Received probe from uuid:<br></div><div dir="ltr">> > > > > >> > >>>>> b71401c3-512a-47cb-ac18-473c4ba7776e<br></div><div dir="ltr">> > > > > >> > >>>>> [2024-01-15 08:02:23.608349 +0000] E [MSGID: 106010]<br></div><div dir="ltr">> > > > > >> > >>>>> [glusterd-utils.c:3824:glusterd_compare_friend_volume] 0-management:<br></div><div dir="ltr">> > > > > >> > >>>>> Version of Cksums sourceimages differ. local cksum = 2204642525,<br></div><div dir="ltr">> > > > > >> > >>>>> remote cksum = 1931483801 on peer gluster190<br></div><div dir="ltr">> > > > > >> > >>>>> [2024-01-15 08:02:23.608584 +0000] I [MSGID: 106493]<br></div><div dir="ltr">> > > > > >> > >>>>> [glusterd-handler.c:3819:glusterd_xfer_friend_add_resp] 0-glusterd:<br></div><div dir="ltr">> > > > > >> > >>>>> Responded to gluster190 (0), ret: 0, op_ret: -1<br></div><div dir="ltr">> > > > > >> > >>>>> [2024-01-15 08:02:23.613553 +0000] I [MSGID: 106493]<br></div><div dir="ltr">> > > > > >> > >>>>> [glusterd-rpc-ops.c:467:__glusterd_friend_add_cbk] 0-glusterd:<br></div><div dir="ltr">> > > > > >> > >>>>> Received RJT from uuid: b71401c3-512a-47cb-ac18-473c4ba7776e, host:<br></div><div dir="ltr">> > > > > >> > >>>>> gluster190, port: 0<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> peer status from rebooted node:<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> <a ymailto="mailto:root@gluster190" href="mailto:root@gluster190">root@gluster190</a> ~ # gluster peer status<br></div><div dir="ltr">> > > > > >> > >>>>> Number of Peers: 2<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> Hostname: gluster189<br></div><div dir="ltr">> > > > > >> > >>>>> Uuid: 50dc8288-aa49-4ea8-9c6c-9a9a926c67a7<br></div><div dir="ltr">> > > > > >> > >>>>> State: Peer Rejected (Connected)<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> Hostname: gluster188<br></div><div dir="ltr">> > > > > >> > >>>>> Uuid: e15a33fe-e2f7-47cf-ac53-a3b34136555d<br></div><div dir="ltr">> > > > > >> > >>>>> State: Peer Rejected (Connected)<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> So the rebooted gluster190 is not accepted anymore. And thus does not<br></div><div dir="ltr">> > > > > >> > >>>>> appear in "gluster volume status". I then followed this guide:<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> <a href="https://gluster-documentations.readthedocs.io/en/latest/Administrator%20Guide/Resolving%20Peer%20Rejected/" target="_blank">https://gluster-documentations.readthedocs.io/en/latest/Administrator%20Guide/Resolving%20Peer%20Rejected/</a><br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> Remove everything under /var/lib/glusterd/ (except glusterd.info) and<br></div><div dir="ltr">> > > > > >> > >>>>> restart glusterd service etc. Data get copied from other nodes,<br></div><div dir="ltr">> > > > > >> > >>>>> 'gluster peer status' is ok again - but the volume info is missing,<br></div><div dir="ltr">> > > > > >> > >>>>> /var/lib/glusterd/vols is empty. When syncing this dir from another<br></div><div dir="ltr">> > > > > >> > >>>>> node, the volume then is available again, heals start etc.<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> Well, and just to be sure that everything's working as it should,<br></div><div dir="ltr">> > > > > >> > >>>>> rebooted that node again - the rebooted node is kicked out again, and<br></div><div dir="ltr">> > > > > >> > >>>>> you have to restart bringing it back again.<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> Sry, but did i miss anything? Has someone experienced similar<br></div><div dir="ltr">> > > > > >> > >>>>> problems? I'll probably downgrade to 10.4 again, that version was<br></div><div dir="ltr">> > > > > >> > >>>>> working...<br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>><br></div><div dir="ltr">> > > > > >> > >>>>> Thx,<br></div><div dir="ltr">> > > > > >> > >>>>> Hubert<br></div><div dir="ltr">> > > > > >> > >>>> ________<br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>> Community Meeting Calendar:<br></div><div dir="ltr">> > > > > >> > >>>><br></div><div dir="ltr">> > > > > >> > >>>> Schedule -<br></div><div dir="ltr">> > > > > >> > >>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br></div><div dir="ltr">> > > > > >> > >>>> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br></div><div dir="ltr">> > > > > >> > >>>> Gluster-users mailing list<br></div><div dir="ltr">> > > > > >> > >>>> <a ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br></div><div dir="ltr">> > > > > >> > >>>> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></div><div dir="ltr">> > > > > >> > > ________<br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > > Community Meeting Calendar:<br></div><div dir="ltr">> > > > > >> > ><br></div><div dir="ltr">> > > > > >> > > Schedule -<br></div><div dir="ltr">> > > > > >> > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br></div><div dir="ltr">> > > > > >> > > Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br></div><div dir="ltr">> > > > > >> > > Gluster-users mailing list<br></div><div dir="ltr">> > > > > >> > > <a ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br></div><div dir="ltr">> > > > > >> > > <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > --<br></div><div dir="ltr">> > > > > >> > Diego Zuccato<br></div><div dir="ltr">> > > > > >> > DIFA - Dip. di Fisica e Astronomia<br></div><div dir="ltr">> > > > > >> > Servizi Informatici<br></div><div dir="ltr">> > > > > >> > Alma Mater Studiorum - Università di Bologna<br></div><div dir="ltr">> > > > > >> > V.le Berti-Pichat 6/2 - 40127 Bologna - Italy<br></div><div dir="ltr">> > > > > >> > tel.: +39 051 20 95786<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > ________<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > Community Meeting Calendar:<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > Schedule -<br></div><div dir="ltr">> > > > > >> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br></div><div dir="ltr">> > > > > >> > Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br></div><div dir="ltr">> > > > > >> > Gluster-users mailing list<br></div><div dir="ltr">> > > > > >> > <a ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br></div><div dir="ltr">> > > > > >> > <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > ________<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > Community Meeting Calendar:<br></div><div dir="ltr">> > > > > >> ><br></div><div dir="ltr">> > > > > >> > Schedule -<br></div><div dir="ltr">> > > > > >> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br></div><div dir="ltr">> > > > > >> > Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br></div><div dir="ltr">> > > > > >> > Gluster-users mailing list<br></div><div dir="ltr">> > > > > >> > <a ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br></div><div dir="ltr">> > > > > >> > <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></div><div dir="ltr">> > > > > >> ________<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> Community Meeting Calendar:<br></div><div dir="ltr">> > > > > >><br></div><div dir="ltr">> > > > > >> Schedule -<br></div><div dir="ltr">> > > > > >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br></div><div dir="ltr">> > > > > >> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br></div><div dir="ltr">> > > > > >> Gluster-users mailing list<br></div><div dir="ltr">> > > > > >> <a ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br></div><div dir="ltr">> > > > > >> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></div> </div> </blockquote></div>