<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Thanks Hari.<br></div><div><br data-mce-bogus="1"></div><div>Firstly <!--StartFragment-->there is no more rejected node today.<!--EndFragment--> <br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>which volfile must be replicated on all nodes ? This ones ? All of them must be the same ?<br data-mce-bogus="1"></div><div>2b4a85d2fef7661be0f38b670b69cd02 node1/vols/tmp/tmp.glusterDevVM1.bricks-tmp-brick1-data.vol<br>dec273968c263e3aa12a8be073bea534 node2/vols/tmp/tmp.glusterDevVM1.bricks-tmp-brick1-data.vol<br>dec273968c263e3aa12a8be073bea534 node3/vols/tmp/tmp.glusterDevVM1.bricks-tmp-brick1-data.vol<br>dec273968c263e3aa12a8be073bea534 node1/vols/tmp/tmp.glusterDevVM2.bricks-tmp-brick1-data.vol<br>2b4a85d2fef7661be0f38b670b69cd02 node2/vols/tmp/tmp.glusterDevVM2.bricks-tmp-brick1-data.vol<br>dec273968c263e3aa12a8be073bea534 node3/vols/tmp/tmp.glusterDevVM2.bricks-tmp-brick1-data.vol<br>a71fabf46ea02ce41c5cc7c3dd3cb86d node1/vols/tmp/tmp.glusterDevVM3.bricks-tmp-brick1-data.vol<br>a71fabf46ea02ce41c5cc7c3dd3cb86d node2/vols/tmp/tmp.glusterDevVM3.bricks-tmp-brick1-data.vol<br>3a6a8cb4804be3453826dd9bdf865859 node3/vols/tmp/tmp.glusterDevVM3.bricks-tmp-brick1-data.vol<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div># diff node1/vols/tmp/tmp.glusterDevVM1.bricks-tmp-brick1-data.vol node2/vols/tmp/tmp.glusterDevVM1.bricks-tmp-brick1-data.vol<br>5c5<br>< option shared-brick-count 1<br>---<br>> option shared-brick-count 0<br><br data-mce-bogus="1"></div><div>Example of another volume :</div><div>2aebab43f5829785e53704328f965f01 node1/vols/rrddata/rrddata.glusterDevVM1.bricks-rrddata-brick1-data.vol<br>35a0ada7ba6542e2ac1bd566f84603c6 node2/vols/rrddata/rrddata.glusterDevVM1.bricks-rrddata-brick1-data.vol<br>35a0ada7ba6542e2ac1bd566f84603c6 node3/vols/rrddata/rrddata.glusterDevVM1.bricks-rrddata-brick1-data.vol<br>35a0ada7ba6542e2ac1bd566f84603c6 node1/vols/rrddata/rrddata.glusterDevVM2.bricks-rrddata-brick1-data.vol<br>2aebab43f5829785e53704328f965f01 node2/vols/rrddata/rrddata.glusterDevVM2.bricks-rrddata-brick1-data.vol<br>35a0ada7ba6542e2ac1bd566f84603c6 node3/vols/rrddata/rrddata.glusterDevVM2.bricks-rrddata-brick1-data.vol<br>5b6b3453be116a2e1489c6ffc6b6fa86 node1/vols/rrddata/rrddata.glusterDevVM3.bricks-rrddata-brick1-data.vol<br>5b6b3453be116a2e1489c6ffc6b6fa86 node2/vols/rrddata/rrddata.glusterDevVM3.bricks-rrddata-brick1-data.vol<br>158d3a073596133643c634c1ecd603ba node3/vols/rrddata/rrddata.glusterDevVM3.bricks-rrddata-brick1-data.vol<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Best regards,<br data-mce-bogus="1"></div><div>Nicolas.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>De: </b>"Hari Gowtham" <hgowtham@redhat.com><br><b>À: </b>nico@furyweb.fr, "Sanju Rakonde" <srakonde@redhat.com><br><b>Cc: </b>"Nikhil Ladha" <nladha@redhat.com>, "gluster-users" <gluster-users@gluster.org><br><b>Envoyé: </b>Mercredi 29 Avril 2020 14:11:29<br><b>Objet: </b>Re: [Gluster-users] never ending logging<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><div dir="ltr"><div>Hi Nicolas,</div><br><div>I would like to mention 2 things here. <br></div><br><div>1) The bricks not coming online might be because of the glusterd not being in the right state.</div><div>As you mentioned above, the node rejection is most probably because of the volfile mismatch. <br></div><div>During the glusterd reboot, we check for the volfile in each of the nodes to be the same. <br></div><div>If any node has a difference in volfile, you will notice that node to be rejected in the peer status.</div><div>To overcome this, we usually delete/copy-paste the volfile from the correct machine to the rest <br></div><div>and then restart glusterd on each node one after the other. <a class="gmail_plusreply" id="plusReplyChip-0" href="mailto:srakonde@redhat.com" tabindex="-1" target="_blank">@Sanju Rakonde</a> can explain more on this.</div><br><div>2) The "another transaction is in progress" is a message you usually get when you try to <br></div><div>run any gluster command from different nodes. This would rarely hit the locking part and throw this message.</div><div>If you see it a number of times, then it might be the fact that glusterd is hanging somewhere. <br></div><div>If you saw this post the above rejection issue, please do fix the glusterd as per the above steps and then try.</div><div>That should fix it. Else we need to restart gluster for this particular issue</div><br><div>Looks like the resultant state is because of the volfile mismatch, please do check fix that. This will inturn fix the <br></div><div>peer rejection. Once the rejection is fixed, rest of the things should fall into the right place. <br></div><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 29, 2020 at 5:01 PM <<a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Thanks for feedback.<br></div><br><div>No problem at all before node rejection, all worked fine but I had to renew SSL certificates which were about to expire. The node rejection occured after certificate renewal.<br></div><br><div>I'll send you links for sosreports (Debian) in separate mail.<br></div><br><div>Regards,<br></div><div>Nicolas.<br></div><br><br><hr id="gmail-m_276718702953597552zwchr"><div><b>De: </b>"Nikhil Ladha" <<a href="mailto:nladha@redhat.com" target="_blank">nladha@redhat.com</a>><br><b>À: </b><a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Envoyé: </b>Mercredi 29 Avril 2020 12:38:00<br><b>Objet: </b>Re: [Gluster-users] never ending logging<br></div><br><div><div dir="ltr">Hi,<br><div>It seems to be like using the step mentioned here <a href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Resolving%20Peer%20Rejected/" target="_blank">https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Resolving%20Peer%20Rejected/</a> is the reason you are facing so much problems. You said that the vol files in all the nodes are different which should never happen, because that's the purpose of shd to heal the vol files in case of any differences in the files, but dues to several deletions of the files and starting the glusterd service again and then disconnection of the nodes has lead to a state where that task is left incomplete. And whenever you are again trying to start the volume or get the status then that heal runs but it's not able to complete as it's not able to get a proper head through which it could make the heal.</div><div>Before the node rejection part, and you applying those steps to rectify it, did everything worked fine?</div><div>Also, can you share the sos-reports if not then the vol files which you see as different. So, that I can be sure of that.</div><br><div><div><div dir="ltr"><div dir="ltr"><div>Regards</div><div>Nikhil Ladha</div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 29, 2020 at 12:54 PM <<a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>I made another test, I restarded glusterd on all 3 nodes and right after restart I can get partial volume status but Locking failed occurs a few seconds after.<br></div><br><div>Example of output on node 2 and 3:<br></div><div>root@glusterDevVM2:~# systemctl restart glusterd<br>root@glusterDevVM2:~# gluster volume status tmp<br>Status of volume: tmp<br>Gluster process TCP Port RDMA Port Online Pid<br>------------------------------------------------------------------------------<br>Brick glusterDevVM2:/bricks/tmp/brick1/data N/A N/A N N/A<br>Self-heal Daemon on localhost N/A N/A N N/A<br><br>Task Status of Volume tmp<br>------------------------------------------------------------------------------<br>There are no active volume tasks<br><br>root@glusterDevVM2:~# gluster volume status tmp<br>Locking failed on glusterDevVM1. Please check log file for details.<br>root@glusterDevVM2:~# gluster volume status tmp<br>Status of volume: tmp<br>Gluster process TCP Port RDMA Port Online Pid<br>------------------------------------------------------------------------------<br>Brick glusterDevVM1:/bricks/tmp/brick1/data 49215 0 Y 5335<br>Brick glusterDevVM2:/bricks/tmp/brick1/data 49215 0 Y 5239<br>Self-heal Daemon on localhost N/A N/A N N/A<br>Self-heal Daemon on glusterDevVM1 N/A N/A N N/A<br><br>Task Status of Volume tmp<br>------------------------------------------------------------------------------<br>There are no active volume tasks<br><br>root@glusterDevVM2:~# gluster volume status tmp<br>Locking failed on glusterDevVM3. Please check log file for details.<br>Locking failed on glusterDevVM1. Please check log file for details.<br><br></div><br><div>root@glusterDevVM3:~# systemctl restart glusterd<br>root@glusterDevVM3:~# gluster volume status tmp<br>Status of volume: tmp<br>Gluster process TCP Port RDMA Port Online Pid<br>------------------------------------------------------------------------------<br>Brick glusterDevVM1:/bricks/tmp/brick1/data 49215 0 Y 5335<br>Brick glusterDevVM2:/bricks/tmp/brick1/data 49215 0 Y 5239<br>Brick glusterDevVM3:/bricks/tmp/brick1/data 49215 0 Y 3693<br>Self-heal Daemon on localhost N/A N/A N N/A<br>Self-heal Daemon on glusterDevVM2 N/A N/A N N/A<br>Self-heal Daemon on glusterDevVM1 N/A N/A Y 102850<br><br>Task Status of Volume tmp<br>------------------------------------------------------------------------------<br>There are no active volume tasks<br><br>root@glusterDevVM3:~# gluster volume status tmp<br>Locking failed on glusterDevVM2. Please check log file for details.<br>root@glusterDevVM3:~# gluster volume status tmp<br>Locking failed on glusterDevVM1. Please check log file for details.<br>Locking failed on glusterDevVM2. Please check log file for details.<br>root@glusterDevVM3:~# systemctl restart glusterd<br>root@glusterDevVM3:~# gluster volume status tmp<br>Status of volume: tmp<br>Gluster process TCP Port RDMA Port Online Pid<br>------------------------------------------------------------------------------<br>Brick glusterDevVM3:/bricks/tmp/brick1/data N/A N/A N N/A<br>Self-heal Daemon on localhost N/A N/A N N/A<br><br>Task Status of Volume tmp<br>------------------------------------------------------------------------------<br>There are no active volume tasks<br><br>root@glusterDevVM3:~# gluster volume status tmp<br>Another transaction is in progress for tmp. Please try again after some time.<br>root@glusterDevVM3:~# gluster volume status tmp<br>Another transaction is in progress for tmp. Please try again after some time.<br>root@glusterDevVM3:~# gluster volume status tmp<br>Another transaction is in progress for tmp. Please try again after some time.<br><br></div><br><hr id="gmail-m_276718702953597552gmail-m_2230220324538369392zwchr"><div><b>De: </b>"Nikhil Ladha" <<a href="mailto:nladha@redhat.com" target="_blank">nladha@redhat.com</a>><br><b>À: </b><a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Envoyé: </b>Mardi 28 Avril 2020 14:17:46<br><b>Objet: </b>Re: [Gluster-users] never ending logging<br></div><br><div><div dir="ltr">Hi,<br><div>It says syntax error in the log you shared, so there must be some mistake in what you are passing as an argument or some spelling mistake. Otherwise, how come it will run on one them and not on other having same configuration?</div><div>Also, can you please share the complete log file.</div><div>And try restarting all the nodes in the tsp, and execute the commands.</div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div>Regards</div><div>Nikhil Ladha</div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 28, 2020 at 5:20 PM <<a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Hi.<br></div><br><div>Not really worked well, I restarted node 2 at least a dozen of times until almost all bricks go online but the Rejected state disapeared after applying the fix.<br></div><div>I'm not able to create a volume as all gluster commands are issuing the "Another transaction is in progress" error.<br></div><div>All ping are less than 0.5ms.<br></div><br><div>I noticed another error in brick logs for a failed brick :<br></div><div>[2020-04-28 10:58:59.009933] E [MSGID: 101021] [graph.y:364:graphyyerror] 0-parser: syntax error: line 140 (volume 'data_export-server'): "!SSLv2"<br>allowed tokens are 'volume', 'type', 'subvolumes', 'option', 'end-volume'()<br><br></div><div>root@glusterDevVM2:/var/lib/glusterd/vols/data_export# grep -n SSLv2 *<br>data_export.gfproxyd.vol:8: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export.gfproxyd.vol:26: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export.gfproxyd.vol:44: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export.glusterDevVM1.bricks-data_export-brick1-data.vol:140: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export.glusterDevVM2.bricks-data_export-brick1-data.vol:140: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export.glusterDevVM3.bricks-data_export-brick1-data.vol:145: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export-shd.vol:7: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export-shd.vol:24: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export-shd.vol:41: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export.tcp-fuse.vol:8: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export.tcp-fuse.vol:24: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>data_export.tcp-fuse.vol:40: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>info:22:ssl.cipher-list=HIGH:\!SSLv2<br>trusted-data_export.tcp-fuse.vol:8: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>trusted-data_export.tcp-fuse.vol:26: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>trusted-data_export.tcp-fuse.vol:44: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br>trusted-data_export.tcp-gfproxy-fuse.vol:8: option transport.socket.ssl-cipher-list HIGH:\!SSLv2<br><br></div><div>Another volume with same parameters don't show this error :<br></div><div>root@glusterDevVM2:/var/lib/glusterd/vols/userfiles# grep '2020-04-28 10:5[89]:' /var/log/glusterfs/bricks/bricks-userfiles-brick1-data.log<br>[2020-04-28 10:58:53.427441] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 7.5 (args: /usr/sbin/glusterfsd -s glusterDevVM2 --volfile-id userfiles.glusterDevVM2.bricks-userfiles-brick1-data -p /var/run/gluster/vols/userfiles/glusterDevVM2-bricks-userfiles-brick1-data.pid -S /var/run/gluster/072c6be1df6e31e4.socket --brick-name /bricks/userfiles/brick1/data -l /var/log/glusterfs/bricks/bricks-userfiles-brick1-data.log --xlator-option *-posix.glusterd-uuid=7f6c3023-144b-4db2-9063-d90926dbdd18 --process-name brick --brick-port 49216 --xlator-option userfiles-server.listen-port=49216)<br>[2020-04-28 10:58:53.428426] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 5184<br>[2020-04-28 10:58:53.432337] I [socket.c:4350:ssl_setup_connection_params] 0-socket.glusterfsd: SSL support for glusterd is ENABLED<br>[2020-04-28 10:58:53.436982] I [socket.c:4360:ssl_setup_connection_params] 0-socket.glusterfsd: using certificate depth 1<br>[2020-04-28 10:58:53.437873] I [socket.c:958:__socket_server_bind] 0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9<br>[2020-04-28 10:58:53.438830] I [socket.c:4347:ssl_setup_connection_params] 0-glusterfs: SSL support on the I/O path is ENABLED<br>[2020-04-28 10:58:53.439206] I [socket.c:4350:ssl_setup_connection_params] 0-glusterfs: SSL support for glusterd is ENABLED<br>[2020-04-28 10:58:53.439238] I [socket.c:4360:ssl_setup_connection_params] 0-glusterfs: using certificate depth 1<br>[2020-04-28 10:58:53.441296] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0<br>[2020-04-28 10:58:53.441434] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2020-04-28 10:59:01.609052] I [rpcsvc.c:2690:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64<br>[2020-04-28 10:59:01.609353] I [socket.c:4347:ssl_setup_connection_params] 0-tcp.userfiles-server: SSL support on the I/O path is ENABLED<br>[2020-04-28 10:59:01.609373] I [socket.c:4350:ssl_setup_connection_params] 0-tcp.userfiles-server: SSL support for glusterd is ENABLED<br>[2020-04-28 10:59:01.609388] I [socket.c:4360:ssl_setup_connection_params] 0-tcp.userfiles-server: using certificate depth 1<br>[2020-04-28 10:59:01.609403] I [socket.c:4363:ssl_setup_connection_params] 0-tcp.userfiles-server: using cipher list HIGH:!SSLv2<br>[2020-04-28 10:59:01.644924] I [socket.c:4350:ssl_setup_connection_params] 0-socket.userfiles-changelog: SSL support for glusterd is ENABLED<br>[2020-04-28 10:59:01.644958] I [socket.c:4360:ssl_setup_connection_params] 0-socket.userfiles-changelog: using certificate depth 1<br></div><br><br><hr id="gmail-m_276718702953597552gmail-m_2230220324538369392gmail-m_-8027700322884173845zwchr"><div><b>De: </b>"Nikhil Ladha" <<a href="mailto:nladha@redhat.com" target="_blank">nladha@redhat.com</a>><br><b>À: </b><a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Envoyé: </b>Mardi 28 Avril 2020 09:31:45<br><b>Objet: </b>Re: [Gluster-users] never ending logging<br></div><br><div><div dir="ltr"><div dir="ltr">Hi,<br><div>Okay. So, after applying the fix everything worked well? Means all the peers were in connected state?</div><div>If so, can you try creating a new volume without enabling SSL and share the log, and also for the volume that is not starting can you try the steps mentioned here <a href="https://docs.gluster.org/en/latest/Troubleshooting/troubleshooting-glusterd/" rel="noopener noreferrer" target="_blank">https://docs.gluster.org/en/latest/Troubleshooting/troubleshooting-glusterd/</a> only in the 'Common issues how to resolve them section" and what logs do you get?</div><div>Also, could you ping test all the peers?</div><div>And the error 'failed to fetch volume files' occurs as it is not able to fetch the vol file from it's peers, as all the peers in the cluster share the same vol file for a volume.</div><div><div><div dir="ltr"><div dir="ltr"><br><div>Regards</div><div>Nikhil Ladha</div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 28, 2020 at 12:37 PM <<a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Hi.<br></div><br><div>No operation on any volume nor brick, the only change was SSL certificate renewal on 3 nodes and all clients. Then, node 2 was rejected and I applied following steps to fix : <span id="gmail-m_276718702953597552gmail-m_2230220324538369392gmail-m_-8027700322884173845gmail-m_-5956154649879068912OBJ_PREFIX_DWT244_com_zimbra_url"><span id="gmail-m_276718702953597552gmail-m_2230220324538369392gmail-m_-8027700322884173845gmail-m_-5956154649879068912OBJ_PREFIX_DWT253_com_zimbra_url"><a href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Resolving%20Peer%20Rejected/" rel="noopener noreferrer" target="_blank">https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Resolving%20Peer%20Rejected/</a></span></span><br>I also saw <span id="gmail-m_276718702953597552gmail-m_2230220324538369392gmail-m_-8027700322884173845gmail-m_-5956154649879068912OBJ_PREFIX_DWT245_com_zimbra_url"><span id="gmail-m_276718702953597552gmail-m_2230220324538369392gmail-m_-8027700322884173845gmail-m_-5956154649879068912OBJ_PREFIX_DWT254_com_zimbra_url"><a href="https://docs.gluster.org/en/latest/Troubleshooting/troubleshooting-glusterd/" rel="noopener noreferrer" target="_blank">https://docs.gluster.org/en/latest/Troubleshooting/troubleshooting-glusterd/</a></span></span> but solution wasn't compatible as cluster.max-op-version doesn't exist and all op-version are the same on all 3 nodes. <br></div><br><div>The strange thing is error "failed to fetch volume file" occurs on the node owning the brick, does it means it can't access it's own brick ?<br></div><br><div>Regards,<br></div><div>Nicolas.<br></div><br><hr id="gmail-m_276718702953597552gmail-m_2230220324538369392gmail-m_-8027700322884173845gmail-m_-5956154649879068912zwchr"><div><b>De: </b>"Nikhil Ladha" <<a href="mailto:nladha@redhat.com" target="_blank">nladha@redhat.com</a>><br><b>À: </b><a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Envoyé: </b>Mardi 28 Avril 2020 07:43:20<br><b>Objet: </b>Re: [Gluster-users] never ending logging<br></div><br><div><div dir="ltr">Hi,<br><div>Since, all things are working fine except few bricks which are not coming up, I doubt there is any issue with gluster itself. Did you by chance made any changes to those bricks or the volume or the node to which they are linked?</div><div>And as far as SSL logs are concerned, I am looking into that matter.</div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div>Regards</div><div>Nikhil Ladha</div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 27, 2020 at 7:17 PM <<a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Thanks for reply.<br></div><br><div>I updated storage pool in 7.5 and restarted all 3 nodes sequentially.<br></div><div>All nodes now appear in Connected state from every node and gluster volume list show all 74 volumes.</div><div>SSL log lines are still flooding glusterd log file on all nodes but don't appear on grick log files. As there's no information about volume nor client on these lines I'm not able to check if a certain volume produce this error or not.<br></div><div>I alos tried pstack after installing Debian package glusterfs-dbg but still getting "No symbols" error<br></div><br><div>I found that 5 brick processes didn't start on node 2 and 1 on node 3<br></div><div>[2020-04-27 11:54:23.622659] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 7.5 (args: /usr/sbin/glusterfsd -s glusterDevVM2 --volfile-id svg_pg_wed_dev_bkp.glusterDevVM2.bricks-svg_pg_wed_dev_bkp-brick1-data -p /var/run/gluster/vols/svg_pg_wed_dev_bkp/glusterDevVM2-bricks-svg_pg_wed_dev_bkp-brick1-data.pid -S /var/run/gluster/5023d38a22a8a874.socket --brick-name /bricks/svg_pg_wed_dev_bkp/brick1/data -l /var/log/glusterfs/bricks/bricks-svg_pg_wed_dev_bkp-brick1-data.log --xlator-option *-posix.glusterd-uuid=7f6c3023-144b-4db2-9063-d90926dbdd18 --process-name brick --brick-port 49206 --xlator-option svg_pg_wed_dev_bkp-server.listen-port=49206)<br>[2020-04-27 11:54:23.632870] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 5331<br>[2020-04-27 11:54:23.636679] I [socket.c:4350:ssl_setup_connection_params] 0-socket.glusterfsd: SSL support for glusterd is ENABLED<br>[2020-04-27 11:54:23.636745] I [socket.c:4360:ssl_setup_connection_params] 0-socket.glusterfsd: using certificate depth 1<br>[2020-04-27 11:54:23.637580] I [socket.c:958:__socket_server_bind] 0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9<br>[2020-04-27 11:54:23.637932] I [socket.c:4347:ssl_setup_connection_params] 0-glusterfs: SSL support on the I/O path is ENABLED<br>[2020-04-27 11:54:23.637949] I [socket.c:4350:ssl_setup_connection_params] 0-glusterfs: SSL support for glusterd is ENABLED<br>[2020-04-27 11:54:23.637960] I [socket.c:4360:ssl_setup_connection_params] 0-glusterfs: using certificate depth 1<br>[2020-04-27 11:54:23.639324] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0<br>[2020-04-27 11:54:23.639380] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2020-04-27 11:54:28.933102] E [glusterfsd-mgmt.c:2217:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server<br>[2020-04-27 11:54:28.933134] E [glusterfsd-mgmt.c:2416:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:svg_pg_wed_dev_bkp.glusterDevVM2.bricks-svg_pg_wed_dev_bkp-brick1-data)<br>[2020-04-27 11:54:28.933361] W [glusterfsd.c:1596:cleanup_and_exit] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe5d1) [0x7f2b08ec35d1] -->/usr/sbin/glusterfsd(mgmt_getspec_cbk+0x8d0) [0x55d46cb5a110] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x54) [0x55d46cb51ec4] ) 0-: received signum (0), shutting down<br></div><br><div>I tried to stop the volume but gluster commands are still locked (Another transaction is in progress.).<br></div><br><div>Best regards,<br></div><div>Nicolas.<br></div><br><hr id="gmail-m_276718702953597552gmail-m_2230220324538369392gmail-m_-8027700322884173845gmail-m_-5956154649879068912gmail-m_5100860692984386704zwchr"><div><b>De: </b>"Nikhil Ladha" <<a href="mailto:nladha@redhat.com" target="_blank">nladha@redhat.com</a>><br><b>À: </b><a href="mailto:nico@furyweb.fr" target="_blank">nico@furyweb.fr</a><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br><b>Envoyé: </b>Lundi 27 Avril 2020 13:34:47<br><b>Objet: </b>Re: [Gluster-users] never ending logging<br></div><br><div><div dir="ltr">Hi,<br><div>As you mentioned that the node 2 is in "semi-connected" state, I think due to that the locking of volume is failing, and since it is failing in one of the volumes the transaction is not complete and you are seeing a transaction error on another volume.</div><div>Moreover, for the repeated logging of lines :</div><div>SSL support on the I/O path is enabled, SSL support for glusterd is enabled and using certificate depth 1</div><div>If you can try creating a volume without having ssl enabled and then check if the same log messages appear.</div><div><div>Also, if you update to 7.5, and find any change in log message with SSL ENABLED, then please do share that.</div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div>Regards</div><div>Nikhil Ladha</div></div></div></div></div></div></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div></div></blockquote></div><br></div></div></div>________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Regards,<br>Hari Gowtham.</div><br></div></div></body></html>