<div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Fri, 21 Dec 2018 at 15:54, Anand Malagi <<a href="mailto:amalagi@commvault.com">amalagi@commvault.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_-1798820892959906676WordSection1">
<p class="MsoNormal">Hi Friends,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Please note that, when replace-brick operation was tried for one of the bad brick present in distributed disperse EC volume, the command actually failed but the brick daemon of new replaced brick came online.</p></div></div></blockquote><div dir="auto"><br></div><div dir="auto">This is probably because the new brick was already added before a step in the transaction failed and unfortunately there’s no rollback mechanism in place to handle such situation.</div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_-1798820892959906676WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal">Please help to understand in what situations this issue may arise and proposed solution if possible ? :<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="m_-1798820892959906676MsoPlainText"><b>glusterd.log :<u></u><u></u></b></p>
<p class="m_-1798820892959906676MsoPlainText"><u></u> <u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:04:43.774120] I [MSGID: 106503] [glusterd-replace-brick.c:147:__glusterd_handle_replace_brick] 0-management: Received replace-brick commit force request.<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:04:44.784578] I [MSGID: 106504] [glusterd-utils.c:13079:rb_update_dstbrick_port] 0-glusterd: adding dst-brick port no 0<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">…<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:04:46.457537] E [MSGID: 106029] [glusterd-utils.c:7981:glusterd_brick_signal] 0-glusterd: Unable to open pidfile: /var/run/gluster/vols/AM6_HyperScale/am6sv0004sds.saipemnet.saipem.intranet-ws-disk3-ws_brick.pid [No such
file or directory]<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:04:53.089810] I [glusterd-utils.c:5876:glusterd_brick_start] 0-management: starting a fresh brick process for brick /ws/disk15/ws_brick<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">…<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText"><span style="background:yellow">[2018-12-11 11:04:53.117935] W [socket.c:595:__socket_rwv] 0-socket.management: writev on <a href="http://127.0.0.1:864" target="_blank">127.0.0.1:864</a> failed (Broken pipe)</span><u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:04:54.014023] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:04:54.273190] I [MSGID: 106005] [glusterd-handler.c:6120:__glusterd_brick_rpc_notify] 0-management: Brick am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick has disconnected from glusterd.<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:04:54.297603] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Commit failed on am6sv0006sds.saipemnet.saipem.intranet. Please check log file for details.<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:04:54.350666] I [MSGID: 106143] [glusterd-pmap.c:278:pmap_registry_bind] 0-pmap: adding brick /ws/disk15/ws_brick on port 49164<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText"><span style="background:yellow">[2018-12-11 11:05:01.137449] E [MSGID: 106123] [glusterd-mgmt.c:1519:glusterd_mgmt_v3_commit] 0-management: Commit failed on peers</span></p></div></div></blockquote><div dir="auto"><br></div><div dir="auto">I’m assuming that this operation actually failed on non local node, did you check the glusterd log of the node where this operation failed?</div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="#0563C1" vlink="#954F72"><div class="m_-1798820892959906676WordSection1"><p class="m_-1798820892959906676MsoPlainText"><span style="background:yellow"><u></u><u></u></span></p>
<p class="m_-1798820892959906676MsoPlainText"><span style="background:yellow">[2018-12-11 11:05:01.137496] E [MSGID: 106123] [glusterd-replace-brick.c:660:glusterd_mgmt_v3_initiate_replace_brick_cmd_phases] 0-management: Commit Op Failed</span><u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 11:06:12.275867] I [MSGID: 106499] [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: Received status volume req for volume AM6_HyperScale<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">[2018-12-11 13:35:51.529365] I [MSGID: 106499] [glusterd-handler.c:4370:__glusterd_handle_status_volume] 0-management: Received status volume req for volume AM6_HyperScale<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="background:yellow">gluster volume replace-brick AM6_HyperScale am6sv0004sds.saipemnet.saipem.intranet:/ws/disk3/ws_brick am6sv0004sds.saipemnet.saipem.intranet:/ws/disk15/ws_brick commit force<u></u><u></u></span></p>
<p class="MsoNormal">Replace brick failure, brick [/ws/disk3], volume [AM6_HyperScale]<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">“gluster volume status” now shows a new disk active /ws/disk15<u></u><u></u></p>
<p class="m_-1798820892959906676MsoPlainText">The replacement appears to be successful, looks like healing started<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><img src="cid:167e1000b54ad7999131" style="width:632px;max-width:100%"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks and Regards,<u></u><u></u></p>
<p class="MsoNormal">--Anand<u></u><u></u></p>
</div>
<div><div><div><div>********************************Legal Disclaimer********************************<div>"This communication may contain confidential and privileged material for the<div>sole use of the intended recipient. Any unauthorized review, use or distribution<div>by others is strictly prohibited. If you have received the message by mistake,<div>please advise the sender by reply email and delete the message. We may process<div>information in the email header of business emails sent and received by us<div>(including the names of recipient and sender, date and time of the email) for<div>the purposes of evaluating our existing or prospective business relationship.<div>The lawful basis we rely on for this processing is our legitimate interests. For<div>more information about how we use personal information please read our privacy<div>policy <a href="https://www.commvault.com/privacy-policy" target="_blank">https://www.commvault.com/privacy-policy</a>. Thank you."<div>********************************************************************************</div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">- Atin (atinm)</div>