<div dir="ltr">I have posted a patch <a href="https://review.gluster.org/#/c/20657/">https://review.gluster.org/#/c/20657/</a> and start brick-mux regression to validate the patch.<div><br></div><div>Thanks</div><div>Mohit Agrawal</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 8, 2018 at 7:22 AM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="auto">+Mohit</div><div dir="auto"><br></div><div dir="auto">Requesting Mohit for help.</div></div><div><br><div class="gmail_quote"><div dir="ltr">On Wed, 8 Aug 2018 at 06:53, Shyam Ranganathan <<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 08/07/2018 07:37 PM, Shyam Ranganathan wrote:<br>
> 5) Current test failures<br>
> We still have the following tests failing and some without any RCA or<br>
> attention, (If something is incorrect, write back).<br>
> <br>
> ./tests/bugs/ec/bug-1236065.t (Ashish)<br>
<br>
Ashish/Atin, the above test failed in run:<br>
<a href="https://build.gluster.org/job/regression-on-demand-multiplex/172/consoleFull" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-on-demand-<wbr>multiplex/172/consoleFull</a><br>
<br>
The above run is based on patchset 4 of<br>
<a href="https://review.gluster.org/#/c/20637/4" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/20637/4</a><br>
<br>
The logs look as below, and as Ashish is unable to reproduce this, and<br>
all failures are on line 78 with a heal outstanding of 105, looks like<br>
this run may provide some possibilities on narrowing it down.<br>
<br>
The problem seems to be glustershd not connecting to one of the bricks<br>
that is restarted, and hence failing to heal that brick. This also looks<br>
like what Ravi RCAd for the test: ./tests/bugs/replicate/bug-<wbr>1363721.t<br>
<br>
==============================<wbr>====================<br>
Test times from: cat ./glusterd.log | grep TEST<br>
[2018-08-06 20:56:28.177386]:++++++++++<br>
G_LOG:./tests/bugs/ec/bug-<wbr>1236065.t: TEST: 77 gluster --mode=script<br>
--wignore volume heal patchy full ++++++++++<br>
[2018-08-06 20:56:28.767209]:++++++++++<br>
G_LOG:./tests/bugs/ec/bug-<wbr>1236065.t: TEST: 78 ^0$ get_pending_heal_count<br>
patchy ++++++++++<br>
[2018-08-06 20:57:48.957136]:++++++++++<br>
G_LOG:./tests/bugs/ec/bug-<wbr>1236065.t: TEST: 80 rm -f 0.o 10.o 11.o 12.o<br>
13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o<br>
++++++++++<br>
==============================<wbr>====================<br>
Repeated connection failure to client-3 in glustershd.log:<br>
[2018-08-06 20:56:30.218482] I [rpc-clnt.c:2087:rpc_clnt_<wbr>reconfig]<br>
0-patchy-client-3: changing port to 49152 (from 0)<br>
[2018-08-06 20:56:30.222738] W [MSGID: 114043]<br>
[client-handshake.c:1061:<wbr>client_setvolume_cbk] 0-patchy-client-3: failed<br>
to set the volume [Resource temporarily unavailable]<br>
[2018-08-06 20:56:30.222788] W [MSGID: 114007]<br>
[client-handshake.c:1090:<wbr>client_setvolume_cbk] 0-patchy-client-3: failed<br>
to get 'process-uuid' from reply dict [Invalid argument]<br>
[2018-08-06 20:56:30.222813] E [MSGID: 114044]<br>
[client-handshake.c:1096:<wbr>client_setvolume_cbk] 0-patchy-client-3:<br>
SETVOLUME on remote-host failed: cleanup flag is set for xlator. Try<br>
again later [Resource tempor<br>
arily unavailable]<br>
[2018-08-06 20:56:30.222845] I [MSGID: 114051]<br>
[client-handshake.c:1201:<wbr>client_setvolume_cbk] 0-patchy-client-3:<br>
sending CHILD_CONNECTING event<br>
[2018-08-06 20:56:30.222919] I [MSGID: 114018]<br>
[client.c:2255:client_rpc_<wbr>notify] 0-patchy-client-3: disconnected from<br>
patchy-client-3. Client process will keep trying to connect to glusterd<br>
until brick's port is<br>
available<br>
==============================<wbr>====================<br>
Repeated connection messages close to above retries in<br>
d-backends-patchy0.log:<br>
[2018-08-06 20:56:38.530009] I [addr.c:55:compare_addr_and_<wbr>update]<br>
0-/d/backends/patchy0: allowed = "*", received addr = "127.0.0.1"<br>
[2018-08-06 20:56:38.530044] I [login.c:111:gf_auth] 0-auth/login:<br>
allowed user names: 756f302a-66eb-4cc0-8f91-<wbr>797183312f05<br>
The message "I [MSGID: 101016] [glusterfs3.h:739:dict_to_xdr] 0-dict:<br>
key 'trusted.ec.version' is would not be sent on wire in future [Invalid<br>
argument]" repeated 6 times between [2018-08-06 20:56:37.931040] and<br>
[2018-08-06 20:56:37.933084]<br>
[2018-08-06 20:56:38.530067] I [MSGID: 115029]<br>
[server-handshake.c:786:<wbr>server_setvolume] 0-patchy-server: accepted<br>
client from<br>
CTX_ID:cb3b4fed-62a4-4ad5-<wbr>8b92-97838c651b22-GRAPH_ID:0-<wbr>PID:10506-HOST:builder104.clo<br>
ud.gluster.org-PC_NAME:patchy-<wbr>client-0-RECON_NO:-0 (version: 4.2dev)<br>
[2018-08-06 20:56:38.540499] I [addr.c:55:compare_addr_and_<wbr>update]<br>
0-/d/backends/patchy1: allowed = "*", received addr = "127.0.0.1"<br>
[2018-08-06 20:56:38.540533] I [login.c:111:gf_auth] 0-auth/login:<br>
allowed user names: 756f302a-66eb-4cc0-8f91-<wbr>797183312f05<br>
[2018-08-06 20:56:38.540555] I [MSGID: 115029]<br>
[server-handshake.c:786:<wbr>server_setvolume] 0-patchy-server: accepted<br>
client from<br>
CTX_ID:cb3b4fed-62a4-4ad5-<wbr>8b92-97838c651b22-GRAPH_ID:0-<wbr>PID:10506-HOST:builder104.clo<br>
ud.gluster.org-PC_NAME:patchy-<wbr>client-1-RECON_NO:-0 (version: 4.2dev)<br>
[2018-08-06 20:56:38.552442] I [addr.c:55:compare_addr_and_<wbr>update]<br>
0-/d/backends/patchy2: allowed = "*", received addr = "127.0.0.1"<br>
[2018-08-06 20:56:38.552472] I [login.c:111:gf_auth] 0-auth/login:<br>
allowed user names: 756f302a-66eb-4cc0-8f91-<wbr>797183312f05<br>
[2018-08-06 20:56:38.552494] I [MSGID: 115029]<br>
[server-handshake.c:786:<wbr>server_setvolume] 0-patchy-server: accepted<br>
client from<br>
CTX_ID:cb3b4fed-62a4-4ad5-<wbr>8b92-97838c651b22-GRAPH_ID:0-<wbr>PID:10506-HOST:builder104.clo<br>
ud.gluster.org-PC_NAME:patchy-<wbr>client-2-RECON_NO:-0 (version: 4.2dev)<br>
[2018-08-06 20:56:38.571671] I [addr.c:55:compare_addr_and_<wbr>update]<br>
0-/d/backends/patchy4: allowed = "*", received addr = "127.0.0.1"<br>
[2018-08-06 20:56:38.571701] I [login.c:111:gf_auth] 0-auth/login:<br>
allowed user names: 756f302a-66eb-4cc0-8f91-<wbr>797183312f05<br>
[2018-08-06 20:56:38.571723] I [MSGID: 115029]<br>
[server-handshake.c:786:<wbr>server_setvolume] 0-patchy-server: accepted<br>
client from<br>
CTX_ID:cb3b4fed-62a4-4ad5-<wbr>8b92-97838c651b22-GRAPH_ID:0-<wbr>PID:10506-HOST:builder104.clo<br>
ud.gluster.org-PC_NAME:patchy-<wbr>client-4-RECON_NO:-0 (version: 4.2dev)<br>
[2018-08-06 20:56:38.580579] I [addr.c:55:compare_addr_and_<wbr>update]<br>
0-/d/backends/patchy5: allowed = "*", received addr = "127.0.0.1"<br>
[2018-08-06 20:56:38.580609] I [login.c:111:gf_auth] 0-auth/login:<br>
allowed user names: 756f302a-66eb-4cc0-8f91-<wbr>797183312f05<br>
[2018-08-06 20:56:38.580630] I [MSGID: 115029]<br>
[server-handshake.c:786:<wbr>server_setvolume] 0-patchy-server: accepted<br>
client from<br>
CTX_ID:cb3b4fed-62a4-4ad5-<wbr>8b92-97838c651b22-GRAPH_ID:0-<wbr>PID:10506-HOST:builder104.clo<br>
ud.gluster.org-PC_NAME:patchy-<wbr>client-5-RECON_NO:-0 (version: 4.2dev)<br>
[2018-08-06 20:56:38.583444] I [addr.c:55:compare_addr_and_<wbr>update]<br>
0-/d/backends/patchy6: allowed = "*", received addr = "127.0.0.1"<br>
[2018-08-06 20:56:38.583472] I [login.c:111:gf_auth] 0-auth/login:<br>
allowed user names: 756f302a-66eb-4cc0-8f91-<wbr>797183312f05<br>
[2018-08-06 20:56:38.583493] I [MSGID: 115029]<br>
[server-handshake.c:786:<wbr>server_setvolume] 0-patchy-server: accepted<br>
client from<br>
CTX_ID:cb3b4fed-62a4-4ad5-<wbr>8b92-97838c651b22-GRAPH_ID:0-<wbr>PID:10506-HOST:builder104.clo<br>
ud.gluster.org-PC_NAME:patchy-<wbr>client-6-RECON_NO:-0 (version: 4.2dev)<br>
<br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><span class="HOEnZb"><font color="#888888"><br>
</font></span></blockquote></div></div><span class="HOEnZb"><font color="#888888">-- <br><div dir="ltr" class="m_2685237664256304868gmail_signature" data-smartmail="gmail_signature">- Atin (atinm)</div>
</font></span></blockquote></div><br></div>