[Gluster-devel] Test: ./tests/bugs/ec/bug-1236065.t

Wed Aug 8 01:52:31 UTC 2018

+Mohit

Requesting Mohit for help.

On Wed, 8 Aug 2018 at 06:53, Shyam Ranganathan <srangana at redhat.com> wrote:

> On 08/07/2018 07:37 PM, Shyam Ranganathan wrote:
> > 5) Current test failures
> > We still have the following tests failing and some without any RCA or
> > attention, (If something is incorrect, write back).
> >
> > ./tests/bugs/ec/bug-1236065.t (Ashish)
>
> Ashish/Atin, the above test failed in run:
>
> https://build.gluster.org/job/regression-on-demand-multiplex/172/consoleFull
>
> The above run is based on patchset 4 of
> https://review.gluster.org/#/c/20637/4
>
> The logs look as below, and as Ashish is unable to reproduce this, and
> all failures are on line 78 with a heal outstanding of 105, looks like
> this run may provide some possibilities on narrowing it down.
>
> The problem seems to be glustershd not connecting to one of the bricks
> that is restarted, and hence failing to heal that brick. This also looks
> like what Ravi RCAd for the test: ./tests/bugs/replicate/bug-1363721.t
>
> ==================================================
> Test times from: cat ./glusterd.log | grep TEST
> [2018-08-06 20:56:28.177386]:++++++++++
> G_LOG:./tests/bugs/ec/bug-1236065.t: TEST: 77 gluster --mode=script
> --wignore volume heal patchy full ++++++++++
> [2018-08-06 20:56:28.767209]:++++++++++
> G_LOG:./tests/bugs/ec/bug-1236065.t: TEST: 78 ^0$ get_pending_heal_count
> patchy ++++++++++
> [2018-08-06 20:57:48.957136]:++++++++++
> G_LOG:./tests/bugs/ec/bug-1236065.t: TEST: 80 rm -f 0.o 10.o 11.o 12.o
> 13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o
> ++++++++++
> ==================================================
> Repeated connection failure to client-3 in glustershd.log:
> [2018-08-06 20:56:30.218482] I [rpc-clnt.c:2087:rpc_clnt_reconfig]
> 0-patchy-client-3: changing port to 49152 (from 0)
> [2018-08-06 20:56:30.222738] W [MSGID: 114043]
> [client-handshake.c:1061:client_setvolume_cbk] 0-patchy-client-3: failed
> to set the volume [Resource temporarily unavailable]
> [2018-08-06 20:56:30.222788] W [MSGID: 114007]
> [client-handshake.c:1090:client_setvolume_cbk] 0-patchy-client-3: failed
> to get 'process-uuid' from reply dict [Invalid argument]
> [2018-08-06 20:56:30.222813] E [MSGID: 114044]
> [client-handshake.c:1096:client_setvolume_cbk] 0-patchy-client-3:
> SETVOLUME on remote-host failed: cleanup flag is set for xlator.  Try
> again later [Resource tempor
> arily unavailable]
> [2018-08-06 20:56:30.222845] I [MSGID: 114051]
> [client-handshake.c:1201:client_setvolume_cbk] 0-patchy-client-3:
> sending CHILD_CONNECTING event
> [2018-08-06 20:56:30.222919] I [MSGID: 114018]
> [client.c:2255:client_rpc_notify] 0-patchy-client-3: disconnected from
> patchy-client-3. Client process will keep trying to connect to glusterd
> until brick's port is
>  available
> ==================================================
> Repeated connection messages close to above retries in
> d-backends-patchy0.log:
> [2018-08-06 20:56:38.530009] I [addr.c:55:compare_addr_and_update]
> 0-/d/backends/patchy0: allowed = "*", received addr = "127.0.0.1"
> [2018-08-06 20:56:38.530044] I [login.c:111:gf_auth] 0-auth/login:
> allowed user names: 756f302a-66eb-4cc0-8f91-797183312f05
> The message "I [MSGID: 101016] [glusterfs3.h:739:dict_to_xdr] 0-dict:
> key 'trusted.ec.version' is would not be sent on wire in future [Invalid
> argument]" repeated 6 times between [2018-08-06 20:56:37.931040] and
>  [2018-08-06 20:56:37.933084]
> [2018-08-06 20:56:38.530067] I [MSGID: 115029]
> [server-handshake.c:786:server_setvolume] 0-patchy-server: accepted
> client from
>
> CTX_ID:cb3b4fed-62a4-4ad5-8b92-97838c651b22-GRAPH_ID:0-PID:10506-HOST:builder104.clo
> ud.gluster.org-PC_NAME:patchy-client-0-RECON_NO:-0 (version: 4.2dev)
> [2018-08-06 20:56:38.540499] I [addr.c:55:compare_addr_and_update]
> 0-/d/backends/patchy1: allowed = "*", received addr = "127.0.0.1"
> [2018-08-06 20:56:38.540533] I [login.c:111:gf_auth] 0-auth/login:
> allowed user names: 756f302a-66eb-4cc0-8f91-797183312f05
> [2018-08-06 20:56:38.540555] I [MSGID: 115029]
> [server-handshake.c:786:server_setvolume] 0-patchy-server: accepted
> client from
>
> CTX_ID:cb3b4fed-62a4-4ad5-8b92-97838c651b22-GRAPH_ID:0-PID:10506-HOST:builder104.clo
> ud.gluster.org-PC_NAME:patchy-client-1-RECON_NO:-0 (version: 4.2dev)
> [2018-08-06 20:56:38.552442] I [addr.c:55:compare_addr_and_update]
> 0-/d/backends/patchy2: allowed = "*", received addr = "127.0.0.1"
> [2018-08-06 20:56:38.552472] I [login.c:111:gf_auth] 0-auth/login:
> allowed user names: 756f302a-66eb-4cc0-8f91-797183312f05
> [2018-08-06 20:56:38.552494] I [MSGID: 115029]
> [server-handshake.c:786:server_setvolume] 0-patchy-server: accepted
> client from
>
> CTX_ID:cb3b4fed-62a4-4ad5-8b92-97838c651b22-GRAPH_ID:0-PID:10506-HOST:builder104.clo
> ud.gluster.org-PC_NAME:patchy-client-2-RECON_NO:-0 (version: 4.2dev)
> [2018-08-06 20:56:38.571671] I [addr.c:55:compare_addr_and_update]
> 0-/d/backends/patchy4: allowed = "*", received addr = "127.0.0.1"
> [2018-08-06 20:56:38.571701] I [login.c:111:gf_auth] 0-auth/login:
> allowed user names: 756f302a-66eb-4cc0-8f91-797183312f05
> [2018-08-06 20:56:38.571723] I [MSGID: 115029]
> [server-handshake.c:786:server_setvolume] 0-patchy-server: accepted
> client from
>
> CTX_ID:cb3b4fed-62a4-4ad5-8b92-97838c651b22-GRAPH_ID:0-PID:10506-HOST:builder104.clo
> ud.gluster.org-PC_NAME:patchy-client-4-RECON_NO:-0 (version: 4.2dev)
> [2018-08-06 20:56:38.580579] I [addr.c:55:compare_addr_and_update]
> 0-/d/backends/patchy5: allowed = "*", received addr = "127.0.0.1"
> [2018-08-06 20:56:38.580609] I [login.c:111:gf_auth] 0-auth/login:
> allowed user names: 756f302a-66eb-4cc0-8f91-797183312f05
> [2018-08-06 20:56:38.580630] I [MSGID: 115029]
> [server-handshake.c:786:server_setvolume] 0-patchy-server: accepted
> client from
>
> CTX_ID:cb3b4fed-62a4-4ad5-8b92-97838c651b22-GRAPH_ID:0-PID:10506-HOST:builder104.clo
> ud.gluster.org-PC_NAME:patchy-client-5-RECON_NO:-0 (version: 4.2dev)
> [2018-08-06 20:56:38.583444] I [addr.c:55:compare_addr_and_update]
> 0-/d/backends/patchy6: allowed = "*", received addr = "127.0.0.1"
> [2018-08-06 20:56:38.583472] I [login.c:111:gf_auth] 0-auth/login:
> allowed user names: 756f302a-66eb-4cc0-8f91-797183312f05
> [2018-08-06 20:56:38.583493] I [MSGID: 115029]
> [server-handshake.c:786:server_setvolume] 0-patchy-server: accepted
> client from
>
> CTX_ID:cb3b4fed-62a4-4ad5-8b92-97838c651b22-GRAPH_ID:0-PID:10506-HOST:builder104.clo
> ud.gluster.org-PC_NAME:patchy-client-6-RECON_NO:-0 (version: 4.2dev)
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
-- 
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180808/cdceb3d1/attachment-0001.html>