[Gluster-devel] Spurious regression of tests/basic/mgmt_v3-locks.t

[Atin Mukherjee]

On 11/03/2014 06:15 PM, Justin Clift wrote:
> On Sun, 02 Nov 2014 21:41:02 +0530
> Atin Mukherjee <amukherj at redhat.com> wrote:
>> On 10/31/2014 07:08 PM, Justin Clift wrote:
>>> On Fri, 31 Oct 2014 10:17:28 +0530
>>> Atin Mukherjee <amukherj at redhat.com> wrote:
>>> <snip>
>>>> Justin,
>>>>
>>>> For last three runs, I've observed the same failure. I think its
>>>> really the time to debug this without any further delay. Can you
>>>> please share a rackspace machine such that I can debug this issue?
>>>
>>> Yep, this is very doable.  It looks like Xavi already has this one
>>> under control though.
>> Xavi has a fix for the ec related failures, mgmt_v3 lock failure is
>> still unknown which I will be looking at. I would request you to lend
>> me a rackspace VM for the same.
> 
> Ahhh, sorry.  My misunderstanding.  slave26.cloud.gluster.org is set
> aside now for you to do stuff on.  The login details are the same as
> last time.  If you need me to resend them, just let me know. :)
> 
Here are my findings looking into a regression failure log [1]

mgmt_v3 lock test forms a cluster of 3 nodes say N1, N2 & N3. Now after
couple of remove-brick operations, peer status command reached to N1 and
after that glusterd stopped responding and I don't see any more logs in
mgmt_v3-locks.t_glusterd1.log. This could happen if glusterd is stuck
while performing a transaction or the glusterd process is down.

When I looked at the log file of other node N2 I could see that N1 is
disconnected which proves that glusterd went down in N1 (it couldn't be
network failure otherwise other commands should have been processed by
N1 itself). The interesting fact here is there is no core captured.

Can there be any cases where glusterd instance may go down unexpectedly
without a crash?

[1] http://build.gluster.org/job/rackspace-regression-2GB-triggered
/2319/consoleFull

~Atin
> + Justin
>