[Gluster-devel] brick stops responding

[Emmanuel Dreyfus]

Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:

> Could you provide statedumps of the bricks and mounts please.

# gluster volume statedump gfs35b1 all 
volume statedump: failed: Commit failed on debacle. Please check log file for details.

glusterd log on server debacle:

[2014-01-17 04:55:17.855158] E [glusterd-utils.c:6579:glusterd_brick_statedump] 0-: error while parsing the statedump options
[2014-01-17 04:55:17.855311] W [glusterd-volume-ops.c:1879:glusterd_op_statedump_volume] 0-management: could not take the statedump of the brick debacle:/export/wd1a. Proceeding to other bricks
[2014-01-17 04:55:17.855343] E [glusterd-op-sm.c:3980:glusterd_op_ac_commit_op] 0-management: Commit of operation 'Volume Statedump' failed: -1

Restarting gluster daemons on debacle does not help. And in the meantime, the volume is reported as heahly:
# gluster volume status gfs35b1        
Status of volume: gfs35b1
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick silo:/export/wd2a                                 49152   Y       22495
Brick hangar:/export/wd1a                               49152   Y       16099
Brick hangar:/export/wd3a                               49153   Y       15711
Brick debacle:/export/wd1a                              49152   Y       611
NFS Server on localhost                                 N/A     N       N/A
Self-heal Daemon on localhost                           N/A     N       N/A
NFS Server on hangar                                    N/A     N       N/A
Self-heal Daemon on hangar                              N/A     N       N/A
NFS Server on debacle                                   N/A     N       N/A
Self-heal Daemon on debacle                             N/A     N       N/A
 
Task Status of Volume gfs35b1
------------------------------------------------------------------------------
There are no active volume tasks


> Just curious what does the test do?

I have always beeing running the same test, which is building NetBSD 
inside a glusterfs volume. This tests a lot of things, and it exposed many 
bugs so far. I think it would be valuable to have it done automatically:

0) setup a replicated and distributed volume, switch to an unprivilegied user
1) fetch .tgz from ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-6.1.2/source/sets/
2) for i in syssrc sharesrc gnusrc src ; do ( tar -xzf $i.tgz & ) ; done
3) cd usr/src && ./build.sh -Um i386 release

Last time I did it with success was on 3.4.0 with eager locks disabled. 
Building took 37 hours.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu at netbsd.org